Data extraction method and device, electronic equipment and storage medium

By generating data extraction interfaces through pre-set data toolkits and big data models, the problem of needing to write custom rules in existing technologies has been solved, enabling efficient and flexible data collection and storage.

CN118916537APending Publication Date: 2024-11-08CHONGQING SELIS PHOENIX INTELLIGENT INNOVATION TECH CO LTD
View PDF 0 Cites 1 Cited by

Patent Information

Application Number
CN202410783508.7
Authority / Receiving Office
CN · China
Patent Type
Applications(China)
Current Assignee / Owner
Filing Date
2024-06-18
Publication Date
2024-11-08

AI Technical Summary

Technical Problem

Existing data collection frameworks require users to write custom webpage extraction rules based on specific website scenarios. This requires high-level and time-consuming code, and the resulting code lacks scalability and is not applicable to a single scenario.

Method used

By extracting webpage text information through a pre-set data toolkit, generating a data extraction interface, obtaining key text data, cleaning and storing it, and using a multi-level data extraction model and a big data model to generate custom extraction rules based on a pre-set scenario, the time spent writing rules is reduced and scalability is improved.

Benefits of technology

This avoids spending a lot of time writing extraction rules, improves the efficiency and flexibility of data collection, and enables cross-scenario data extraction and storage.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure CN118916537A_ABST
    Figure CN118916537A_ABST
Patent Text Reader

Abstract

The embodiment of the invention discloses a data extraction method and device, equipment and a storage medium. The method comprises the following steps: extracting text information in a webpage based on a preset data toolkit; if the text information is effective text information, generating a data extraction interface based on preset scene information; acquiring key text data in the effective text information based on the data extraction interface, wherein the key text data comprises a webpage title, a webpage abstract and a webpage tag; and performing data cleaning on the key text data, and storing the cleaned key text data in combination with the webpage title, the webpage abstract and the webpage tag. According to the embodiment of the invention, a large amount of time can be prevented from being consumed to compile the extraction rule, the data acquisition technology is enabled to have expansibility through the preset scene information, and the use simplification of the applicable data extraction scene is avoided.
Need to check novelty before this filing date? Find Prior Art

Description

Technical Field

[0001] The present application relates to the field of computer network technology, and in particular to a data extraction method, device, electronic device, and computer-readable storage medium. Background Art

[0002] Data scraping is a program or script that automatically crawls the World Wide Web information according to certain rules. It is mainly used in search engines to read all the content and links of a website and create relevant full-text indexes in the database for comparison and retrieval when users search.

[0003] There are many existing data collection frameworks, such as selenium, scrapy, requests, html2text and urllib.request library packages in Python, and WebMagic and other framework packages in Java. However, using these data collection framework packages requires users to customize web page extraction rules according to specific website scenarios, which places extremely high demands on the user's code and takes a lot of time to write extraction rules. In addition, the data collection code written is not scalable and is used in a single scenario. Summary of the invention

[0004] To solve the above technical problems, embodiments of the present application provide a data extraction method, device, electronic device, and computer-readable storage medium.

[0005] According to one aspect of an embodiment of the present application, a data extraction method is provided, comprising: extracting text information from a web page based on a preset data toolkit; if the text information is valid text information, generating a data extraction interface based on preset scenario information; acquiring key text data in the valid text information based on the data extraction interface, the key text data including a web page title, a web page summary, and a web page tag; performing data cleaning on the key text data, and storing the cleaned key text data in combination with the web page title, the web page summary, and the web page tag.

[0006] According to one aspect of an embodiment of the present application, extracting text information from a web page based on a preset data toolkit includes: parsing the preset data toolkit to obtain a multi-level data extraction model included in the preset data packet; and extracting text information from the web page based on at least one level of data extraction model in the multi-level data extraction model.

[0007] According to one aspect of an embodiment of the present application, the method further includes: converting the text information into text data corresponding to a plain text format; if the text data is complete text data, determining that the text data is valid text information.

[0008] According to one aspect of an embodiment of the present application, generating a data extraction interface based on preset scene information includes: selecting a corresponding big data model based on the preset scene, the big data model being used to extract key text information corresponding to the preset scene; and generating a corresponding data extraction interface based on the big data model.

[0009] According to one aspect of an embodiment of the present application, the method also includes: generating custom extraction rules based on scene parameters corresponding to the preset scene; inputting the custom extraction rules and the valid text information into the big data model; generating a corresponding data extraction interface based on the big data model to extract key text data from the valid text information through the data extraction interface.

[0010] According to one aspect of an embodiment of the present application, the data cleaning of the key text data includes: if the key text data is complete text data, converting the key text data into preset character type data; and deduplicating the preset character type data to obtain target text data.

[0011] According to one aspect of an embodiment of the present application, the cleaned key text data is stored in combination with the web page title, the web page summary and the web page tag, including: performing data parsing on the target text data to obtain file data corresponding to the directory tree information of the storage space of the key text data; generating naming information of the target text data based on the web page title and the web page summary; and storing the target text data and the web page tag in the file data corresponding to the target number information based on the naming information.

[0012] According to one aspect of an embodiment of the present application, a data extraction device is provided, comprising: an extraction module, used to extract text information in a web page based on a preset data toolkit; a generation module, used to generate a data extraction interface based on preset scenario information if the text information is valid text information; an acquisition module, used to acquire key text data in the valid text based on the data extraction interface, the key text data including a web page title, a web page summary and a web page tag; and a storage module, used to clean the key text data and store the cleaned key text data in combination with the web page title, the web page summary and the web page tag.

[0013] According to one aspect of an embodiment of the present application, an electronic device is provided, comprising: one or more processors; a storage device for storing one or more programs, wherein when the one or more programs are executed by the one or more processors, the electronic device implements the data extraction method as described above.

[0014] According to one aspect of an embodiment of the present application, a computer-readable storage medium is provided, on which computer-readable instructions are stored. When the computer-readable instructions are executed by a processor of a computer, the computer executes the data extraction method as described above.

[0015] In the technical solution provided in the embodiments of the present application, a data extraction interface can be generated according to the needs of a preset scenario through a preset data toolkit from text information in a web page, after determining that the text information is valid text information, and then the key text data in the valid text data can be extracted according to the data extraction interface, so that the key text data can be cleaned and then the cleaned key text data can be stored in combination with the web page title, web page summary and web page tag of the web page, thereby avoiding spending a lot of time on writing extraction rules, and in this embodiment, the data collection technology is extensible through the preset scenario information, avoiding the use of a single applicable data extraction scenario.

[0016] It should be understood that the foregoing general description and the following detailed description are exemplary and explanatory only and are not restrictive of the present application. BRIEF DESCRIPTION OF THE DRAWINGS

[0017] The drawings herein are incorporated into the specification and constitute a part of the specification, showing embodiments consistent with the present application, and together with the specification, are used to explain the principles of the present application. Obviously, the drawings described below are only some embodiments of the present application, and for those of ordinary skill in the art, other drawings can be obtained based on these drawings without creative work. In the drawings:

[0018] Figure 1 is a schematic diagram of an implementation environment for extracting data from a web page, shown in an exemplary embodiment of the present application;

[0019] Figure 2 is a flow chart of a data extraction method shown in an exemplary embodiment of the present application;

[0020] Figure 3 is a flow chart of a data extraction method shown in another exemplary embodiment of the present application;

[0021] Figure 4 is a flow chart of a data extraction method shown in another exemplary embodiment of the present application;

[0022] Figure 5 is a flow chart of a data extraction method shown in another exemplary embodiment of the present application;

[0023] Figure 6 is a flow chart of a data extraction method shown in another exemplary embodiment of the present application;

[0024] Figure 7 is a flow chart of a data extraction method shown in another exemplary embodiment of the present application;

[0025] Figure 8 is a flow chart of a data extraction method shown in another exemplary embodiment of the present application;

[0026] Fig. 9 It is a schematic diagram of a brief process of extracting data from a web page in an exemplary application scenario;

[0027] Fig.10 is a block diagram of a data extraction device shown in an exemplary embodiment of the present application;

[0028] Fig.11 A schematic diagram of the structure of a computer system suitable for implementing an electronic device of an embodiment of the present application is shown. DETAILED DESCRIPTION

[0029] Here, exemplary embodiments will be described in detail, examples of which are shown in the accompanying drawings. When the following description refers to the drawings, unless otherwise indicated, the same numbers in different drawings represent the same or similar elements. The implementations described in the following exemplary embodiments do not represent all implementations consistent with the present application. Instead, they are only examples of devices and methods consistent with some aspects of the present application as detailed in the attached claims.

[0030] The block diagrams shown in the accompanying drawings are merely functional entities and do not necessarily correspond to physically independent entities. That is, these functional entities may be implemented in software form, or in one or more hardware modules or integrated circuits, or in different networks and / or processor devices and / or microcontroller devices.

[0031] The flowcharts shown in the accompanying drawings are only exemplary and do not necessarily include all the contents and operations / steps, nor must they be executed in the order described. For example, some operations / steps can be decomposed, and some operations / steps can be combined or partially combined, so the actual execution order may change according to actual conditions.

[0032] The term "multiple" as used in this application refers to two or more than two. "And / or" describes the relationship between related objects, indicating that three relationships can exist. For example, A and / or B can mean: A exists alone, A and B exist at the same time, and B exists alone. The character " / " generally indicates that the related objects are in an "or" relationship.

[0033] First of all, it should be noted that web data collection is a program or script that automatically crawls the World Wide Web information according to certain rules. It is usually written in programming languages ​​(such as Python, Java, C++, etc.) and uses network libraries (such as Python's requests library, Java's HttpClient library, etc.) to send HTTP requests and receive responses. The main features of web data collection include: Automation: It can automatically access websites and extract required information without human intervention; Efficiency: It can quickly crawl a large amount of web page data and improve data acquisition efficiency; Flexibility: It can customize collection strategies according to different needs, such as crawling according to keywords, links, time and other strategies.

[0034] The application scenarios of web data collection are very wide, including but not limited to the following aspects: Search engine: Search engine is one of the most important application scenarios of general network data collection. As the basic part of the search engine, network data collection automatically grabs data from the Internet and provides massive information for the search engine; Public opinion analysis and monitoring: Governments or enterprises automatically collect massive data from forum comments, online blogs, news media or Weibo and other websites through network data collection technology to discover and track public opinion hotspots; Aggregation platform: Some aggregation platforms use network data collection technology to collect product information on e-commerce platforms, and display it on their own platforms, providing horizontal data comparison services.

[0035] Figure 1 FIG. 1 is a schematic diagram of an implementation environment for data extraction on a web page according to an exemplary embodiment of the present application. Figure 1 As shown, a web page to be extracted through data acquisition technology is displayed on the terminal 110, and then the server 120 extracts text information in the web page through a preset data toolkit, and detects whether the extracted text information is valid text information. If the text information is valid text information, the server 120 generates a data extraction interface based on preset scenario information, and obtains key text data in the valid text information according to the data extraction interface, wherein the key text data includes web page titles, web page summaries, and web page tags, etc. Finally, the key text data is cleaned, and the cleaned key text data is stored in combination with its web page title, web page summary, and web page tags, thereby realizing rapid extraction and storage of web page data.

[0036] in, Figure 1 The terminal 110 shown can be any terminal device that supports the installation of navigation map software, such as a smart phone, a car computer, a tablet computer, a laptop computer, or a wearable device, but is not limited thereto. Figure 1The server 120 shown, for example, can be an independent physical server, or a server cluster or distributed system composed of multiple physical servers, or a cloud server that provides basic cloud computing services such as cloud services, cloud databases, cloud computing, cloud functions, cloud storage, network services, cloud communications, middleware services, domain name services, security services, CDN (Content Delivery Network), and big data and artificial intelligence platforms, etc., which are not limited here. The terminal 110 can communicate with the server 120 through wireless networks such as 3G (third generation mobile information technology), 4G (fourth generation mobile information technology), and 5G (fifth generation mobile information technology), which are not limited here.

[0037] There are many existing data collection frameworks, such as selenium, scrapy, requests, html2text and urllib.request library packages in Python, and WebMagic and other framework packages in Java. However, using these data collection framework packages requires users to customize web page extraction rules according to specific website scenarios, which places extremely high demands on the user's code and takes a lot of time to write extraction rules. In addition, the data collection code written is not scalable and is used in a single scenario.

[0038] It can be understood that in the specific implementation of the present application, it involves page data extraction, page data collection, data screening and other related data such as operation information on the web page. When the above embodiments of the present application are applied to specific products or technologies, it is necessary to obtain the permission or consent of the user, and the collection, extraction and use of relevant data need to comply with the relevant laws, regulations and standards of the relevant countries and regions.

[0039] The above-mentioned problems are generally applicable to general web page collection. To solve these problems, the embodiments of the present application respectively propose a data extraction method, a data extraction device, an electronic device, a computer-readable storage medium and a computer program product, which will be described in detail below.

[0040] See also Figure 2 , Figure 2 is a flowchart of a data extraction method shown in an exemplary embodiment of the present application. The method can be applied to Figure 1 The implementation environment shown is specifically executed by the server 120 in the implementation environment. It should be understood that the method can also be applied to other exemplary implementation environments and be specifically executed by devices in other implementation environments. This embodiment does not limit the implementation environment to which the method is applicable.

[0041] like Figure 2As shown, in an exemplary embodiment, a specific implementation method of the data extraction method may include steps S210 to S240, which are described in detail as follows:

[0042] Step S210: extracting text information from the web page based on a preset data toolkit.

[0043] Specifically, text information is usually extracted from web pages by using various tools and techniques, such as web scrapers, libraries (such as Python's requests and BeautifulSoup or Scrapy), APIs (if the website provides a data API), or integrated preset data toolkits.

[0044] For example, you can use the corresponding tools in the preset data toolkit to extract text information from a web page. If the web page has an anti-data collection mechanism, you need to process the cookies, headers, sessions, etc. of the web page, and select tags based on the specific HTML structure of the web page. For example, if you know that the text is located in a certain ID or class name, Tags, you can use soup.find('div',{'id':'some_id'}) or soup.find('div',{'class':'some_class'}) to locate it. If the website provides an API, then using the API is usually more stable and legal than directly crawling the web page. It should be noted that when conducting network data collection activities, please make sure to comply with the website's robots.txt file and relevant laws and regulations.

[0045] Step S220: If the text information is valid text information, a data extraction interface is generated based on preset scene information.

[0046] Specifically, after all text information in the webpage is extracted based on the preset data packet, a corresponding data extraction interface can be generated according to the scene information preset according to actual needs, wherein the data extraction interface is used for extracting and outputting data.

[0047] Exemplarily, the extracted text information is parsed and cleaned as necessary to ensure that it meets your data requirements, such as removing HTML tags, processing special characters, segmenting text, etc., and matching the cleaned data with the preset scene information. This may involve mapping the text information to preset data fields, or converting or filtering the data according to preset rules, and then generating a data extraction interface based on the matching results. This can be an API, JSON file, CSV file, or other format of data output, so it is necessary to ensure that the generated data interface conforms to the data format and structure defined in the preset scene information.

[0048] Step S230, obtaining key text data in the valid text information based on the data extraction interface, the key text data including the web page title, web page summary and web page tag.

[0049] Specifically, before executing step S230, it is also possible to detect whether the data extraction interface has a web page title extraction item, a web page summary extraction item, and a web page tag extraction item, etc. If the data extraction interface contains the above extraction items, the target data interface can be parsed, and the data extraction interface can be parsed using an appropriate programming language and data parsing library. Exemplarily, if a data interface in JSON format is used, a json library such as Python can be used for parsing, and key text data can be extracted from the parsed data structure, which usually involves accessing specific attributes or keys of the data structure.

[0050] In some feasible embodiments, these data can also be extracted from the web page by using network data collection and HTML parsing technology, and stored in the data extraction interface.

[0051] Step S240, cleaning the key text data, and storing the cleaned key text data in combination with the web page title, web page summary, and web page tag.

[0052] Specifically, data containing key text data, web page titles, web page summaries, and web page tags are read from a data extraction interface (such as JSON, a database, etc.), and then data cleaning is performed on the key text data, wherein data cleaning may include: removing HTML tags: if the key text data contains HTML tags, use an HTML parsing library (such as BeautifulSoup) or regular expressions to remove them; removing special characters and punctuation marks: removing or replacing special characters and punctuation marks as required; standardizing text: converting text to lowercase, removing extra spaces, unifying text encoding, etc.; processing null values ​​and duplicates: checking null values ​​and duplicates in the data, and processing them according to business logic (such as filling null values, deleting duplicates). The key text data after data cleaning is then stored in combination with web page titles, web page summaries, and web page tags. For example, the cleaned data is stored together with the web page titles, web page summaries, and web page tags in an appropriate storage medium, such as a database, a CSV file, a NoSQL database, etc.

[0053] In this embodiment, a preset data toolkit is used to obtain text information from a web page. After determining that the text information is valid text information, a data extraction interface can be generated according to the needs of a preset scenario, and then key text data in the valid text data can be extracted according to the data extraction interface. This allows the key text data to be cleaned and then stored in combination with the web page title, web page summary, and web page tag of the web page, thereby avoiding the need to spend a lot of time writing extraction rules. In this embodiment, the preset scenario information makes the data collection technology extensible, avoiding the use of a single scenario.

[0054] Further, based on the above embodiments, please refer to Figure 3 In one of the exemplary embodiments provided in the present application, the specific implementation process of extracting text information from a web page based on a preset data toolkit includes at least step S310 and step S320, which are described in detail as follows:

[0055] Step S310, parsing a preset data toolkit to obtain a multi-level data extraction model included in the preset data package;

[0056] Step S320: extracting text information from the web page based on at least one level of data extraction model in the multi-level data extraction model.

[0057] Specifically, the preset data toolkit includes a multi-level data extraction model, which can be created based on the same compilation language or based on different compilation languages. Then, by parsing the preset toolkit, the multi-level data extraction model in the preset toolkit is obtained, and then the text information in the web page is extracted according to at least one level of the data extraction model in the multi-level data extraction model.

[0058] Exemplarily, if the text content of the webpage cannot be completely extracted by the first-level data extraction model, the text content of the webpage can be extracted by the second-level data extraction model. If the text content of the webpage cannot be completely extracted by the second data extraction model, the text content of the webpage can be extracted by the third-level data extraction model, and so on. The first-level data extraction model, the second-level data extraction model, and the third-level data extraction model can be used alone or in combination. For example, if the three libraries of requests, html2text, and urllib.request are used. First try to use the requests library to obtain the text content of the webpage. If it fails, use the urllib.request library. After obtaining the text content, use the html2text library to convert it into plain text format.

[0059] In this embodiment, the webpage text content corresponding to the webpage is extracted by using a multi-level data extraction model in a preset data toolkit. On the one hand, the text content on the webpage is extracted more thoroughly, and on the other hand, the extraction of the webpage text content is made more flexible and effective.

[0060] Further, based on the above embodiments, please refer to Figure 4 In one of the exemplary embodiments provided in the present application, the specific implementation method of the above data extraction method may further include step S410 and step S420, which are described in detail as follows:

[0061] Step S410, converting the text information into text data corresponding to a plain text format;

[0062] Step S420: If the text data is complete text data, determine that the text data is valid text information.

[0063] Specifically, all text information in a web page is extracted based on a preset data toolkit, wherein all text information can be converted into text data of a corresponding format. For example, the conversion of text information into text data corresponding to a plain text format usually involves removing all rich text tags, HTML tags, formatting characters, etc., and only retaining the basic text content. Afterwards, verify whether the converted text data is complete text data, wherein the verification content includes: non-empty check: ensure that the text is not empty; length check: if the original text has an expected minimum or maximum length, it can be checked whether the converted text meets these conditions; content verification: if possible, a hash function or other checksum algorithm can be used to compare the original text and the converted text to ensure that the content has not changed. However, this method may not be applicable when converting text (such as from HTML to plain text) because the conversion may lose some information (such as format and tags); structure verification (if applicable): if the text should follow a specific structure (such as JSON, XML, etc.), the converted text can be parsed to verify whether its structure is correct; business logic verification: according to the specific business logic, check whether the text contains all necessary information or elements. Afterwards, if the text data is detected to be complete text data, it can be determined that the text data is valid text information.

[0064] In this embodiment, by converting text information into text data corresponding to a plain text format, and verifying that the text data is complete text data, the text information is determined to be valid text information, and the integrity is verified in a plain text format, so that the integrity verification result can be made more accurate.

[0065] Further, based on the above embodiments, please refer to Figure 5 In one of the exemplary embodiments provided in the present application, the specific execution process of generating the data extraction interface based on the preset scene information may also include step S510 and step S520, which are described in detail as follows:

[0066] Step S510, selecting a corresponding big data model based on a preset scenario, where the big data model is used to extract key text information corresponding to the preset scenario.

[0067] First of all, it should be noted that the big data model is an abstract concept used to describe how to process and parse data in a big data environment in order to extract valuable information and insights from the data. Big data models can involve multiple aspects such as data storage, processing, analysis, and visualization. Among them, big data models include data processing models: processing: such as Apache Hadoop and Spark Batch, used to process static or historical data; stream processing: such as Apache Flink, Apache Kafka Streams, and Apache Storm, used to process data streams in real time; Lambda architecture: combines the advantages of batch processing and stream processing, used to process real-time and batch data at the same time. Data analysis model: Data mining: using statistical and machine learning algorithms to discover patterns and trends from large data sets; machine learning: using algorithms to enable systems to learn and improve without explicit programming; predictive analysis: predicting future events or behaviors based on historical data.

[0068] Specifically, in a preset scenario, the corresponding big data model is selected to extract key text information. The specific needs, data types, and required analysis depth of the scenario need to be considered. For example, the corresponding big data model selected for social media monitoring and public opinion analysis performs the following process: Text information extraction: Use natural language processing (NLP) models, such as BERT, GPT, etc., to extract information from text data on social media, including keywords, topics, sentiment tendencies, etc.; Time series analysis: For dynamic data on social media, time series analysis models (such as ARIMA models) can be used to monitor and predict the popularity of topics.

[0069] Step S520: Generate a corresponding data extraction interface based on the big data model.

[0070] Specifically, when generating a corresponding data extraction interface based on a specific big data model, it is necessary to consider the function, input parameters, output format, and integration method of the data interface with other systems. For example, first define the interface requirements, for example, clarify the functions that the data interface needs to implement, such as query, analysis, extraction, etc., define the input parameters required by the interface, such as query conditions, time range, keywords, etc., and determine the format of the data returned by the interface, such as JSON, XML, CSV, etc. Then, design the interface structure, design the access URL of the interface, ensure that the URL can clearly express the function of the interface, select the appropriate HTTP request method, such as GET, POST, PUT, DELETE, etc., define the request header and request body structure required by the interface, including authentication information, parameter passing method, etc. Finally, implement the interface logic, verify the input parameters, ensure the validity and security of the parameters, call the corresponding big data model for processing according to the interface function, and format, filter, aggregate, and process the output data of the big data model.

[0071] In this embodiment, a big data model is selected through a preset scenario, and then the key text information corresponding to the preset scenario is extracted through the big data model. The big data model can then generate a corresponding data extraction interface based on the preset scenario information, thereby avoiding the tedious work of rule analysis and meeting the diverse scenario needs of different users.

[0072] Further, based on the above embodiment, please refer to Figure 6 In one of the exemplary embodiments provided in this application, the specific implementation process of the above data extraction method may also include steps S610 to S630, which are described in detail as follows:

[0073] Step S610: Generate a custom extraction rule based on the scene parameters corresponding to the preset scene.

[0074] Specifically, when generating custom extraction rules based on the scenario parameters corresponding to the preset scenario, it is necessary to clarify the specific requirements of the scenario, parameter types, and the structure of the target data. For example, understand the business background of the preset scenario, clarify which types of data need to be extracted from which data sources, determine the goals of the data to be extracted, such as whether it is for analysis, reporting, or other purposes, identify the types of scenario parameters, such as time range, geographic location, user behavior, etc., determine the format of the parameters, such as date format, string format, etc., and finally, name each extraction rule to make it descriptive.

[0075] Step S620, inputting the custom extraction rules and valid text information into the big data model.

[0076] Specifically, custom extraction rules are loaded into the data processing flow. These rules define how to extract key information from text information or perform specific data transformations. Use the extraction rules to process valid text information, which may involve conditional judgment, data screening, data aggregation and other operations to generate a data set that meets the input requirements of the big data model. Ensure that the extracted data conforms to the input format expected by the big data model. This may include converting the data into a specific data structure (such as JSON, CSV) or encoding method (such as UTF-8), and passing the formatted data to the big data model. This can be done through API calls, database queries, or other data transmission mechanisms, after which the big data model is triggered to run and the processed data set is passed in. The model will perform operations such as analysis, prediction, or classification based on the input data and internal algorithm logic.

[0077] Step S630, generating a corresponding data extraction interface based on the big data model, so as to extract key text data from the valid text information through the data extraction interface.

[0078] Specifically, after inputting the custom extraction rules into the big data model, the big data model can be OpenAI's ChatGPT, MetaGPT, Wenxin Yiyan, Tongyi Qianwen and other big models, then the extracted text content and the custom extraction rules can be used as prompt inputs of ChatGPT to obtain the extraction results. In this way, there is no need to spend a lot of effort to analyze the html elements of the page like traditional data collection methods.

[0079] For example, the rapid data collection and extraction of web page content based on large models such as ChatGPT is mainly divided into the following three steps:

[0080] #Step 1: Extract all text content from the web page

[0081] url='xxx'

[0082] flag,text=get_web_page_text(url)

[0083] #Step 2: Use the extraction results and custom extraction rules as the prompt of ChatGPT, input them into ChatGPT, and obtain the extraction results

[0084] if flag:

[0085] prompt=text+'\n\n'+'Please extract the article title, publishing time, and number of likes.'

[0086] response=get_LLM_reply(prompt)

[0087] #Step 3: Clean the response content and store it

[0088] data_process_and_store(response)

[0089] In this embodiment, the custom extraction rules corresponding to the scene and the valid text information of the web page are input into the big data model, and then the big data model generates a corresponding data interface to extract key text data from the valid text information through the data extraction interface, thereby reducing the tedious work of rule analysis on HTML elements in traditional collection methods and improving data collection efficiency and flexibility.

[0090] Further, based on the above embodiment, please refer to Figure 7 In one of the exemplary embodiments provided in the present application, the specific implementation process of data cleaning of the key text data may further include step S710 and step S720, which are described in detail as follows:

[0091] Step S710, if the key text data is complete text data, convert the key text data into preset character type data;

[0092] Step S720, performing deduplication processing on the preset character type data to obtain target text data.

[0093] Specifically, following the above embodiment, if the key text data extracted from the web page is complete text data after detection, the key text data can be converted into preset character type data, wherein the preset character type can be set according to the actual data extraction requirements, or can be determined according to the storage type corresponding to the text data. Then, the converted preset character type data is deduplicated to obtain the deduplicated target text data.

[0094] Exemplarily, data conversion is to ensure the consistency and processability of data. For example, if the original data contains multiple character encodings or data types (such as numbers, dates, strings, etc.), they may need to be uniformly converted into strings or specific character encodings, so the complete text data can be converted into data of a preset character type, and then, among them, duplicate data may cause misleading results or unnecessary computing overhead, and the converted preset character type data needs to be deduplicated to eliminate duplicates, thereby reducing data redundancy, making subsequent data analysis or processing more efficient and accurate.

[0095] Further, based on the above embodiment, please refer to Figure 8 In one of the exemplary embodiments provided in the present application, the specific implementation process of storing the cleaned key text data in combination with the web page title, web page summary and web page tag may also include steps S810 to S830, which are described in detail as follows:

[0096] Step S810, performing data analysis on the target text data to obtain file data corresponding to the directory tree information of the storage space of the key text data.

[0097] Specifically, use an appropriate programming language (such as Python, Java, etc.) to read the target text data. If the text data is large, consider using streaming reading or batch reading to improve efficiency. Select an appropriate parsing method based on the format of the text data (such as CSV, JSON, XML, custom format, etc.), and use regular expressions, string segmentation, and specialized parsing libraries (such as csv, json, xml.etree.ElementTree in Python, etc.) to parse the text. Extract the file path or directory information from the parsed data. Build a directory tree structure based on the file path. This can usually be achieved through a tree data structure (such as a tree object, nested dictionary, etc.). Store the extracted directory tree information and file data in an appropriate data structure (such as a list, dictionary, database, etc.).

[0098] Step S820: Generate naming information of target text data based on the web page title and web page summary.

[0099] Specifically, we extract titles and abstracts from web pages. Titles are usually located in <title> tags, while the abstract may be part of the meta description or other web page content. Clean the title and abstract, remove HTML tags, special characters, extra spaces and punctuation, etc., and convert the text to lowercase (optional) to ensure consistent naming. If the title is already short and contains all the key information, you can use it directly as the basis for the naming information; if the title is too long or not descriptive enough, consider extracting additional information from the abstract to supplement it, or combining the key information in the title and / or abstract to form a concise and descriptive name. Separators (such as hyphens -, underscores _, or spaces) can be used to separate different parts to ensure that the generated naming information is unique in your dataset or application. Check the readability of the naming information to ensure that it is easy to understand and identify.< / title>

[0100] Step S830: storing the target text data and the web page tag in the file data corresponding to the target number information based on the naming information.

[0101] Specifically, to store the target text data, web page tags, and previously generated naming information in the target file data, it is necessary to first determine the storage format and structure of the file data. Common storage formats include text files (such as CSV, JSON, XML, etc.), databases (such as SQLite, MySQL, MongoDB, etc.), or dedicated data storage systems (such as Elasticsearch, Hadoop, etc.).

[0102] In this embodiment, the target text data is parsed to obtain file data corresponding to the directory tree information of the storage space corresponding to the key text data, and then the naming information of the target text is generated according to the title and summary of the web page. The data naming method is more standardized and effective, and the storage of the target text data is more standardized.

[0103] Fig. 9 The following is a brief flow chart of data extraction from a web page in an exemplary application scenario. Fig. 9 In the application scenario shown, the text information in the web page is extracted based on the preset data toolkit; the preset data toolkit is parsed to obtain the multi-level data extraction model included in the preset data package; the text information in the web page is extracted based on at least one level of the multi-level data extraction model, and if the text information is valid text information, a data extraction interface is generated based on the preset scene information; a custom extraction rule is generated based on the scene parameters corresponding to the preset scene; the custom extraction rule and the valid text information are input into the big data model; a corresponding data extraction interface is generated based on the big data model to extract key text data from the valid text information through the data extraction interface. If the key text data is complete text data, the key text data is converted into preset character type data; the preset character type data is deduplicated to obtain the target text data, and the target text data and the web page tag are stored in the file data corresponding to the target number information based on the naming information. For the detailed implementation process, please refer to the records in the aforementioned embodiments, which will not be repeated here.

[0104] Fig.10 is a block diagram of a data extraction device shown in an exemplary embodiment of the present application. The device can be applied to Figure 1 The implementation environment shown in the figure is specifically configured in the server 120. The device can also be applied to other exemplary implementation environments and specifically configured in other devices. This embodiment does not limit the implementation environment to which the device is applied.

[0105] like Fig.10 As shown, the exemplary data extraction device 1000 includes: an extraction module 1010, which is used to extract text information from a web page based on a preset data toolkit; a generation module 1020, which is used to generate a data extraction interface based on preset scene information if the text information is valid text information; an acquisition module 1030, which is used to acquire key text data in the valid text information based on the data extraction interface, and the key text data includes a web page title, a web page summary, and a web page tag; a storage module 1040, which is used to clean the key text data and store the cleaned key text data in combination with the web page title, the web page summary, and the web page tag.

[0106] According to one aspect of an embodiment of the present application, the above-mentioned extraction module 1010 is also specifically used to parse a preset data toolkit to obtain a multi-level data extraction model included in the preset data package; and extract text information from the web page based on at least one level of the multi-level data extraction model.

[0107] According to one aspect of an embodiment of the present application, the above-mentioned data extraction device 1000 also includes: a conversion module, used to convert text information into text data corresponding to a plain text format; and a determination module, used to determine that the text data is valid text information if the text data is complete text data.

[0108] According to one aspect of an embodiment of the present application, the above-mentioned generation module 1020 is also used to select a corresponding big data model based on a preset scenario, and the big data model is used to extract key text information corresponding to the preset scenario; and generate a corresponding data extraction interface based on the big data model.

[0109] According to one aspect of an embodiment of the present application, the above-mentioned generation module 1020 is also used to generate custom extraction rules based on scene parameters corresponding to preset scenes; input the custom extraction rules and valid text information into the big data model; generate a corresponding data extraction interface based on the big data model to extract key text data from the valid text information through the data extraction interface.

[0110] According to one aspect of an embodiment of the present application, the storage module 1040 is also used to, if the key text data is complete text data, convert the key text data into preset character type data; perform deduplication processing on the preset character type data to obtain target text data.

[0111] According to one aspect of an embodiment of the present application, the above-mentioned storage module 1040 is also used to perform data analysis on the target text data to obtain file data corresponding to the directory tree information of the storage space of the key text data; generate naming information of the target text data based on the web page title and web page summary; and store the target text data and web page tags in the file data corresponding to the target number information based on the naming information.

[0112] It should be noted that the data extraction device provided in the above embodiment and the data extraction method provided in the above embodiment belong to the same concept, wherein the specific manner in which each module and unit performs the operation has been described in detail in the method embodiment, and will not be repeated here. In practical applications, the data extraction device provided in the above embodiment can distribute the above functions to different functional modules as needed, that is, divide the internal structure of the device into different functional modules to complete all or part of the functions described above, and this is not limited here.

[0113] An embodiment of the present application also provides an electronic device, comprising: one or more processors; a storage device for storing one or more programs, when the one or more programs are executed by the one or more processors, the electronic device implements the data extraction method provided in the above-mentioned embodiments.

[0114] Fig.11 The structure diagram of the computer system suitable for implementing the electronic device of the embodiment of the present application is shown. It should be noted that: Fig.11 The computer system 1100 of the electronic device shown is only an example and should not bring any limitation to the functions and scope of use of the embodiments of the present application.

[0115] like Fig.11 As shown, the computer system 1100 includes a central processing unit (CPU) 1101, which can perform various appropriate actions and processes according to the program stored in the read-only memory (ROM) 1102 or the program loaded from the storage part 1108 to the random access memory (RAM) 1103, such as executing the method described in the above embodiment. In the RAM 1103, various programs and data required for system operation are also stored. The CPU 1101, the ROM 1102 and the RAM 1103 are connected to each other through the bus 1104. The input / output (I / O) interface 1105 is also connected to the bus 1104.

[0116] The following components are connected to the I / O interface 1105: an input section 1106 including a keyboard, a mouse, etc.; an output section 1107 including a cathode ray tube (CRT), a liquid crystal display (LCD), etc., and a speaker, etc.; a storage section 1108 including a hard disk, etc.; and a communication section 1109 including a network interface card such as a LAN (Local Area Network) card, a modem, etc. The communication section 1109 performs communication processing via a network such as the Internet. A drive 1110 is also connected to the I / O interface 1105 as needed. A removable medium 1111, such as a magnetic disk, an optical disk, a magneto-optical disk, a semiconductor memory, etc., is installed on the drive 1110 as needed so that a computer program read therefrom is installed into the storage section 1108 as needed.

[0117] In particular, according to an embodiment of the present application, the process described above with reference to the flowchart can be implemented as a computer software program. For example, an embodiment of the present application includes a computer program product, which includes a computer program carried on a computer-readable medium, and the computer program includes a computer program for executing the method shown in the flowchart. In such an embodiment, the computer program can be downloaded and installed from a network through a communication section 1109, and / or installed from a removable medium 1111. When the computer program is executed by a central processing unit (CPU) 1101, various functions defined in the system of the present application are executed.

[0118] It should be noted that the computer-readable medium shown in the embodiment of the present application can be a computer-readable signal medium or a computer-readable storage medium or any combination of the above two. The computer-readable storage medium can be, for example, an electrical, magnetic, optical, electromagnetic, infrared, or semiconductor system, device or device, or any combination of the above. More specific examples of computer-readable storage media can include, but are not limited to: an electrical connection with one or more wires, a portable computer disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM), a flash memory, an optical fiber, a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the above. In the present application, a computer-readable signal medium may include a data signal propagated in a baseband or as part of a carrier wave, wherein a computer-readable computer program is carried. This propagated data signal can take a variety of forms, including but not limited to an electromagnetic signal, an optical signal, or any suitable combination of the above. A computer-readable signal medium may also be any computer-readable medium other than a computer-readable storage medium, which may send, propagate or transmit a program for use by or in conjunction with an instruction execution system, apparatus or device. A computer program contained on a computer-readable medium may be transmitted using any appropriate medium, including but not limited to: wireless, wired, etc., or any suitable combination of the above.

[0119] The flowchart and block diagram in the accompanying drawings illustrate the possible architecture, functions and operations of the system, method and computer program product according to various embodiments of the present application. Wherein, each box in the flowchart or block diagram can represent a module, a program segment, or a part of the code, and the above-mentioned module, program segment, or a part of the code contains one or more executable instructions for realizing the specified logical function. It should also be noted that in some alternative implementations, the functions marked in the box can also occur in a different order from the order marked in the accompanying drawings. For example, two boxes represented in succession can actually be executed substantially in parallel, and they can sometimes be executed in the opposite order, depending on the functions involved. It should also be noted that each box in the block diagram or flowchart, and the combination of boxes in the block diagram or flowchart can be implemented with a dedicated hardware-based system that performs a specified function or operation, or can be implemented with a combination of dedicated hardware and computer instructions.

[0120] The units involved in the embodiments described in this application may be implemented by software or hardware, and the units described may also be set in a processor. The names of these units do not, in some cases, constitute limitations on the units themselves.

[0121] Another aspect of the present application also provides a computer-readable storage medium on which a computer program is stored, and when the computer program is executed by a processor, the data extraction method as described above is implemented. The computer-readable storage medium may be included in the electronic device described in the above embodiment, or may exist independently without being assembled into the electronic device.

[0122] Another aspect of the present application also provides a computer program product or a computer program, which includes a computer instruction stored in a computer-readable storage medium. A processor of a computer device reads the computer instruction from the computer-readable storage medium, and the processor executes the computer instruction, so that the computer device executes the data extraction method provided in each of the above embodiments.

[0123] The above content is only a preferred exemplary embodiment of the present application and is not intended to limit the implementation scheme of the present application. A person skilled in the art can easily make corresponding changes or modifications based on the main concept and spirit of the present application. Therefore, the scope of protection of the present application shall be based on the scope of protection required by the claims.

Claims

1. A data extraction method, characterized in that: include: Extract text information from web pages based on a preset data toolkit; If the text information is valid text information, generating a data extraction interface based on preset scene information; Acquire key text data in the valid text information based on the data extraction interface, wherein the key text data includes a web page title, a web page summary, and a web page tag; The key text data is cleaned, and the cleaned key text data is combined with the web page title, the web page summary and the web page tag for storage.

2. The method according to claim 1, characterized in that The extracting text information from the web page based on the preset data toolkit includes: Parsing the preset data toolkit to obtain a multi-level data extraction model included in the preset data package; The text information in the web page is extracted based on at least one level of data extraction model in the multi-level data extraction model.

3. The method according to claim 2, characterized in that The method further comprises: Converting the text information into text data corresponding to a plain text format; If the text data is complete text data, it is determined that the text data is valid text information.

4. The method according to claim 1, characterized in that The generating of the data extraction interface based on the preset scene information includes: Selecting a corresponding big data model based on a preset scenario, wherein the big data model is used to extract key text information corresponding to the preset scenario; A corresponding data extraction interface is generated based on the big data model.

5. The method according to claim 4, characterized in that The method further comprises: Generate a custom extraction rule based on the scene parameters corresponding to the preset scene; Inputting the custom extraction rule and the valid text information into the big data model; A corresponding data extraction interface is generated based on the big data model, so as to extract key text data from the valid text information through the data extraction interface.

6. The method according to claim 1, characterized in that The step of performing data cleaning on the key text data includes: If the key text data is complete text data, converting the key text data into preset character type data; The preset character type data is deduplicated to obtain target text data.

7. The method according to claim 6, characterized in that The storing of the cleaned key text data in combination with the webpage title, the webpage summary and the webpage tag includes: Performing data parsing on the target text data to obtain file data corresponding to the directory tree information of the storage space of the key text data; Generate naming information of the target text data based on the webpage title and the webpage summary; Based on the naming information, the target text data and the web page tag are stored in the file data corresponding to the target number information.

8. A data extraction device, characterized in that: The device comprises: An extraction module, used to extract text information from a web page based on a preset data toolkit; A generating module, configured to generate a data extraction interface based on preset scene information if the text information is valid text information; An acquisition module, configured to acquire key text data in the valid text information based on the data extraction interface, wherein the key text data includes a web page title, a web page summary, and a web page tag; The storage module is used to clean the key text data and store the cleaned key text data in combination with the web page title, the web page summary and the web page tag.

9. An electronic device, characterized in that: include: one or more processors; A storage device for storing one or more programs, which, when executed by the one or more processors, enables the electronic device to implement the data extraction method as described in any one of claims 1 to 7.

10. A computer-readable storage medium, characterized in that: Computer-readable instructions are stored thereon, and when the computer-readable instructions are executed by a processor of a computer, the computer is caused to execute the data extraction method according to any one of claims 1 to 7.

Citation Information

Cited By

  • Intelligent data acquisition method and device, computer equipment and storage medium

    CN120994890A