Acquisition and format conversion method based on webpage report data

Abstract

The invention discloses a collection and format conversion method based on webpage report data, relates to the technical field of data processing, and mainly solves the problems of low data collection accuracy, low conversion efficiency and poor data protectiveness. The collection and format conversion method based on the webpage report data comprises the implementation steps of collecting the webpage report data; processing and analyzing the data information; storing and backing up the data subjected to format conversion; converting the processed data into a format which can be checked by a social tool; security and privacy protection is carried out on data transmission; the webpage report data is accurately extracted through the data acquisition module, so that the accuracy of the webpage report data is ensured; the data is subjected to identification and format conversion through a deep learning model and a Python library, so that the conversion efficiency of the data is improved; the data is protected through the RSA encryption algorithm, the electronic password and the desensitization processing, so that the security of the data is improved.

Description

Technical Field

[0001] The present invention relates to the technical field of data processing, and in particular to a method for collecting and formatting web page report data. Background Art

[0002] In the insurance industry, a large amount of data needs to be processed, and the types are diverse, including customer information, transaction records, claims information, etc. At the same time, the report formats between different departments are not unified, which increases the difficulty of data integration and analysis. With the continuous updating and iteration of technology, more intelligent report data collection and format conversion methods have greatly improved the intelligence of report data collection and format unification. However, the traditional web report data collection and format conversion methods still have problems such as low data collection accuracy, low conversion efficiency and poor data protection.

[0003] Therefore, the present invention provides a method for collecting and converting the format of web page report data, which accurately extracts the web page report data through the data collection module, thereby ensuring the accuracy of the web page report data; identifies and converts the format of the data through the deep learning model and Python library, thereby improving the data conversion efficiency; and protects the data through the RSA encryption algorithm, electronic password and desensitization processing, thereby improving the security of the data. Summary of the invention

[0004] The purpose of the present invention is to solve the problems of low data collection accuracy, low conversion efficiency and poor data protection in the prior art, and to propose a method for collecting and formatting report data based on web page reports.

[0005] In order to achieve the above object, the present invention adopts the following technical solutions:

[0006] A method for collecting and converting web page report data, comprising the following steps:

[0007] S1. Collecting web page report data through the report address, wherein the web page report data includes text, numbers and images;

[0008] S2. Process and analyze data information to improve data quality;

[0009] S3, storing and backing up the data after format conversion;

[0010] S4. Convert the processed data into a format that can be viewed by social tools, including pictures, text, Excel and PPT, and transfer the data to external systems;

[0011] S5. Ensure security and privacy protection for data transmission;

[0012] The method steps include a data acquisition module, a data processing module, a data management module, a conversion and import module and a security protection module.

[0013] As a further description of the above technical solution, the data acquisition module is used to collect and read web page report data. After the data acquisition is completed, the original web page content is passed as input to the data processing module; the data processing module is used to parse the web page structure to extract the required data, and store the parsed data in a structured form in the data management module; the data management module is used to store and back up the preliminarily processed data, and export the data into a specific format for the conversion and import module to perform format conversion; the conversion and import module is used to convert the format of the data and transmit the data to an external system. During the data transmission to the external system, the security protection module is used to protect the security and privacy of the data transmission.

[0014] As a further description of the above technical solution, the data acquisition module receives user input through a web server to initialize the data acquisition process; sends an HTTP request to the web page through the Python Requests library to load the HTML page to obtain the web page report content; at the same time, DOM parsing is used to analyze the page structure to distinguish between the report area and the non-report area content; report data is extracted from the located report area through regular expression matching to obtain sales business data in the report, and the sales business data includes financial data, business data, risk management data, market and customer data, and operational efficiency data.

[0015] As a further description of the above technical solution, the financial data includes premium income, claims data and sales profit; the business data includes the number of new contracts, renewal rate and number of claims cases; the risk management data includes risk reserves, compliance audit records and legal proceedings; the market and customer data includes customer satisfaction, product sales distribution and market share; the operational efficiency data includes return on investment, expense ratio and personnel effectiveness.

[0016] As a further description of the above technical solution, the data processing module includes data cleaning, data type integration, data analysis and data enhancement. The workflow of the data processing module is: identify and remove noise, redundancy and irrelevant information in the data through natural language processing (NLP); after cleaning the data, use ETL to integrate data types to unify the data format; based on the integrated data, analyze the data through big data and statistics to extract patterns and development trends in the company's business data; based on the analysis results, improve data quality through data interpolation, synthetic sample generation and feature engineering.

[0017] As a further description of the above technical solution, the data management module includes a data storage unit, a data retrieval unit and a backup and recovery unit. The data storage unit uses MySQL to store report data; the data retrieval unit quickly locates and extracts required data information through hash indexes; the backup and recovery unit regularly performs data backup through Alibaba Cloud OSS cloud storage, the output end of the data storage unit is connected to the input end of the data retrieval unit to achieve rapid retrieval of data, and the output end of the data retrieval unit is connected to the input end of the backup and recovery unit to achieve data backup. At the same time, when data loss is encountered during the retrieval process, the backup and recovery unit immediately intervenes and uses the backup data for recovery.

[0018] As a further description of the above technical solution, the conversion and import module includes data format recognition and conversion, data encapsulation and packaging, data import and data reprocessing. It automatically identifies the structure and fields of web report data through a deep learning model, uses pandas in the Python library to crawl data, converts the data into a structure such as DataFrame, and then uses pandas' to_excel and to_ppt functions to convert the data into Excel or PPT format; packages the format-converted data in the form of a compressed package to facilitate data transmission; uses an API call import function to import the packaged data into a target system, and the target system includes Excel and PPT; after importing the data into Excel and PPT, uses the target system's own functions to perform secondary processing on the data, and the target system's own functions include VBA macros, Python-PPTX library, adding annotations and filtering effects to achieve display in social tools.

[0019] As a further description of the above technical solution, the security protection module encrypts, stores and transmits data through the RSA encryption algorithm; uses electronic passwords to authenticate business personnel to ensure secure access to data; and desensitizes personal information and financial records to protect data privacy.

[0020] Other advantages, objectives and features of the present invention will be described in part in the following description; and in part, will be apparent to those skilled in the art based on an examination of the following; or, may be taught from the practice of the present invention. BRIEF DESCRIPTION OF THE DRAWINGS

[0021] In order to understand and grasp the technical solution more intuitively and clearly, when describing the embodiments of the present invention or the prior art, drawings are often used to supplement and illustrate. It should be noted that the drawings are only one way of expressing the embodiments of the present invention or the prior art. In fact, the technical solution can also have other implementation methods and changes, which all fall within the protection scope of the present invention. Therefore, technicians can design other drawings as needed to implement the technical solution of the present invention, among which,

[0022] Figure 1 The figure is a schematic diagram of the steps of implementing the process of the present invention.

[0023] Figure 2 It is a schematic diagram of the structure of the financial data of the present invention.

[0024] Figure 3 It is a structural schematic diagram of the data processing module of the present invention.

[0025] Figure 4 It is a structural schematic diagram of the data management module of the present invention.

[0026] Figure 5 It is a structural schematic diagram of the conversion import module of the present invention. DETAILED DESCRIPTION

[0027] The following will clearly and completely describe the technical solutions in the embodiments of this invention through the accompanying drawings in the embodiments of this invention. Obviously, the described embodiments are only part of the embodiments of this invention, not all of the embodiments. At the same time, the description of well-known structures and technologies is omitted in the following description to avoid unnecessary confusion of the concepts of the present invention.

[0028] like Figure 1-Figure 5 As shown, a method for collecting and converting web report data includes the following steps:

[0029] S1. Collecting web page report data through the report address, wherein the web page report data includes text, numbers and images;

[0030] S2. Process and analyze data information to improve data quality;

[0031] S3, storing and backing up the data after format conversion;

[0032] S4. Convert the processed data into a format that can be viewed by social tools, including pictures, text, Excel and PPT, and transfer the data to external systems;

[0033] S5. Ensure security and privacy protection for data transmission;

[0034] The method steps include a data acquisition module, a data processing module, a data management module, a conversion and import module and a security protection module.

[0035] In a specific embodiment, the data acquisition module uses Python's BeautifulSoup, Scrapy and other libraries to capture web page report data. After collecting and reading data, the original web page content is passed as input to the data processing module; the data processing module natural language processing NLP identifies and removes noise, redundancy and irrelevant information in the data, parses the web page structure, and extracts required data, such as tables, text, pictures, etc.; the data management module is responsible for classifying, storing, backing up and verifying the security of the preliminarily processed data to ensure the security and accessibility of the data. At the same time, it converts the data into a list data format to meet the subsequent conversion and import requirements, and realizes the efficient flow and compatibility of data; the conversion and import module is responsible for converting structured data into the format required by the external system, using pandas' to_excel, to_json and other functions to convert the data into the target format, and transmits the data to the external system through the API, cooperating with the security protection module to ensure the security of the data transmission process; the security protection module ensures that the security and privacy of the data during the transmission process are not leaked, and protects the final application effect of the data and the overall stability of the system.

[0036] In a specific embodiment, the data collection module receives user input through a web server and initializes the data collection process. With the Python Requests library as the core, it sends precise HTTP requests to the target web page and loads the complete HTML page in order to obtain the web report content. In this process, DOM parsing accurately analyzes the page structure and quickly distinguishes the report area from the non-report area, ensuring that each capture can effectively collect the target data. Then, regular expressions are used to accurately extract sales business data from the report area. The sales business data classification is shown in Table 1:

[0037] Table 1 Classification of sales business data

[0038] Data Types Data Details Financial Data Premium income, claims data and sales profit Business data Number of new policies, renewal rate and number of claims Risk management data Risk reserves, compliance audit records and legal proceedings Market and customer data Customer satisfaction, product sales distribution and market share Operational efficiency data Return on investment, expense ratio and staff effectiveness

[0039] In summary, this process ensures the efficient and accurate extraction of required sales business data from web pages, laying a solid foundation for subsequent data processing and analysis.

[0040] In a specific embodiment, the data processing module includes data cleaning, data type integration, data analysis and data enhancement; the data cleaning is the starting point of the entire process, and the natural language processing NLP accurately identifies and removes noise, redundant information and irrelevant impurities in the data to improve the purity and availability of the data, laying a solid foundation for subsequent data processing. In this process, NLP technology can not only remove obvious errors and outliers, but also effectively eliminate those seemingly reasonable but actually meaningless information by understanding the context, ensuring the authenticity and accuracy of the data; after data cleaning, the data type integration standardizes the format of the cleaned raw data through the ETL extraction, conversion and loading process to achieve the unification of data types. It solves the problem of different formats caused by the diversity of data sources, and facilitates the smooth progress of subsequent processing and analysis. ETL not only effectively integrates diversified data sources, but also optimizes the data storage structure and improves data processing efficiency; after cleaning and integrating the data, the data analysis is based on the integrated high-quality data, using big data technology and statistical methods to deeply explore the hidden patterns and development trends in the enterprise business data, identify the key factors affecting the business, and provide strong data support for decision makers. At the same time, the data analysis can also reveal potential opportunities and risks in the market to help companies adjust their strategies; after processing and analyzing the data, the data enhancement further improves the data quality, fills in missing values ​​through data interpolation technology, and ensures data integrity; generates rich data sets with the help of synthetic samples, implements feature engineering, refines key features, and optimizes model performance. The data processing module makes the data processing results closer to business needs and provides more powerful support for the company's data analysis and decision support.

[0041] In a specific embodiment, the data management module includes a data storage unit, a data retrieval unit and a backup and recovery unit. The data storage unit adopts MySQL database technology to ensure that the report data can not only be effectively stored, but also query efficiency and data consistency can be guaranteed in massive data; the data retrieval unit relies on the hash index, an efficient indexing mechanism, to calculate the hash value of the key field of the data and map it to the position in the table, thereby achieving nearly instantaneous data retrieval, and can quickly locate and extract the required data information, greatly improving the data access speed and user experience, greatly reducing the delay of data query, and ensuring the smoothness of business operations; and the backup and recovery unit uses the Alibaba Cloud OSS cloud storage platform to implement regular data backup strategies to ensure data security and continuity. The output end of the data storage unit is directly connected to the input end of the data retrieval unit, realizing seamless connection from data storage to retrieval; the output end of the data retrieval unit is connected to the input end of the backup and recovery unit. Once an unexpected situation such as data loss is encountered during the retrieval process, the backup and recovery unit is immediately started and seamlessly restored using the backup data to ensure that business continuity is not affected.

[0042] In a specific embodiment, the conversion and import module includes data format recognition and conversion, data encapsulation and packaging, data import and data reprocessing. The data format recognition and conversion adopts an advanced deep learning model, which can automatically identify the complex structure and field type of web report data and accurately capture the essence of data. Through pandas in the Python library, the processed and analyzed web report data is organized into a well DataFrame structure. Subsequently, using the to_excel and to_ppt functions of pandas, the data can be easily converted into Excel and PPT formats to meet the diversified needs of data display; the data encapsulation and packaging uses a compressed package to package the converted data, which greatly improves the efficiency and security of data transmission, ensures the integrity and confidentiality of data during transmission, simplifies the transmission process, and improves efficiency; the data import imports the packaged data into the target system through API calls, including Excel and PPT; after importing the data, the data reprocessing uses the built-in functions of Excel and PPT, including VBA macros, Python-PPTX libraries, adding annotations and filtering effects, etc., to process and optimize the data. The VBA macro realizes automatic adaptive processing of data, making complex data processing tasks simple and fast; the Python-PPTX library makes the PPT data display more beautiful and improves the aesthetics of the battle report; the annotation function facilitates data interpretation; the filtering effect makes the data presentation more intuitive and clear, and finally generates files, texts, Excel tables and PPT screenshots that can be sent to social tools.

[0043] In a specific embodiment, the security protection module includes an RSA encryption algorithm, an electronic password and a desensitization process. The RSA encryption algorithm performs deep encryption storage and secure transmission of sensitive data to ensure that the data is not spied on or tampered with by unauthorized visitors at any stage. The electronic password acts as a second layer of security barrier to perform multiple identity authentication for business personnel, combined with biometric technology, including fingerprint and facial recognition, and dynamic password means to ensure that only personnel with strict identity confirmation can access sensitive data, significantly improving the security level of data access. For highly sensitive data such as personal information and financial records, the security protection module implements desensitization processing measures. By masking or replacing key fields, the data maintains its functionality and usability while avoiding the leakage of any information that may be used for personal identification, effectively protecting the privacy and security of the data, allowing users to enjoy convenient services without worrying about the risk of user information being abused.

[0044] The above description is only a preferred specific implementation manner of the present invention, but the protection scope of the present invention is not limited thereto. Any technician familiar with the technical field can make equivalent replacements or changes according to the technical scheme and inventive concept of the present invention within the technical scope disclosed by the present invention, which should be covered by the protection scope of the present invention.

Claims

1. A method for collecting and converting web page report data, characterized in that: The steps include: S1. Collecting web page report data through the report address, wherein the web page report data includes text, numbers and images; S2. Process and analyze data information to improve data quality; S3, storing and backing up the data after format conversion; S4. Convert the processed data into a format that can be viewed by social tools, including pictures, text, Excel and PPT, and transfer the data to external systems; S5. Ensure security and privacy protection for data transmission; The method steps include a data acquisition module, a data processing module, a data management module, a conversion and import module and a security protection module.

2. A method for collecting and converting web page report data according to claim 1, characterized in that: The data acquisition module is used to collect and read web page report data, and after data acquisition is completed, the original web page content is passed as input to the data processing module; the data processing module is used to parse the web page structure to extract the required data, and store the parsed data in a structured form in the data management module; the data management module is used to store and back up the initially processed data, and export the data into a specific format for the conversion import module to perform format conversion; The conversion import module is used to convert the format of the data and transmit the data to the external system. During the data transmission to the external system, the security protection module is used to protect the security and privacy of the data transmission.

3. The method for collecting and converting web page report data according to claim 1, characterized in that: The data collection module receives user input through the web server and initializes the data collection process; sends HTTP requests to the web page through the PythonRequests library, loads the HTML page to obtain the web report content; and uses DOM parsing to analyze the page structure to distinguish between the report area and the non-report area content; Report data is extracted from the located report area through regular expression matching to obtain sales business data in the report, wherein the sales business data includes financial data, business data, risk management data, market and customer data, and operational efficiency data.

4. The method for collecting and converting web page report data according to claim 3, characterized in that: The financial data includes premium income, claims data and sales profit; the business data includes the number of new contracts, renewal rate and number of claims cases; the risk management data includes risk reserves, compliance audit records and legal proceedings; the market and customer data includes customer satisfaction, product sales distribution and market share; the operational efficiency data includes return on investment, expense ratio and personnel effectiveness.

5. The method for collecting and converting web page report data according to claim 1, characterized in that: The data processing module includes data cleaning, data type integration, data analysis and data enhancement. The workflow of the data processing module is: identify and remove noise, redundancy and irrelevant information in the data through natural language processing (NLP); after cleaning the data, use ETL to integrate data types to unify the data format; Based on the integrated data, analyze the data through big data and statistics to extract patterns and development trends in the company's business data; Based on the analysis results, improve data quality through data interpolation, synthetic sample generation and feature engineering.

6. The method for collecting and converting web page report data according to claim 1, characterized in that: The data management module includes a data storage unit, a data retrieval unit and a backup and recovery unit. The data storage unit uses MySQL to store report data; the data retrieval unit quickly locates and extracts required data information through hash indexes; the backup and recovery unit regularly performs data backup through Alibaba Cloud OSS cloud storage, the output end of the data storage unit is connected to the input end of the data retrieval unit to achieve rapid retrieval of data, and the output end of the data retrieval unit is connected to the input end of the backup and recovery unit to achieve data backup. At the same time, when data loss is encountered during the retrieval process, the backup and recovery unit immediately intervenes and uses the backup data for recovery.

7. The method for collecting and converting web page report data according to claim 1, characterized in that: The conversion and import module includes data format recognition and conversion, data encapsulation and packaging, data import and data reprocessing. It automatically recognizes the structure and fields of web report data through a deep learning model, uses pandas in the Python library to crawl data, converts the data into a structure such as DataFrame, and then uses pandas' to_excel and to_ppt functions to convert the data into Excel or PPT format; packages the format-converted data in the form of a compressed package to facilitate data transmission; uses an API call import function to import the packaged data into a target system, which includes Excel and PPT; after importing the data into Excel and PPT, uses the target system's built-in functions to perform secondary processing on the data, which includes VBA macros, Python-PPTX library, adding annotations and filtering effects to achieve display in social tools.

8. The method for collecting and converting web page report data according to claim 1, characterized in that: The security protection module encrypts, stores and transmits data through the RSA encryption algorithm; uses electronic passwords to authenticate business personnel to ensure secure access to data; and desensitizes personal information and financial records to protect data privacy.

Images