Method for real-time collection and manual analysis of webpage data without fixed format based on plug-in

Abstract

The invention discloses a plug-in-based real-time acquisition and manual analysis method for webpage data without a fixed format. Real-time integrated operation of data acquisition, labeling and uploading is realized through a browser plug-in. A user selects target data through a floating window interface, and a plug-in automatically extracts content, identifies a platform source, performs quick labeling in combination with a preset analysis label and uploads the data to a background. According to the method, the flexibility of manual judgment and the high efficiency of an automation technology are fused, and the method can adapt to multiple platforms including high-protection dynamic webpages without developing a customized tool for a target website. The floating window supports operations such as dragging and zooming, so that the user experience is improved; the labeling is completed by data acquisition, so that the processing chain is effectively shortened, and the efficiency and accuracy are remarkably improved. The system supports authority management, encrypted transmission and real-time statistical analysis, and ensures data security and efficient utilization.

Description

Technical Field

[0001] The present invention relates to the technical field of e-commerce and data analysis, and in particular to a plug-in-based real-time collection and manual analysis method for web page data without a fixed format. Background Art With the rapid development of Internet technology, data collection and analysis are playing an increasingly important role in e-commerce, social media, market research and other fields. Especially in e-commerce platforms and social networks, web data without fixed formats (such as user comments, product information, behavior records, etc.) are widely considered to be important resources for studying user preferences and optimizing market strategies. However, these data are often scattered in different web pages and platforms in an unstructured form, which brings significant challenges to data collection.

[0002] In the prior art, the following two main methods are usually used to collect data from web pages with no fixed format: Automated collection tools (such as crawler technology): This method crawls web page content in a programmatic manner, parses and extracts target data. This method is more efficient when facing websites with a unified format and strong openness, but there are the following problems: 1. Poor applicability: Crawler tools are difficult to work effectively on dynamic web pages and protected websites (such as login restrictions and anti-crawling mechanisms).

[0003] 2. High development cost: Each target website usually needs to develop a collection script separately. When the structure of the target website changes, the script needs to be frequently adjusted and maintained.

[0004] 3. Legal and compliance risks: Crawler technology may involve unauthorized data capture, which can easily lead to legal disputes.

[0005] Manual data collection, when crawler tools cannot be used, is often done by manual data collectors who browse web pages manually and record target data. Although this method can cover niche or protected pages, it also relies entirely on manual operations, has a slow collection speed, and consumes a lot of time and manpower costs; and when manually inputting data, omissions, duplications, or input errors are prone to occur, making it difficult to ensure data quality. In addition, since data is collected and then handed over to analysts for classification and labeling, analysis delays occur, making it difficult to meet real-time data needs.

[0006] Therefore, how to improve collection efficiency while reducing operation complexity has become a technical problem to be solved by the present invention. Summary of the invention

[0007] The technical problem solved by the present invention is to provide a plug-in-based real-time collection and manual analysis method for web page data without a fixed format, in response to the defects existing in the above-mentioned prior art, so as to solve the problems of poor applicability, complex operation and disconnection between collection and analysis proposed in the above-mentioned background technology.

[0008] In order to solve the above technical problems, the technical solution adopted by the present invention is as follows: A plug-in-based real-time collection and manual analysis method for non-fixed format web page data, the method comprising the following steps: Step 1, preset collection content and analysis tags: preset collection content in the data analysis system, including collection object, user name, collection platform, analysis object and its associated analysis tags, where the analysis tags are used to mark the classification and purpose of the collected data; Step 2, install the browser plug-in: Install the plug-in for collecting data in the browser of the data collector. The plug-in can be presented on the web page in the form of a floating window and supports dragging, zooming and collapsing; Step 3, data collection trigger: select the non-fixed format data in the target web page through the floating window interface of the browser plug-in. The plug-in automatically captures the selected web page data, extracts the data content through the built-in parsing module, and automatically identifies the data source platform; Step 4, real-time analysis and annotation: In the plug-in floating window interface, the collector classifies and annotates the selected web page data according to the preset analysis tags, and the plug-in generates complete form data according to the annotated information; Step 5, data upload: the form data is uploaded to the backend server of the data analysis system through the communication module of the plug-in; Step 6, data display and subsequent processing: The uploaded data is presented in the data analysis system in various ways, including data statistics, deep learning model training, visualization display and user portrait analysis.

[0009] As a further solution of the present invention, the floating window interface of the browser plug-in can adjust the interface display mode according to the browsing needs of the collector, including minimization, floating display and hidden states to ensure normal web browsing.

[0010] As a further solution of the present invention, the automatic parsing module of the plug-in includes the following functions: automatically extracting the content of the selected data and filling it into the plug-in form; automatically identifying the URL address of the current web page and recording it as platform source information in the form.

[0011] As a further solution of the present invention, the analysis tags are preset by the administrator of the data analysis system in the background system, and the collectors select the analysis tags through the plug-in interface. The analysis tags may include but are not limited to user classification tags, product feature tags and behavior description tags.

[0012] As a further solution of the present invention, the form data is attached with timestamp information when it is uploaded, so as to record the time point of the collection operation to support subsequent data sorting and time series analysis.

[0013] As a further solution of the present invention, the plug-in supports shortcut key operation, and the data collector can quickly call up the plug-in floating window through the shortcut key to perform data collection operations.

[0014] As a further solution of the present invention, the data analysis system provides a data authority management function, and the uploaded data of different collectors can be allocated to different analysis modules according to preset authorities to ensure data security and targeted use.

[0015] As a further solution of the present invention, the data analysis system supports real-time statistical analysis of the uploaded data and presents the distribution, trend and correlation of the data in the form of charts.

[0016] As a further solution of the present invention, the floating window interface of the browser plug-in supports a language switching function, and the operation interface language of the plug-in can be switched according to the language environment of the acquisition platform.

[0017] As a further solution of the present invention, the plug-in supports uploading the collected web page data to the server of the data analysis system through an encrypted transmission protocol to ensure the security of the data transmission process.

[0018] Compared with the prior art, the present invention has the following beneficial effects: 1. Through the browser plug-in, real-time operations of data collection, analysis and annotation are realized, which deeply combines the flexibility of manual work and the efficiency of automation, making up for the limitations of existing fully automated tools in high protection mechanisms and complex dynamic web pages, while avoiding the low efficiency and error-proneness of fully manual collection, and significantly improving the accuracy and applicability of data collection.

[0019] 2. The real-time labeling of analysis tags is completed during the data collection process, and the step of "post-collection processing" in traditional technology is pre-placed as "collection and analysis", which effectively shortens the data processing chain, realizes the rapid flow and efficient use of data, and provides timely and accurate data support for subsequent statistical analysis and intelligent applications. The floating window interface of the plug-in supports dragging, zooming, hiding and other operations to avoid interfering with users' normal browsing of web pages. At the same time, by automatically extracting data content and identifying platform source information, the user's operating burden is reduced. The introduction of shortcut key function further simplifies the collection process and provides users with an efficient and smooth operating experience.

[0020] 3. Through the preset analysis tags of the background system and the real-time annotation function of the plug-in, multi-dimensional and multi-classification data collection is realized. This annotation method not only reduces the complexity of manual input, but also ensures the consistency of the annotation of the collected data, laying a high-quality data foundation for the subsequent deep learning model training and user portrait analysis. In addition, the plug-in does not need to be complexly developed for specific websites, and can be adapted to multiple platforms. Whether it is a public platform or a highly private, niche website, efficient data collection can be achieved through the combination of manual assistance and automated collection.

[0021] 4. The encrypted transmission protocol ensures that data is protected from theft and tampering during the upload process. At the same time, the permission management function ensures the accurate allocation of data access scope and usage scenarios for different collectors, which improves the efficiency of data collection while fully ensuring the security and privacy of the data. Through timestamp recording, real-time statistical analysis and visual display, the data collected by the present invention can more efficiently serve trend analysis, user profiling and deep learning model construction, providing users with full-chain technical support from data collection to intelligent analysis.

[0022] Additional aspects and advantages of the present invention will be given in part in the following description and in part will be obvious from the following description, or will be learned through practice of the present invention. BRIEF DESCRIPTION OF THE DRAWINGS

[0023] In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative labor.

[0024] Figure 1 It is a flow chart of the present invention. DETAILED DESCRIPTION

[0025] The technical solutions in the embodiments of the present invention are described clearly and completely below. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of them. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0026] See also Figure 1 In an embodiment of the present invention, a plug-in-based real-time collection and manual analysis method for non-fixed format web page data includes the following steps: Step 1, preset collection content and analysis tags: preset collection content in the data analysis system, including collection object, user name, collection platform, analysis object and its associated analysis tags, where the analysis tags are used to mark the classification and purpose of the collected data; Step 2, install the browser plug-in: Install the plug-in for collecting data in the browser of the data collector. The plug-in can be presented on the web page in the form of a floating window and supports dragging, zooming and collapsing; Step 3, data collection trigger: select the non-fixed format data in the target web page through the floating window interface of the browser plug-in. The plug-in automatically captures the selected web page data, extracts the data content through the built-in parsing module, and automatically identifies the data source platform; Step 4, real-time analysis and annotation: In the plug-in floating window interface, the collector classifies and annotates the selected web page data according to the preset analysis tags, and the plug-in generates complete form data according to the annotated information; Step 5, data upload: the form data is uploaded to the backend server of the data analysis system through the communication module of the plug-in; Step 6, data display and subsequent processing: The uploaded data is presented in the data analysis system in various ways, including data statistics, deep learning model training, visualization display and user portrait analysis.

[0027] As a further solution of the present invention, the floating window interface of the browser plug-in can adjust the interface display mode according to the browsing needs of the collector, including minimization, floating display and hidden states to ensure normal web browsing.

[0028] As a further solution of the present invention, the automatic parsing module of the plug-in includes the following functions: automatically extracting the content of the selected data and filling it into the plug-in form; automatically identifying the URL address of the current web page and recording it as platform source information in the form.

[0029] As a further solution of the present invention, the analysis tags are preset by the administrator of the data analysis system in the background system, and the collectors select the analysis tags through the plug-in interface. The analysis tags may include but are not limited to user classification tags, product feature tags and behavior description tags.

[0030] As a further solution of the present invention, the form data is attached with timestamp information when it is uploaded, so as to record the time point of the collection operation to support subsequent data sorting and time series analysis.

[0031] As a further solution of the present invention, the plug-in supports shortcut key operation, and the data collector can quickly call up the plug-in floating window through the shortcut key to perform data collection operations.

[0032] As a further solution of the present invention, the data analysis system provides a data authority management function, and the uploaded data of different collectors can be allocated to different analysis modules according to preset authorities to ensure data security and targeted use.

[0033] As a further solution of the present invention, the data analysis system supports real-time statistical analysis of the uploaded data and presents the distribution, trend and correlation of the data in the form of charts.

[0034] As a further solution of the present invention, the floating window interface of the browser plug-in supports a language switching function, and the operation interface language of the plug-in can be switched according to the language environment of the acquisition platform.

[0035] As a further solution of the present invention, the plug-in supports uploading the collected web page data to the server of the data analysis system through an encrypted transmission protocol to ensure the security of the data transmission process.

[0036] Embodiment 1: This embodiment provides a specific application scenario of a plug-in-based method for real-time collection and manual analysis of non-fixed format web page data. The following describes in detail the implementation method and implementation effect of this technical solution in conjunction with the collection and analysis of user comment data in an e-commerce platform.

[0037] In e-commerce platforms, user review data is a high-value unstructured data that contains users' evaluations of product quality, logistics services, and purchasing experience. This information is of great significance to merchants in optimizing product design and adjusting sales strategies. However, this type of data is scattered across different product pages and has no fixed format, making it difficult to collect. Although traditional technologies (such as crawler technology) can capture some data on some public platforms, crawler technology often fails when facing dynamically loaded pages or highly protected platforms (such as those that require login to access), and may cause legal and compliance issues; while manual collection has the problems of low efficiency and low data quality. Therefore, a technical solution that can efficiently and flexibly collect web page data with no fixed format is needed to solve the above problems.

[0038] In actual operation, the data analyst uses the browser plug-in provided by the present invention to perform the following steps: the data analyst first sets the content to be collected and the analysis tags in the background of the data analysis system. For example, in this case, the collection content set includes fields such as comment text, rating star, comment time, user ID and product name; the analysis tags include classification tags such as "positive evaluation", "neutral evaluation", "negative evaluation" and "logistics problem" and "product quality problem". These tags will be used for subsequent classification and analysis of the collected data. Then, the data analyst installs and starts the plug-in in the browser. When entering the comment page of a certain product, the plug-in appears in the form of a floating window on the right side of the browser page, supporting dragging, zooming and folding to ensure that it does not affect normal web browsing. When the data analyst sees the comment to be collected, he can select the target comment text with the mouse, and the plug-in automatically evokes the form interface and completes the following operations: (1) automatically extract the comment content selected by the user; (2) automatically identify and fill in the name, comment time and rating information of the current product page; (3) fill the URL of the current web page as the platform source into the form.

[0039] In addition, the data analyst selects the analysis tag corresponding to the comment (for example, marking the comment as "negative review" and "logistics problem") from the drop-down menu in the plug-in form and clicks the submit button. After submission, the plug-in automatically uploads the collected data and analysis tags to the server of the data analysis system through an encrypted protocol, and also attaches the collection timestamp for subsequent time series analysis.

[0040] The data analysis system processes the uploaded data in real time and generates multi-dimensional statistical analysis results. For example, the user evaluation distribution, main problem types, regional distribution, and trends over time of a certain product can be presented through data visualization charts. In addition, the system can further train user portraits based on deep learning models to provide merchants with personalized marketing strategies and product optimization suggestions. Through the practical application of this embodiment, the data collection problem of highly protected platforms is solved, that is, through the combination of manual judgment and plug-ins, the limitations of dynamically loaded pages and protected pages that traditional automation tools cannot handle are successfully bypassed. Classification and labeling are completed during the data collection process, shortening the data processing chain and reducing the time for manual secondary processing. In addition, the consistency and integrity of the collected data labeling are ensured through the background preset tags and plug-in automatic recognition technology, which is suitable for further data analysis in multiple scenarios.

[0041] Embodiment 2: In the monitoring scenario of social media platforms, enterprises need to collect user discussions, hot events and potential crisis information related to specific brands or products in a timely manner. These data are usually distributed in various comment areas, posts and dynamic pages without fixed formats, and have the characteristics of high real-time and high noise. Traditional automation technologies (such as crawlers) are difficult to meet the needs because they cannot accurately identify specific semantic content or bypass platform protection mechanisms, while pure manual monitoring is inefficient and cannot adapt to the rapid flow of information on social media. Therefore, this embodiment shows how to use the present invention to achieve efficient collection and real-time analysis of social media platform data.

[0042] In actual application, the analysis team of the enterprise uses the browser plug-in provided by the present invention to perform the following operations: First, set the key monitoring content and analysis tags in the background analysis system. For example, for the monitoring of a certain brand of mobile phones, the collection content set includes fields such as post title, release time, user ID, comment content, number of likes, etc.; analysis tags include "positive feedback", "negative feedback", "quality issues", "functional innovation suggestions", etc. Through these preset tags, subsequent collection work can quickly complete classification and labeling.

[0043] The analysis team members log in to the social media platform (such as Weibo or a forum) and use the browser with the plug-in installed to visit the target page. When the team members browse to a user comment related to the target brand, they select the comment with the mouse, and the plug-in automatically completes the following operations: extract the comment content and related information (such as the number of likes and the time of posting); automatically identify the platform information (such as the current URL and the ID of the user who posted the post), and fill this information into the corresponding fields of the form; in the floating form interface, the team members quickly select the preset analysis label (such as "negative feedback" and "quality issues") through the drop-down menu.

[0044] After completing the labeling, click the Submit button, and the plug-in will transmit the collected content and labels to the backend analysis system through an encrypted protocol. The backend system processes the uploaded data in real time, including classification statistics, trend analysis, and sentiment analysis.

[0045] In a specific scenario, for example, within a week after a brand released a new mobile phone, user feedback focused on the problem of "low screen brightness", with negative comments accounting for 30%. Through the technical solution of the present invention, the company quickly collected these negative comments, and combined with labels and trend analysis, it was found that the problem was mainly concentrated on a batch of products shipped in a specific region. Based on this, the company promptly issued a compensation plan and optimized the subsequent product debugging strategy, thereby effectively controlling the fermentation of negative feedback.

[0046] This embodiment also demonstrates that the real-time operation of the plug-in ensures that the enterprise can collect and analyze data in the first place. Through preset tags and automatic recognition technology, the accuracy and consistency of the collected data are significantly improved. In addition, since the present invention does not require the development of independent tools for different social media platforms, it is applicable to most mainstream platforms, saving development costs.

[0047] In the present invention, unless otherwise clearly stipulated and limited, the terms "installation", "setting", "connection", "fixation", "screw-on" and the like should be understood in a broad sense. For example, it can be a fixed connection, a detachable connection, or an integral one; it can be a mechanical connection or an electrical connection; it can be a direct connection or an indirect connection through an intermediate medium; it can be the internal connection of two elements or the interaction relationship between two elements. Unless otherwise clearly defined, ordinary technicians in this field can understand the specific meanings of the above terms in the present invention according to the specific circumstances.

[0048] It will be apparent to those skilled in the art that the present invention is not limited to the details of the exemplary embodiments described above and that the present invention can be implemented in other specific forms without departing from the spirit or essential features of the present invention. Therefore, the embodiments should be considered exemplary and non-restrictive in all respects, and the scope of the present invention is defined by the appended claims rather than the above description, and it is intended that all changes falling within the meaning and scope of the equivalent elements of the claims be included in the present invention.

Claims

1. A plug-in-based method for real-time collection and manual analysis of non-fixed format web page data, characterized in that: The method comprises the following steps: Step 1, preset collection content and analysis tags: preset collection content in the data analysis system, including collection object, user name, collection platform, analysis object and its associated analysis tags, where the analysis tags are used to mark the classification and purpose of the collected data; Step 2, install the browser plug-in: Install the plug-in for collecting data in the browser of the data collector. The plug-in can be presented on the web page in the form of a floating window and supports dragging, zooming and collapsing; Step 3, data collection trigger: select the non-fixed format data in the target web page through the floating window interface of the browser plug-in. The plug-in automatically captures the selected web page data, extracts the data content through the built-in parsing module, and automatically identifies the data source platform; Step 4, real-time analysis and annotation: In the plug-in floating window interface, the collector classifies and annotates the selected web page data according to the preset analysis tags, and the plug-in generates complete form data according to the annotated information; Step 5, data upload: the form data is uploaded to the backend server of the data analysis system through the communication module of the plug-in; Step 6, data display and subsequent processing: The uploaded data is presented in the data analysis system in various ways, including data statistics, deep learning model training, visualization display and user portrait analysis.

2. The method for real-time collection and manual analysis of non-fixed format web page data based on plug-in according to claim 1, characterized in that: The floating window interface of the browser plug-in can adjust the interface display mode according to the browsing needs of the collector, including minimization, floating display and hidden states to ensure normal web browsing.

3. The plug-in-based real-time collection and manual analysis method of non-fixed format web page data according to claim 1, characterized in that: The automatic parsing module of the plug-in includes the following functions: automatically extracting the content of the selected data and filling it into the plug-in form; Automatically identify the URL address of the current web page and record it in the form as platform source information.

4. The plug-in-based real-time collection and manual analysis method of non-fixed format web page data according to claim 1, characterized in that: The analysis tags are preset by the administrator of the data analysis system in the background system, and the data collector selects the analysis tags through the plug-in interface. The analysis tags may include but are not limited to user classification tags, product feature tags and behavior description tags.

5. The plug-in-based real-time collection and manual analysis method of non-fixed format web page data according to claim 1, characterized in that: The form data is attached with timestamp information when it is uploaded, which is used to record the time point of the collection operation to support subsequent data sorting and time series analysis.

6. The plug-in-based real-time collection and manual analysis method of non-fixed format web page data according to claim 1, characterized in that: The plug-in supports shortcut key operation, and the data collector can quickly call up the plug-in floating window through the shortcut key to perform data collection operations.

7. The plug-in-based real-time collection and manual analysis method of non-fixed format web page data according to claim 1, characterized in that: The data analysis system provides a data authority management function, and the uploaded data of different collectors can be allocated to different analysis modules according to preset authorities to ensure data security and targeted use.

8. The plug-in-based real-time collection and manual analysis method of non-fixed format web page data according to claim 1, characterized in that: The data analysis system supports real-time statistical analysis of uploaded data and presents the distribution, trend and correlation of the data in the form of charts.

9. The plug-in-based real-time collection and manual analysis method of non-fixed format web page data according to claim 1, characterized in that: The floating window interface of the browser plug-in supports a language switching function, and the operation interface language of the plug-in can be switched according to the language environment of the acquisition platform.

10. The plug-in-based real-time collection and manual analysis method of non-fixed format web page data according to claim 1, characterized in that: The plug-in supports uploading the collected web page data to the server of the data analysis system through an encrypted transmission protocol to ensure the security of the data transmission process.

Images