A method for automatic data collection based on web page tag analysis

Abstract

The present invention relates to the technical field of web page data collection, and in particular to an automatic data collection method based on web page tag analysis. The method comprises the following steps: extracting inter-cluster structural features of the acquired target collection web page data to obtain initial DOM tree structure feature data; monitoring page dynamic loading according to the initial DOM tree structure feature data to generate page dynamic loading data; performing dynamic tag combination processing according to the page dynamic loading data to obtain dynamic tag combination data; identifying brand word-of-mouth tags according to the dynamic tag combination data to generate brand word-of-mouth tag data; performing adaptive collection optimization according to the brand word-of-mouth tag data to generate an adaptive page collection strategy; performing multimodal brand data collection according to the adaptive page collection strategy, and performing product word-of-mouth prediction to obtain product word-of-mouth prediction data. The present invention realizes adaptive page data collection through dynamic tag combination.

Description

Technical Field

[0001] The present invention relates to the technical field of web page data collection, and in particular to an automatic data collection method based on web page tag analysis. Background Art

[0002] With the rapid development of Internet technology, the amount of network data has shown explosive growth. These data contain rich information and are of great value to business analysis, market forecasting, social research and other fields. Therefore, in the context of the big data era, how to quickly and accurately extract valuable data from massive web pages has become an urgent issue. However, traditional automatic data collection methods based on web page tag analysis often only target a single type of data and are unable to cope with the changing and complex data environment. Not only do they lack the ability to monitor web page changes in real time, they are also unable to adaptively adjust collection strategies to cope with these changes. In addition, these methods are relatively weak in the semantic understanding of web page content, resulting in a low level of collected data and difficulty in extracting higher-level, semantically rich information. Summary of the invention

[0003] Based on this, the present invention provides an automatic data collection method based on web page tag analysis to solve at least one of the above technical problems.

[0004] To achieve the above purpose, a method for automatic data collection based on web page tag analysis includes the following steps:

[0005] Step S1: acquiring target collected web page data; parsing the response code of the target collected web page data to generate initial DOM tree snapshot data; extracting inter-cluster structural features based on the initial DOM tree snapshot data to obtain initial DOM tree structural feature data;

[0006] Step S2: Monitor page dynamic loading according to the initial DOM tree structure feature data to generate page dynamic loading data; perform snapshot node structure change detection on the page dynamic loading data to generate page dynamic structure change data; perform dynamic tag combination processing according to the page dynamic structure change data to obtain dynamic tag combination data; perform brand word-of-mouth tag recognition according to the dynamic tag combination data to generate brand word-of-mouth tag data;

[0007] Step S3: Perform real-time element recognition on brand word-of-mouth tag data to generate real-time web page element data; perform adaptive collection optimization based on real-time web page element data to generate an adaptive page collection strategy; perform multimodal brand data collection based on the adaptive page collection strategy to generate multimodal brand data;

[0008] Step S4: Perform multimodal data fusion based on the multimodal brand data to obtain fused brand data; perform feature semantic analysis on the fused brand data to generate semantic analysis data; perform brand tendency evaluation based on the semantic analysis data, and perform product word-of-mouth prediction, thereby obtaining product word-of-mouth prediction data.

[0009] The present invention can obtain the web page content that needs to be analyzed by acquiring the target collected web page data. The response code analysis of the target collected web page data is to extract useful information from the web page source code, such as text content, links, pictures, etc. The process of generating the initial DOM tree snapshot data is helpful to obtain the overall structural information of the web page, as well as the layout and hierarchical relationship of each HTML element. By analyzing the initial DOM tree snapshot data, the inter-cluster structural features, that is, the structural relationship and interaction between different DOM elements, can be extracted. These inter-cluster structural features can understand the overall layout and organizational structure of the web page. According to the initial DOM tree structural feature data, the page dynamic loading monitoring is performed, the dynamic loading process of the web page can be monitored in real time, the content dynamically presented by the page as the user operates is recorded, and then the complete page information including the asynchronously loaded data is obtained, which helps to improve the perception ability of the web page changes. The snapshot node structure change detection is performed on the page dynamic loading data, and the structural changes of the page after the user operation can be captured. This snapshot node structure change detection helps to track and record the dynamic changes of the page elements such as addition, deletion, and modification. According to the page dynamic structure change data, dynamic tag combination processing is performed, and the dynamic changes of the page elements can be integrated and summarized to form a higher level of tag combination. According to the dynamic tag combination data, brand reputation tag identification is performed. By identifying and extracting tags related to brand reputation, users are provided with more accurate and targeted information. Through real-time element recognition, the system can dynamically capture new elements appearing on the web page, ensure timely acquisition of the latest user-generated content, including comments, ratings, pictures, etc., maintain real-time sensitivity to brand reputation, and enable the system to respond to user behavior and web page content changes in a timely manner. Flexible adjustment of acquisition strategies according to actual conditions. Through adaptive acquisition optimization, the crawling frequency, depth and method can be dynamically adjusted to adapt to the changes in different web page structures and contents, while dynamic invisible data mining is performed to improve the efficiency and accuracy of data acquisition. By utilizing adaptive acquisition strategies and combining multiple data modalities, more abundant and multi-dimensional brand data can be provided. Through multimodal data fusion, multiple data modalities such as text, images, and videos can be organically combined to form more comprehensive and comprehensive brand information. By performing feature semantic analysis on the fused brand data, the system can deeply mine the semantic information of brand data, extract key features, understand the implicit information, emotional tendencies and important features in brand data, and further enhance the comprehensibility and application value of brand data. By analyzing semantic analysis data, we can evaluate the overall tendency of brand reputation and predict the development trend of product reputation based on this information. This provides a strong basis for decision-making for brand managers and market analysts, and helps to formulate more accurate marketing strategies and brand management plans.Therefore, the automatic data collection method based on web page tag analysis of the present invention deeply analyzes the initial page DOM tree, and extracts the inter-cluster structure through effective tags, so as to obtain the initial DOM tree structure feature data; the page is dynamically loaded and monitored, and the dynamic structure change detection is performed on the initial DOM tree structure feature data, and the dynamic combination of web page tags is considered to realize the intelligent data recognition of dynamically changing tags. Considering the user interaction process, the user's high interest area is intelligently divided, so as to extract the user's real-time invisible demand data and realize the adaptive adjustment of the data collection strategy. The data collected based on dynamic tags is subjected to in-depth semantic analysis to predict the development trend of the product reputation.

[0010] Preferably, step S1 comprises the following steps:

[0011] Step S11: Acquire target collection web page data;

[0012] Step S12: performing initial web page response processing according to the target collected web page data to generate initial page response data;

[0013] Step S13: parsing the response code of the initial page response data to generate initial DOM tree snapshot data;

[0014] Step S14: constructing a valid tag graph according to the initial DOM tree snapshot data, thereby obtaining valid tag relationship graph data;

[0015] Step S15: Calculate the optimal number of clusters for the effective label relationship graph data using the preset internal cluster density data to generate optimal number of clusters data;

[0016] Step S16: Perform hierarchical clustering analysis based on the optimal cluster quantity data to generate node hierarchical clustering data;

[0017] Step S17: extracting inter-cluster structural features from the node hierarchical clustering data, thereby obtaining initial DOM tree structural feature data.

[0018] The present invention obtains the original HTML or related data of the target website through a data acquisition tool. The obtained web page data is preliminarily processed, the web page response is parsed, and useful information such as response header, status code, etc. is extracted to generate initial page response data. By parsing the HTML code of the initial page response, the initial DOM tree snapshot data can be constructed. This DOM tree snapshot is a static representation of the web page structure, including various HTML tags and their hierarchical structure. An effective tag graph is constructed based on the initial DOM tree snapshot to describe the relationship between the tags in the web page. This helps to highlight the important tags in the web page and form effective tag relationship graph data. Using the preset internal cluster density data and the relatively tight metric of the member points within the cluster, the optimal number of clusters in the effective tag relationship graph is calculated. This helps to determine the appropriate number of tag clusters, so as to better capture the correlation in the web page structure and reduce noise interference. By using the optimal number of clusters data, we perform hierarchical clustering analysis and group the effective tags to form a hierarchical clustering structure. Such analysis helps to identify the hierarchical relationship between tags and reveal the potential hierarchical nature of the web page structure. By extracting the inter-cluster structural features of the node hierarchical clustering data, the initial DOM tree structure feature data is obtained. This includes the relationship between nodes in the cluster, the structural features between clusters, etc. These feature data help to understand the structure of web pages more comprehensively.

[0019] Preferably, step S14 comprises the following steps:

[0020] Step S141: performing tree-level traversal on the initial DOM tree snapshot data to generate tree-level traversal data;

[0021] Step S142: performing node type classification according to the tree level traversal data to generate node type data;

[0022] Step S143: performing a valid label node check on the node type data to obtain valid label node data;

[0023] Step S144: recursively traverse the child nodes of the valid label node data to generate parent-child node relationship data; traverse the same-level nodes of the valid label node data to generate brother node relationship data;

[0024] Step S145: constructing a graph data structure for the valid label node data through the parent-child node relationship data and the sibling node relationship data to generate node graph structure data;

[0025] Step S146: marking the node level depth according to the node graph structure data to generate node level depth data;

[0026] Step S147: Draw a directed label relationship graph for the node graph structure data using the node hierarchy depth data to generate valid label relationship graph data.

[0027] The present invention obtains the hierarchical position information of each node in the DOM tree by performing tree-level traversal on the initial DOM tree snapshot data. This helps to establish the nested relationship between each node and reveal the tree-shaped hierarchy of the web page structure. Based on the tree-level traversal data, the DOM tree nodes are divided into types, and nodes of similar types are classified into one category. This helps to clarify the structure and function of the nodes and improve the understanding of the web page content. The node type data is screened, and the nodes that meet specific standards are determined as valid label nodes, which helps to filter out non-critical nodes, reduce interference, and improve the accuracy of subsequent analysis of cluster structures. By recursively traversing and traversing the valid label node data, the parent-child relationship and brother relationship data between the nodes are obtained, and these relationship data help to reveal the specific position of the label in the DOM tree and the association between each other. Based on the established parent-child node relationship and brother node relationship data, a graph data structure is constructed, in which the node represents the valid label node and the edge represents the relationship between the nodes. The construction of this graph enables us to better understand the connection between the nodes. By performing depth-first search or other traversal algorithms on the node graph structure data, the hierarchical depth information of each node is marked. This helps to understand the depth position of the node in the DOM tree. Based on the node hierarchy depth data, a directed tag relationship graph is drawn, where the nodes represent valid tags and the directed edges represent the hierarchical relationship between tags.

[0028] Preferably, step S2 comprises the following steps:

[0029] Step S21: Monitor page dynamic loading according to the initial DOM tree structure feature data to generate page dynamic loading data;

[0030] Step S22: capturing a dynamic DOM tree snapshot of the page dynamically loaded data to generate dynamic DOM tree snapshot data;

[0031] Step S23: Detecting snapshot node structure changes of the initial DOM tree structure feature data using the dynamic DOM tree snapshot data to generate page dynamic structure change data;

[0032] Step S24: Timestamp the page dynamic structure change data to generate time series dynamic structure change data;

[0033] Step S25: extracting the tag frequency from the time series dynamic structure change data to generate web page tag frequency data;

[0034] Step S26: performing dynamic tag combination processing according to the web page tag frequency data to obtain dynamic tag combination data; performing tag annotation processing according to the dynamic tag combination data to generate dynamic tag annotation data;

[0035] Step S27: Based on the preset graph neural network model, perform label recognition transfer learning on the dynamic label annotation data through the effective label relationship graph data to generate a dynamic label recognition model;

[0036] Step S28: Transmit the time-series dynamic structure change data to the dynamic tag recognition model for brand word-of-mouth tag recognition to generate brand word-of-mouth tag data.

[0037] The present invention can monitor the dynamic loading events in the page in real time by monitoring the initial DOM tree structure feature data. This includes asynchronous requests, AJAX calls or other operations that cause page changes. Based on the page dynamic loading data, a DOM tree snapshot in the dynamic loading process is captured. Such a snapshot contains the structure and content changes of the page after dynamic loading. By comparing the initial DOM tree structure feature data and the dynamic DOM tree snapshot data, the structural changes of the page after dynamic loading can be detected. The generated page dynamic structure change data records the node information that is added, modified or deleted. The page dynamic structure change data is timestamped, and the order and time of the structural changes can be traced. By analyzing the time-series dynamic structure change data, the frequency of each tag in the page structure change is extracted. The generated web page tag frequency data is used to understand the relative activity of each tag in the page dynamic change. Combined with the web page tag frequency data, dynamic tag combination processing is performed. In this process, the association between tags and frequently occurring combinations are identified, and dynamic tag combination data is generated. Based on the dynamic tag combination data, tag annotation processing is performed. By giving meaningful tag annotations to different tags, the tags are easier to understand and identify. Using the preset graph neural network model, label recognition transfer learning is performed through the effective label relationship graph data. In this process, the dynamic label annotation data is used for model training to generate a dynamic label recognition model for the current page. The temporal dynamic structure change data is input into the trained dynamic label recognition model to identify the brand word-of-mouth label. The generated brand word-of-mouth label data reflects the brand characteristics and user evaluation information reflected in the temporal dynamic structure change of the page.

[0038] Preferably, step S3 comprises the following steps:

[0039] Step S31: performing real-time element recognition on brand word-of-mouth tag data through a preset browser plug-in architecture to generate real-time web page element data;

[0040] Step S32: performing adaptive collection optimization according to real-time web page element data to generate an adaptive page collection strategy;

[0041] Step S33: formulating a sampling task for the real-time web page element data according to a preset web page data collection rule to obtain sampling task data;

[0042] Step S34: Perform distributed queue processing on the sampling task data to generate distributed sampling task data;

[0043] Step S35: Based on the distributed sampling task data, multimodal brand data is collected through an adaptive page collection strategy to generate multimodal brand data.

[0044] The present invention uses a browser plug-in architecture to perform element recognition on brand word-of-mouth tag data in real time, which helps to capture elements that change in real time in the page and generate real-time web page element data. Adaptive collection optimization is performed using real-time web page element data. This includes optimizing strategies such as determining the timing, frequency, and method of collection to adapt to the dynamic changes of different web pages and generate more flexible and efficient adaptive page collection strategies. Based on preset web page data collection rules, sampling tasks are formulated for real-time web page element data, which helps to determine the scope, depth, and frequency of sampling. Processing sampling task data through distributed queues to achieve parallel processing of tasks helps to improve collection efficiency and ensure that sampling tasks can be efficiently executed in a distributed environment. Combining distributed sampling task data and adaptive page collection strategies, the collection of multimodal brand data is implemented to ensure that the collected data is comprehensive and diverse, and has a comprehensive understanding of brand multimodal information.

[0045] Preferably, step S32 includes the following steps:

[0046] Step S321: Perform user interaction monitoring according to the real-time web page element data to generate user interaction monitoring data.

[0047] Step S322: Generate a page heat map using the user interaction monitoring data, thereby obtaining corrected heat map data;

[0048] Step S323: Formulate a page collection strategy based on the modified heat map data to generate an initial page collection strategy;

[0049] Step S324: extracting behavior key indicator data from the modified heat map data to generate behavior key indicator data;

[0050] Step S325: performing importance evaluation processing according to the behavior key indicator data to generate key indicator evaluation data;

[0051] Step S326: Adaptively update the acquisition weight according to the key indicator evaluation data to obtain real-time acquisition feature weight data;

[0052] Step S327: dynamically adjusting the preset initial acquisition rate using the real-time acquisition feature weight data to obtain dynamic acquisition rate data;

[0053] Step S328: Adaptively optimize the initial page acquisition strategy through dynamic acquisition rate data to generate an adaptive page acquisition strategy.

[0054] The present invention generates user interaction monitoring data by real-time monitoring of user interactions on a web page, including behaviors such as clicking, scrolling, and inputting, which is helpful for understanding the user's behavior pattern on the page. Based on the user interaction monitoring data, a heat map of the page is generated. The heat map reflects the user's attention and activity in different areas on the page, and then provides modified heat map data. This helps to determine the hot spots on the page. Based on the modified heat map data, an initial page acquisition strategy is formulated. This strategy takes into account the user's attention and interest distribution on the page, optimizes the direction of data acquisition, and ensures that the content of interest to the user is captured more effectively. By analyzing the modified heat map data, key user behavior indicators are extracted. This may include indicators such as click-through rate, dwell time, and scroll depth. These key indicators can be used to further evaluate the user's interest in different elements and areas. By evaluating the key indicators of behavior, the importance of each behavior in the user experience can be determined. Through the key indicator evaluation data, the weights of each acquisition feature are adaptively updated. Such weight update can make the acquisition strategy more flexible and intelligent, and dynamically adjust according to real-time data changes, so as to better adapt to changes in user behavior. Based on the real-time collection of feature weight data, the initial collection rate can be adjusted dynamically according to the actual situation of the current data to ensure the timeliness and accuracy of data collection. By combining the dynamic collection rate data, the initial page collection strategy is optimized to make the collection more intelligent and efficient. This helps to improve the quality of data collection and ensure that the collected data better meets the actual needs of users.

[0055] Preferably, step S322 includes the following steps:

[0056] Step S3221: performing interaction event classification processing on the user interaction monitoring data to obtain interaction event classification data, wherein the interaction event classification data includes user interaction behavior event data, mouse movement track event data and keyboard input event data;

[0057] Step S3222: performing time series behavior processing according to the mouse movement trajectory event data to generate time series behavior point data; performing keyword extraction according to the keyboard input event data to generate input keyword data;

[0058] Step S3223: performing cross-trajectory analysis on the time series behavior point data to generate interest area trajectory data;

[0059] Step S3224: calculating the interest depth of the interest region trajectory data by using an interest depth algorithm, thereby obtaining interest depth data;

[0060] Step S3225: Generate a page heat map for the interest depth data through the user interaction behavior event data to generate page heat map data;

[0061] Step S3226: Use the input keyword data to perform heat map correction on the page heat map data to generate corrected heat map data.

[0062] The present invention performs interactive event classification processing on user interaction monitoring data to obtain interactive event classification data, including user interactive behavior event data, mouse movement trajectory event data and keyboard input event data. By classifying and processing user interaction data, different types of user behaviors can be effectively distinguished. By processing the temporal behavior of the mouse movement trajectory, the specific operation process of the user on the page can be captured, providing more detailed information for behavior analysis. At the same time, extracting keywords in the keyboard input event helps to understand the specific needs and intentions of the user. Through cross-trajectory analysis, the user's interest area on the page, that is, the area that is often paid attention to and operated by the user, can be identified, which helps to better understand the user's interest points. The interest depth is calculated on the interest area trajectory data through the interest depth algorithm to quantify the user's interest in different areas of the page. Through the calculation of the interest depth, the user's attention to specific content can be more accurately understood. With the help of the user's interactive behavior data, such as clicking, browsing, etc., a page heat map is generated to intuitively display the heat distribution of each area on the page. By combining the keyword information input by the user, the page heat map can be more finely corrected to ensure that the heat map is more in line with the user's real interest points and needs.

[0063] Preferably, step S4 comprises the following steps:

[0064] Step S41: performing data standardization processing on the multimodal brand data to generate standard multimodal brand data;

[0065] Step S42: performing multimodal data fusion according to standard multimodal brand data, thereby obtaining fused brand data;

[0066] Step S43: performing natural language processing on the fused brand data to generate fused brand text data;

[0067] Step S44: extracting brand word-of-mouth keywords from the fused brand text data using a preset word-of-mouth keyword extraction rule to generate word-of-mouth keyword data;

[0068] Step S45: performing feature semantic analysis on the fused brand text data through word-of-mouth keyword data;

[0069] Step S46: using a sentiment analysis algorithm to evaluate the brand tendency of the semantic analysis data and generate a brand sentiment tendency score;

[0070] Step S47: Predict product word-of-mouth based on the brand sentiment tendency score to generate product word-of-mouth prediction data.

[0071] The present invention can ensure the consistency and comparability of different modal data by standardizing multimodal brand data, and the standardized data helps to reduce the complexity of data processing. Through the fusion of multimodal data, multiple data sources such as images, texts, and audios can be comprehensively utilized to obtain more comprehensive and richer brand information. The fused data helps to build a brand analysis model with more information and dimensions, and improves the comprehensiveness and accuracy of brand analysis. Through natural language processing technology, text information, including product descriptions, user comments, etc., can be extracted from the fused brand data. This helps to understand the characteristics of the brand and user feedback more deeply. By defining preset word-of-mouth keyword extraction rules, keywords related to brand word-of-mouth can be selectively extracted. This helps to focus on key information related to the brand in user evaluations and provide targeted keyword data for word-of-mouth analysis. Use word-of-mouth keywords to conduct in-depth analysis of brand texts to explore key features and semantic information hidden in the text. Feature semantic analysis helps to extract the theme, key information, and important features of the text. Through sentiment analysis, the system can identify the emotional color in the text, including positive, negative, and neutral emotions. The brand sentiment tendency score reflects the user's overall emotional attitude towards the brand and provides an objective indicator for the quantitative evaluation of brand reputation. As an important reference for reputation, the brand sentiment tendency score can be used to predict the performance of products in the market. Product reputation prediction data based on sentiment analysis results can help companies adjust marketing strategies, optimize product design, and better meet consumer needs, thereby improving brand competitiveness.

[0072] The beneficial effect of the present application is that, in the data collection process, the target web page data is first obtained through HTTP request, and the web page content is analyzed. The response code is parsed, and information such as text, links, and pictures are extracted from the source code to generate an initial DOM tree snapshot. This helps to understand the overall structure of the web page and the layout of HTML elements. Through the analysis of inter-cluster structural features, the relationship and interaction between different DOM elements can be understood. Monitor the dynamic loading events of the page, record the dynamic changes of the page after the user operation, and obtain complete page information including asynchronous loading data. Perform node structure change detection on the dynamically loaded data and track the addition, deletion and change of page elements. Integrate these changes to form a high-level tag combination, perform brand word-of-mouth tag identification, and provide accurate and targeted information. Design a browser plug-in to process data in real time, and dynamically capture new elements such as comments, ratings, pictures, etc. according to user behavior and changes in web page content. Adjust the collection strategy according to the actual situation, dynamically adjust the crawling frequency, depth and method, and improve the efficiency and accuracy of data collection. Utilize adaptive collection strategies, combine multiple data modalities, such as text, images, videos, etc., and form comprehensive brand information through multimodal data fusion. Perform feature semantic analysis on the fused data, mine the semantic information of brand data, extract key features, and understand implicit information and emotional tendencies. Through semantic analysis of data, evaluate the overall tendency of brand reputation, predict the development trend of product reputation, and provide decision-making basis for brand management and market analysis. BRIEF DESCRIPTION OF THE DRAWINGS

[0073] Figure 1 The present invention is a schematic diagram of the steps of an automatic data collection method based on web page tag analysis.

[0074] Figure 2 for Figure 1 Detailed implementation steps of step S1 in FIG.

[0075] Figure 3 for Figure 1 Detailed implementation steps of step S3 in FIG.

[0076] The realization of the purpose, functional features and advantages of the present invention will be further explained in conjunction with embodiments and with reference to the accompanying drawings. DETAILED DESCRIPTION

[0077] The following is a clear and complete description of the technical method of the present invention in conjunction with the accompanying drawings. Obviously, the described embodiments are part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by technicians in this field without creative work are within the scope of protection of the present invention.

[0078] In addition, the accompanying drawings are only schematic illustrations of the present invention and are not necessarily drawn to scale. The same reference numerals in the figures represent the same or similar parts, and their repeated description will be omitted. Some of the block diagrams shown in the accompanying drawings are functional entities and do not necessarily correspond to physically or logically independent entities. The functional entities can be implemented in software form, or implemented in one or more hardware modules or integrated circuits, or implemented in different networks and / or processor methods and / or microcontroller methods.

[0079] It should be understood that, although the terms "first", "second", etc. may be used herein to describe various units, these units should not be limited by these terms. These terms are used only to distinguish one unit from another unit. For example, without departing from the scope of the exemplary embodiments, the first unit may be referred to as the second unit, and similarly the second unit may be referred to as the first unit. The term "and / or" used herein includes any and all combinations of one or more of the listed associated items.

[0080] To achieve this, please refer to Figures 1 to 3 The present invention provides a method for automatically collecting data based on web page tag analysis, comprising the following steps:

[0081] Step S1: acquiring target collected web page data; parsing the response code of the target collected web page data to generate initial DOM tree snapshot data; extracting inter-cluster structural features based on the initial DOM tree snapshot data to obtain initial DOM tree structural feature data;

[0082] Step S2: Monitor page dynamic loading according to the initial DOM tree structure feature data to generate page dynamic loading data; perform snapshot node structure change detection on the page dynamic loading data to generate page dynamic structure change data; perform dynamic tag combination processing according to the page dynamic structure change data to obtain dynamic tag combination data; perform brand word-of-mouth tag recognition according to the dynamic tag combination data to generate brand word-of-mouth tag data;

[0083] Step S3: Perform real-time element recognition on brand word-of-mouth tag data to generate real-time web page element data; perform adaptive collection optimization based on real-time web page element data to generate an adaptive page collection strategy; perform multimodal brand data collection based on the adaptive page collection strategy to generate multimodal brand data;

[0084] Step S4: Perform multimodal data fusion based on the multimodal brand data to obtain fused brand data; perform feature semantic analysis on the fused brand data to generate semantic analysis data; perform brand tendency evaluation based on the semantic analysis data, and perform product word-of-mouth prediction, thereby obtaining product word-of-mouth prediction data.

[0085] The present invention can obtain the web page content that needs to be analyzed by acquiring the target collected web page data. The response code analysis of the target collected web page data is to extract useful information from the web page source code, such as text content, links, pictures, etc. The process of generating the initial DOM tree snapshot data is helpful to obtain the overall structural information of the web page, as well as the layout and hierarchical relationship of each HTML element. By analyzing the initial DOM tree snapshot data, the inter-cluster structural features, that is, the structural relationship and interaction between different DOM elements, can be extracted. These inter-cluster structural features can understand the overall layout and organizational structure of the web page. According to the initial DOM tree structural feature data, the page dynamic loading monitoring is performed, the dynamic loading process of the web page can be monitored in real time, the content dynamically presented by the page as the user operates is recorded, and then the complete page information including the asynchronously loaded data is obtained, which helps to improve the perception ability of the web page changes. The snapshot node structure change detection is performed on the page dynamic loading data, and the structural changes of the page after the user operation can be captured. This snapshot node structure change detection helps to track and record the dynamic changes of the page elements such as addition, deletion, and modification. According to the page dynamic structure change data, dynamic tag combination processing is performed, and the dynamic changes of the page elements can be integrated and summarized to form a higher level of tag combination. According to the dynamic tag combination data, brand reputation tag identification is performed. By identifying and extracting tags related to brand reputation, users are provided with more accurate and targeted information. Through real-time element recognition, the system can dynamically capture new elements appearing on the web page, ensure timely acquisition of the latest user-generated content, including comments, ratings, pictures, etc., maintain real-time sensitivity to brand reputation, and enable the system to respond to user behavior and web page content changes in a timely manner. Flexible adjustment of acquisition strategies according to actual conditions. Through adaptive acquisition optimization, the crawling frequency, depth and method can be dynamically adjusted to adapt to the changes in different web page structures and contents, while dynamic invisible data mining is performed to improve the efficiency and accuracy of data acquisition. By utilizing adaptive acquisition strategies and combining multiple data modalities, more abundant and multi-dimensional brand data can be provided. Through multimodal data fusion, multiple data modalities such as text, images, and videos can be organically combined to form more comprehensive and comprehensive brand information. By performing feature semantic analysis on the fused brand data, the system can deeply mine the semantic information of brand data, extract key features, understand the implicit information, emotional tendencies and important features in brand data, and further enhance the comprehensibility and application value of brand data. By analyzing semantic analysis data, we can evaluate the overall tendency of brand reputation and predict the development trend of product reputation based on this information. This provides a strong basis for decision-making for brand managers and market analysts, and helps to formulate more accurate marketing strategies and brand management plans.Therefore, the automatic data collection method based on web page tag analysis of the present invention deeply analyzes the initial page DOM tree, and extracts the inter-cluster structure through effective tags, so as to obtain the initial DOM tree structure feature data; the page is dynamically loaded and monitored, and the dynamic structure change detection is performed on the initial DOM tree structure feature data, and the dynamic combination of web page tags is considered to realize the intelligent data recognition of dynamically changing tags. Considering the user interaction process, the user's high interest area is intelligently divided, so as to extract the user's real-time invisible demand data and realize the adaptive adjustment of the data collection strategy. The data collected based on dynamic tags is subjected to in-depth semantic analysis to predict the development trend of the product reputation.

[0086] In the embodiment of the present invention, reference Figure 1 The above is a schematic diagram of the steps of a method for automatic data collection based on web page tag analysis of the present invention. In this embodiment, the method for automatic data collection based on web page tag analysis includes the following steps:

[0087] Step S1: acquiring target collected web page data; parsing the response code of the target collected web page data to generate initial DOM tree snapshot data; extracting inter-cluster structural features based on the initial DOM tree snapshot data to obtain initial DOM tree structural feature data;

[0088] In an embodiment of the present invention, during the data collection process, the target website and the required data type, such as text, picture or link, are first determined. A GET request is sent to the target website through an HTTP request tool (such as curl) to obtain a web page response. Ensure that the response status code is 200, indicating successful reception. Then, extract the HTML response body, which contains the web page structure and content. Use a parsing library (such as Beautiful Soup) to parse HTML and build a DOM tree. Traverse the DOM tree, identify the tag type and attributes of the node, and build a node object. Ignore irrelevant tags and only consider meaningful tags (such as div, p, a). Calculate the internal density of each tag cluster and set a density threshold (such as 0.6 to 0.8) to determine the optimal number of clusters. Convert the tag relationship graph into a hierarchical clustering format using a node-edge representation. Select an agglomerative hierarchical clustering algorithm that is suitable for large data sets. Merge nodes step by step to form a cluster hierarchy. According to the optimal number of clusters, select the cluster in the corresponding hierarchy as the clustering result. Traverse each cluster and extract structural features, such as the number of nodes, depth, and breadth. Integrate these features to construct the initial DOM tree structure feature data.

[0089] Step S2: Monitor page dynamic loading according to the initial DOM tree structure feature data to generate page dynamic loading data; perform snapshot node structure change detection on the page dynamic loading data to generate page dynamic structure change data; perform dynamic tag combination processing according to the page dynamic structure change data to obtain dynamic tag combination data; perform brand word-of-mouth tag recognition according to the dynamic tag combination data to generate brand word-of-mouth tag data;

[0090] In the embodiment of the present invention, in web page data collection, the browser developer tool is first used to monitor page dynamic loading events and capture newly added resources and nodes. The DOM is operated by JavaScript to record the snapshot of the DOM tree after dynamic loading, and the initial DOM tree is compared to detect node changes. Timestamps are added to the change data to form a time series dynamic structure change record. Time series data is processed to count the frequency of tag changes, and tags with high frequency are selected to form a dynamic tag group, such as and Combination <divp>. Semantically annotate dynamic tag groups, such as <divp> Marked as< / divp> < / divp> <section>. Use graph neural network models (such as GCN, GAT) to build a transfer learning model for label recognition. Use label relationship graphs and annotated data to train the model to adapt to the dynamic label environment. After training, obtain the label recognition model, input time series data, and obtain label recognition result data.

[0091] Step S3: Perform real-time element recognition on brand word-of-mouth tag data to generate real-time web page element data; perform adaptive collection optimization based on real-time web page element data to generate an adaptive page collection strategy; perform multimodal brand data collection based on the adaptive page collection strategy to generate multimodal brand data;

[0092] In an embodiment of the present invention, when designing a browser plug-in, first ensure that it can process brand word-of-mouth tag data in real time. Configure the plug-in to identify specific elements, such as HTML tags or CSS classes, in real time, and set an interface to receive back-end data. Analyze users' focus on brand word-of-mouth tags, formulate adaptive collection strategies, and adjust the collection frequency to reflect user attention. For example, increase the collection frequency of the comment box to obtain the latest comments. Set collection rules, such as sampling frequency and collection depth, to ensure that data is obtained in a prescribed manner. Design a distributed queue architecture to handle a large number of sampling tasks, and determine a multimodal data collection strategy based on the task results. The task node performs the collection operation and uploads the collected data to the central server.

[0093] Step S4: Perform multimodal data fusion based on the multimodal brand data to obtain fused brand data; perform feature semantic analysis on the fused brand data to generate semantic analysis data; perform brand tendency evaluation based on the semantic analysis data, and perform product word-of-mouth prediction, thereby obtaining product word-of-mouth prediction data.

[0094] In an embodiment of the present invention, when processing multimodal brand data, standardization is first performed, such as pixel standardization of images and length standardization of text. CNN is used to extract image features, natural language processing technology is used to extract text features, and the two are fused through a multimodal fusion network. The fusion model weights are optimized to improve performance, and the model is applied to standardized data to generate fused brand data. The cleaned text is segmented and tagged with parts of speech, converted into word vectors, and sentiment analysis is used to determine sentiment polarity. Word-of-mouth keywords, such as "good quality" and "service attitude", are extracted, and feature semantic analysis is performed. The sentiment polarity of the features is evaluated using a sentiment analysis algorithm, and the overall sentiment tendency score of the brand is calculated. The sentiment scores are classified according to thresholds to generate word-of-mouth prediction data.

[0095] Preferably, step S1 comprises the following steps:

[0096] Step S11: Acquire target collection web page data;

[0097] Step S12: performing initial web page response processing according to the target collected web page data to generate initial page response data;

[0098] Step S13: parsing the response code of the initial page response data to generate initial DOM tree snapshot data;

[0099] Step S14: constructing a valid tag graph according to the initial DOM tree snapshot data, thereby obtaining valid tag relationship graph data;

[0100] Step S15: Calculate the optimal number of clusters for the effective label relationship graph data using the preset internal cluster density data to generate optimal number of clusters data;

[0101] Step S16: Perform hierarchical clustering analysis based on the optimal cluster quantity data to generate node hierarchical clustering data;

[0102] Step S17: extracting inter-cluster structural features from the node hierarchical clustering data, thereby obtaining initial DOM tree structural feature data.

[0103] The present invention obtains the original HTML or related data of the target website through a data acquisition tool. The obtained web page data is preliminarily processed, the web page response is parsed, and useful information such as response header, status code, etc. is extracted to generate initial page response data. By parsing the HTML code of the initial page response, the initial DOM tree snapshot data can be constructed. This DOM tree snapshot is a static representation of the web page structure, including various HTML tags and their hierarchical structure. An effective tag graph is constructed based on the initial DOM tree snapshot to describe the relationship between the tags in the web page. This helps to highlight the important tags in the web page and form effective tag relationship graph data. Using the preset internal cluster density data and the relatively tight metric of the member points within the cluster, the optimal number of clusters in the effective tag relationship graph is calculated. This helps to determine the appropriate number of tag clusters, so as to better capture the correlation in the web page structure and reduce noise interference. By using the optimal number of clusters data, we perform hierarchical clustering analysis and group the effective tags to form a hierarchical clustering structure. Such analysis helps to identify the hierarchical relationship between tags and reveal the potential hierarchical nature of the web page structure. By extracting the inter-cluster structural features of the node hierarchical clustering data, the initial DOM tree structure feature data is obtained. This includes the relationship between nodes in the cluster, the structural features between clusters, etc. These feature data help to understand the structure of web pages more comprehensively.

[0104] As an example of the present invention, refer to Figure 2 As shown, Figure 1 Detailed implementation steps of step S1 in the flowchart, in this example, step S1 includes:

[0105] Step S11: Acquire target collection web page data;

[0106] In the embodiment of the present invention, before data collection is performed, it is firstly determined which target website is and what specific data types need to be collected, such as text content, pictures, links, etc.

[0107] Step S12: performing initial web page response processing according to the target collected web page data to generate initial page response data;

[0108] In an embodiment of the present invention, an HTTP request tool, such as curl or a browser, is used to send a GET request to a target website to obtain the initial response data of a web page. For example, the target website is accessed through a curl command or a browser to obtain the response data of the homepage. The HTTP status code of the initial response data is parsed to ensure that the status code is 200, indicating that the request is successful. For example, the HTTP status code in the response data is checked. If it is 200, the response data is continued to be processed; otherwise, the request is processed in the case of failure.

[0109] Step S13: parsing the response code of the initial page response data to generate initial DOM tree snapshot data;

[0110] In an embodiment of the present invention, the HTML response body part is extracted from the initial page response data, and the part includes the structure and content of the web page. For example, by parsing the initial page response data, the HTML response body part is extracted. The HTML response body is parsed using the loaded parsing library to construct an initial DOM tree. For example, the parser of the Beautiful Soup library is used to parse the HTML response body and generate a DOM tree.

[0111] Step S14: constructing a valid tag graph according to the initial DOM tree snapshot data, thereby obtaining valid tag relationship graph data;

[0112] In an embodiment of the present invention, the nodes of the initial DOM tree are traversed, the tag type, attributes and other information of each node are identified, and the node object is constructed. For example, the nodes of the DOM tree are traversed, and the tag type, class name, ID and other information of each node are extracted to construct the node object. Through the traversal process, according to the relationship between nodes (such as parent-child relationship, brother relationship), a valid tag graph is constructed. Here, only tags with practical significance, such as div, p, a, etc., are considered, and insignificant tags are ignored.

[0113] Step S15: Calculate the optimal number of clusters for the effective label relationship graph data using the preset internal cluster density data to generate optimal number of clusters data;

[0114] In an embodiment of the present invention, the internal closeness of each cluster, that is, the strength of the relationship between the labels within the cluster, is calculated based on the effective label relationship graph data. For example, the effective label relationship graph is traversed to calculate the internal closeness of each cluster. A threshold or range of the internal cluster closeness is pre-set. The internal cluster closeness represents the close relationship between the labels within the cluster. For example, the range of the internal cluster closeness is set to 0.6 to 0.8. The calculated internal closeness of each cluster is analyzed to determine the optimal number of clusters. Generally, the optimal number of clusters corresponds to the number of clusters whose internal cluster closeness reaches a preset range.

[0115] Step S16: Perform hierarchical clustering analysis based on the optimal cluster quantity data to generate node hierarchical clustering data;

[0116] In an embodiment of the present invention, the effective label relationship graph data is converted into a format suitable for hierarchical clustering. A node-edge representation method is usually adopted, in which nodes represent effective labels and edges represent the relationship between labels. An agglomerative hierarchical clustering algorithm is selected because it is suitable for large data sets and less noise. Using the agglomerative hierarchical clustering algorithm, nodes are gradually merged to form a hierarchical structure of clusters. According to the optimal number of clusters data, a cluster in the hierarchy that matches it is selected as the final hierarchical clustering result.

[0117] Step S17: extracting inter-cluster structural features from the node hierarchical clustering data, thereby obtaining initial DOM tree structural feature data.

[0118] In the embodiment of the present invention, each cluster is traversed to extract the structural features between clusters. This may include the number, depth, breadth, average node degree, etc. of nodes in the cluster. For example, each cluster is traversed to calculate the number, depth, breadth, etc. of nodes in the cluster. The structural features extracted from each cluster are integrated into the initial DOM tree structural feature data.

[0119] Preferably, step S14 comprises the following steps:

[0120] Step S141: performing tree-level traversal on the initial DOM tree snapshot data to generate tree-level traversal data;

[0121] Step S142: performing node type classification according to the tree level traversal data to generate node type data;

[0122] Step S143: performing a valid label node check on the node type data to obtain valid label node data;

[0123] Step S144: recursively traverse the child nodes of the valid label node data to generate parent-child node relationship data; traverse the same-level nodes of the valid label node data to generate brother node relationship data;

[0124] Step S145: constructing a graph data structure for the valid label node data through the parent-child node relationship data and the sibling node relationship data to generate node graph structure data;

[0125] Step S146: marking the node level depth according to the node graph structure data to generate node level depth data;

[0126] Step S147: Draw a directed label relationship graph for the node graph structure data using the node hierarchy depth data to generate valid label relationship graph data.

[0127] In an embodiment of the present invention, a DOM tree is traversed using algorithms such as depth-first traversal or breadth-first traversal to record the hierarchical information of the nodes. Based on the depth information of the nodes, the node types are divided, such as root nodes, internal nodes, leaf nodes, etc. For example, based on the depth information, the node types are divided into root nodes (depth is 0), internal nodes (depth is greater than 0 but has child nodes), and leaf nodes (depth is greater than 0 but has no child nodes). Based on the defined valid tag rules, check whether each node is a valid tag node. For example, define a valid tag as 、 、 Etc., check whether each node belongs to these valid labels. For valid label node data, use recursion or other traversal algorithms to traverse the child nodes of each node and record the parent-child node relationship. Perform peer node traversal on valid label node data, record the sibling node information of each node, and form sibling node relationship data. Use parent-child node relationship data and sibling node relationship data to construct node graph structure data. For example, construct the adjacency list data structure of the graph based on the parent-child node relationship and sibling node relationship. According to the relationship between nodes, draw directed edges in the graphical environment to represent the relationship between nodes, traverse the node graph structure data, and draw each node in the appropriate position according to the depth information, so as to obtain a valid label relationship graph.

[0128] Preferably, step S2 comprises the following steps:

[0129] Step S21: Monitor page dynamic loading according to the initial DOM tree structure feature data to generate page dynamic loading data;

[0130] Step S22: capturing a dynamic DOM tree snapshot of the page dynamically loaded data to generate dynamic DOM tree snapshot data;

[0131] Step S23: Detecting snapshot node structure changes of the initial DOM tree structure feature data using the dynamic DOM tree snapshot data to generate page dynamic structure change data;

[0132] Step S24: Timestamp the page dynamic structure change data to generate time series dynamic structure change data;

[0133] Step S25: extracting the tag frequency from the time series dynamic structure change data to generate web page tag frequency data;

[0134] Step S26: performing dynamic tag combination processing according to the web page tag frequency data to obtain dynamic tag combination data; performing tag annotation processing according to the dynamic tag combination data to generate dynamic tag annotation data;

[0135] Step S27: Based on the preset graph neural network model, perform label recognition transfer learning on the dynamic label annotation data through the effective label relationship graph data to generate a dynamic label recognition model;

[0136] Step S28: Transmit the time-series dynamic structure change data to the dynamic tag recognition model for brand word-of-mouth tag recognition to generate brand word-of-mouth tag data.

[0137] In an embodiment of the present invention, a browser developer tool or other monitoring tool is used to monitor the dynamic loading events of a page and capture newly loaded resources and nodes. The captured dynamic loading information is organized into page dynamic loading data, including newly added nodes, resources, etc. For example, JavaScript is used to operate the DOM to capture a snapshot of the DOM tree after dynamic loading and obtain the structural information of the newly added nodes. The dynamic DOM tree snapshot data is compared with the initial DOM tree structure feature data to detect changes in the node structure, including addition, modification, deletion, etc. A timestamp is added to each page dynamic structure change data to mark the time point when the node change occurs. The page dynamic structure change data with a timestamp is organized into time-series dynamic structure change data to form a time series record. The time-series dynamic structure change data is processed, and the label frequency of newly added, modified, and deleted nodes in each time period is counted to obtain the frequency data of the node change type. According to the label frequency data, select labels with higher frequency of occurrence, and try to combine adjacent labels to form a dynamic label combination. For example, select labels with higher frequency of occurrence such as 、 , and try to combine these adjacent tags into a new dynamic tag, such as <divp>For each dynamic tag combination, the tag is annotated according to the tag information before and after the combination, and more specific semantics are given to it. <divp> , marked as< / divp> < / divp> <section> , giving more specific semantics. Based on the preset graph neural network model (such as GCN, GAT, etc.), build a transfer learning model for label recognition. For example, select GCN as the graph neural network model to build a transfer learning model for label recognition. Use valid label relationship graph data and dynamic label annotation data for transfer learning training, and adjust model parameters to adapt to new dynamic label scenarios. After the training is completed, a label recognition model that adapts to dynamic label scenarios is obtained. Input the time series dynamic structure change data into the dynamic label recognition model to obtain the brand word-of-mouth label recognition results at each time point.

[0138] Preferably, step S3 comprises the following steps:

[0139] Step S31: performing real-time element recognition on brand word-of-mouth tag data through a preset browser plug-in architecture to generate real-time web page element data;

[0140] Step S32: performing adaptive collection optimization according to real-time web page element data to generate an adaptive page collection strategy;

[0141] Step S33: formulating a sampling task for the real-time web page element data according to a preset web page data collection rule to obtain sampling task data;

[0142] Step S34: Perform distributed queue processing on the sampling task data to generate distributed sampling task data;

[0143] Step S35: Based on the distributed sampling task data, multimodal brand data is collected through an adaptive page collection strategy to generate multimodal brand data.

[0144] The present invention uses a browser plug-in architecture to perform element recognition on brand word-of-mouth tag data in real time, which helps to capture elements that change in real time in the page and generate real-time web page element data. Adaptive collection optimization is performed using real-time web page element data. This includes optimizing strategies such as determining the timing, frequency, and method of collection to adapt to the dynamic changes of different web pages and generate more flexible and efficient adaptive page collection strategies. Based on preset web page data collection rules, sampling tasks are formulated for real-time web page element data, which helps to determine the scope, depth, and frequency of sampling. Processing sampling task data through distributed queues to achieve parallel processing of tasks helps to improve collection efficiency and ensure that sampling tasks can be efficiently executed in a distributed environment. Combining distributed sampling task data and adaptive page collection strategies, the collection of multimodal brand data is implemented to ensure that the collected data is comprehensive and diverse, and has a comprehensive understanding of brand multimodal information.

[0145] As an example of the present invention, refer to Figure 3 As shown, Figure 1 Detailed implementation steps of step S3 in the flowchart, in this example, step S3 includes:

[0146] Step S31: performing real-time element recognition on brand word-of-mouth tag data through a preset browser plug-in architecture to generate real-time web page element data;

[0147] In an embodiment of the present invention, the architecture of the browser plug-in is designed and preset to ensure that it can obtain and process brand word-of-mouth tag data in real time. The rules and algorithms for real-time element recognition are configured in the plug-in to adapt to the characteristics of the brand word-of-mouth tag data and ensure real-time and accuracy. For example, configure the rules to define the elements that need to be identified in the brand word-of-mouth tag data, such as specific HTML tags or CSS classes. An interface for receiving brand word-of-mouth tag data is set in the plug-in to ensure that the latest data from the back end can be received in real time. For example, when a user is browsing a web page, the plug-in identifies the elements corresponding to the brand word-of-mouth tag data in real time according to the configured rules, such as comment boxes, rating stars, etc.

[0148] Step S32: performing adaptive collection optimization according to real-time web page element data to generate an adaptive page collection strategy;

[0149] In an embodiment of the present invention, based on real-time web page element data, the user's focus and collection requirements for brand word-of-mouth tags are analyzed to formulate an adaptive collection strategy. Different focus weights are assigned to different types of real-time elements to reflect the user's emphasis on each element. According to the analysis results and the focus weights, the collection frequency of brand word-of-mouth tag data is adjusted to update the focus more frequently. For example, the collection frequency of comment box data is adjusted to ensure that the latest user comments are obtained more frequently. The adjusted collection frequency and focus weights are organized into an adaptive page collection strategy.

[0150] Step S33: formulating a sampling task for the real-time web page element data according to a preset web page data collection rule to obtain sampling task data;

[0151] In the embodiment of the present invention, web page data collection rules are set, including sampling frequency, collection elements, collection depth, etc., to ensure that the system can obtain real-time web page element data in a prescribed manner. For example, it is stipulated that data is collected every 15 minutes, and comment boxes, rating stars, and related product information are collected. According to the formulated rules, sampling tasks for real-time web page element data are formulated, and the goals and time points of the collection tasks are clarified.

[0152] Step S34: Perform distributed queue processing on the sampling task data to generate distributed sampling task data;

[0153] In an embodiment of the present invention, a distributed queue architecture is designed and prepared to ensure that it can effectively process a large amount of sampling task data. For example, a distributed queue structure is designed, including modules such as task distribution, processing, and result storage. The sampling task data is queued according to the requirements of the queue architecture, so that it enters the distributed queue system. For example, the comment box and rating star rating element data obtained from the first sampling task are queued according to the task ID. The queue system distributes the task data to different processing nodes according to the set task distribution rules for parallel processing.

[0154] Step S35: Based on the distributed sampling task data, multimodal brand data is collected through an adaptive page collection strategy to generate multimodal brand data.

[0155] In an embodiment of the present invention, distributed sampling task data is used to analyze the results of each task and determine the strategy for multimodal brand data collection to adapt to the characteristics of different pages. The multimodal data collection task is sent to the task node in the distributed sampling task data to ensure that each node performs the task according to the strategy. For example, the multimodal data collection task is sent to the task node that has processed the comment box and the rating star rating, and requests the collection of relevant data. Each task node performs the multimodal data collection operation according to the task requirements and uploads the collected data to the central server.

[0156] Preferably, step S32 includes the following steps:

[0157] Step S321: Perform user interaction monitoring based on real-time web page element data to generate user interaction monitoring data.

[0158] Step S322: Generate a page heat map using the user interaction monitoring data, thereby obtaining corrected heat map data;

[0159] Step S323: Formulate a page collection strategy based on the modified heat map data to generate an initial page collection strategy;

[0160] Step S324: extracting behavior key indicator data from the modified heat map data to generate behavior key indicator data;

[0161] Step S325: performing importance evaluation processing according to the behavior key indicator data to generate key indicator evaluation data;

[0162] Step S326: Adaptively update the acquisition weight according to the key indicator evaluation data to obtain real-time acquisition feature weight data;

[0163] Step S327: dynamically adjusting the preset initial acquisition rate using the real-time acquisition feature weight data to obtain dynamic acquisition rate data;

[0164] Step S328: Adaptively optimize the initial page acquisition strategy through dynamic acquisition rate data to generate an adaptive page acquisition strategy.

[0165] In an embodiment of the present invention, user interaction monitoring rules are defined according to interactive elements in real-time web page element data, such as button clicks, input box filling, etc. A script is embedded in a web page to implement a user interaction monitoring function. When a user triggers an action in an interaction rule, the corresponding information is recorded. A page heat map is generated according to the user interaction monitoring data to reflect the hot spots of the user on the web page. For example, according to the frequency and position of the user clicking the comment button, an algorithm is designed to generate a page heat map to emphasize the area of ​​user attention. The heat map data is analyzed and corrected to understand the hot spots of user attention and the elements of frequent interaction. For example, the corrected heat map data is parsed to confirm that the user has more interactions in the comment button area. The defined behavior key indicator data is extracted using the corrected heat map data. For example, the number of times the user clicks the comment button and the click rate of the comment button are extracted from the corrected heat map data. According to the defined importance standard, the behavior key indicator data is evaluated and processed to obtain the importance score of each indicator. According to the key indicator evaluation data, the weights of different features in the collection are updated to adapt to changes in user behavior. The real-time collection feature weight data is used to adjust the initial collection rate to make it closer to the actual user behavior, and the dynamic collection rate data is obtained. For example, according to the increase in the real-time collection feature weight of the comment box element, the collection rate of the comment box element is adjusted to 40%. According to the increase in the dynamic collection rate of the comment box element, the collection rate of the comment box element is adjusted to 40%, and an adaptive page collection strategy is generated.

[0166] Preferably, step S322 includes the following steps:

[0167] Step S3221: performing interaction event classification processing on the user interaction monitoring data to obtain interaction event classification data, wherein the interaction event classification data includes user interaction behavior event data, mouse movement track event data and keyboard input event data;

[0168] Step S3222: performing time series behavior processing according to the mouse movement trajectory event data to generate time series behavior point data; performing keyword extraction according to the keyboard input event data to generate input keyword data;

[0169] Step S3223: performing cross-trajectory analysis on the time series behavior point data to generate interest area trajectory data;

[0170] Step S3224: calculating the interest depth of the interest region trajectory data by using an interest depth algorithm, thereby obtaining interest depth data;

[0171] Step S3225: Generate a page heat map for the interest depth data through the user interaction behavior event data to generate page heat map data;

[0172] Step S3226: Use the input keyword data to perform heat map correction on the page heat map data to generate corrected heat map data.

[0173] In an embodiment of the present invention, user interaction monitoring data is classified into user interaction behavior events, mouse movement trajectory events, and keyboard input events. For example, a mouse click is defined as a user interaction behavior event, a mouse movement is defined as a mouse movement trajectory event, and a keyboard key is defined as a keyboard input event. The mouse movement trajectory event data is processed in time series, the mouse position information at key moments is extracted, and time series behavior point data is generated. For example, the mouse position information within a certain time interval is extracted from the mouse movement trajectory to obtain time series behavior point data. Keyword extraction is performed on the keyboard input event data, the user's keyboard input is recorded, and input keyword data is obtained. Cross-trajectory analysis is performed on the time series behavior point data to extract interest area trajectory data that meets the conditions. For example, the time series behavior point data is analyzed to find the situation where the mouse stays in a certain area for more than 5 seconds, and the interest area trajectory data is obtained. The defined interest depth algorithm is applied to the interest area trajectory data to calculate the interest depth score of each area. The user interaction behavior event data is obtained, including clicks, scrolling and other behaviors, and the interest depth data is combined with the user interaction behavior event data to generate page heat map data. According to the input keyword data, the page heat map data is modified to increase the heat of the area related to the input keyword. For example, it is stipulated that the heat of the area related to the input keyword is increased.

[0174] Preferably, the interest depth algorithm formula in step S3224 is as follows:

[0175]

[0176] Where D(i) represents the interest depth data of the i-th interest area, i represents the index value of the interest area, t represents the residence time of the i-th interest area, T represents the total residence time of the user on the page, e represents a natural constant, k represents a time sensitivity parameter, and T i is represented as the attention time threshold of the i-th interest area, n is represented as the number of time series behavior points, j is represented as the index value of the time series behavior point, and w j It is expressed as the weight of the jth time series behavior point, x ij It is represented as the interaction intensity value of the user in the i-th interest area to the j-th behavior point, It is represented as the average interaction strength value of all interest areas to the jth behavior point, and L is represented as the total depth value of the web page.

[0177] The present invention utilizes an interest depth algorithm, the algorithm formula of which fully considers the index value i of the interest area, the residence time t of the i-th interest area, the total residence time T of the user on the page, the natural constant e, the time sensitivity parameter k, the attention time threshold T of the i-th interest area i , the number of timing behavior points n, the index value of the timing behavior point j, the weight of the jth timing behavior point w j , the interaction intensity value x of the user in the i-th interest area to the j-th behavior point ij , the average interaction strength value of all interest areas to the jth behavior point The total depth value L of the web page and the interaction relationship between the functions form a functional relationship:

[0178] Right now, By considering the residence time t of the i-th region of interest, it directly affects the calculation of the depth of interest. is a logistic function, which is used to simulate the growth and saturation of user interest. The parameter k controls the growth rate of the function. When t is close to T i hour, The value of will become smaller, making the value of the entire expression close to 1, which means that the user's interest is close to saturation. On the contrary, when t is much smaller than T i hour, The value of will become larger, and the value of the entire expression is close to 0, indicating that the user's interest is not yet saturated. It can measure the concentration and diversity of the user's interactive behavior in the i-th area of ​​interest. is the sum of the weighted interaction intensities of all temporal behavior points of the user in the i-th interest area. Here, w j is the weight of the jth time series behavior point, x ij is the user's interaction intensity value for the jth behavior point. This sum expresses the user's total interaction intensity in the area of ​​interest. is the user's interaction intensity and average interaction intensity for all temporal behavior points in the i-th interest area The square root of the sum of the squared differences. This expression calculates the standard deviation of the user interaction strength, reflecting the variability of user interaction behaviors. Combining these two parts to get a ratio and then taking its logarithm (base 2) shows that if the weighted sum of the interaction strengths increases or the variability of the interaction strengths decreases, the value of the entire expression will increase. It can simulate the distribution of user interest in web page depth. When t is 0, that is, when the user just starts browsing the web page, the value of the sin function is 0, indicating the beginning of interest. When t reaches L, the value of the sin function reaches its maximum, indicating that the user's interest reaches its peak. When t continues to increase to L, the value of the sin function returns to 0 again, indicating the end of a cycle. This periodic change can understand the dynamic changes in user interest when browsing web pages. For example, if the content of a web page is gradually unfolded, the user's interest will increase with in-depth reading, reach a certain high point, and then decrease with the end of the content. Finding the limit of t indicates the trend of the depth of interest when the user's stay time t is close to the total stay time T of the page. If t can be close to or equal to T, it means that the user is very interested in the area and the depth of interest reaches its maximum value.

[0179] Preferably, step S4 comprises the following steps:

[0180] Step S41: performing data standardization processing on the multimodal brand data to generate standard multimodal brand data;

[0181] Step S42: performing multimodal data fusion according to standard multimodal brand data, thereby obtaining fused brand data;

[0182] Step S43: performing natural language processing on the fused brand data to generate fused brand text data;

[0183] Step S44: extracting brand word-of-mouth keywords from the fused brand text data using a preset word-of-mouth keyword extraction rule to generate word-of-mouth keyword data;

[0184] Step S45: performing feature semantic analysis on the fused brand text data through word-of-mouth keyword data;

[0185] Step S46: using a sentiment analysis algorithm to evaluate the brand tendency of the semantic analysis data and generate a brand sentiment tendency score;

[0186] Step S47: Predict product word-of-mouth based on the brand sentiment tendency score to generate product word-of-mouth prediction data.

[0187] In an embodiment of the present invention, the multimodal brand data is standardized, for example, the image data is pixel standardized, and the text data is length standardized. Feature extraction is performed on the image data and the text data to capture their respective key information. For example, a convolutional neural network (CNN) is used to extract image features, and a natural language processing technology is used to extract text features. An appropriate fusion algorithm, such as a multimodal fusion network, is used to effectively fuse the image features and the text features. For example, the weight of the fusion model is optimized by a back propagation algorithm to improve the overall fusion performance. The trained fusion model is applied to the standard multimodal brand data to generate fused brand data. The cleaned text is segmented and the text is split into a sequence of words. Parts of speech are tagged for each word to understand the grammatical role of each word in the sentence. The text after word segmentation and part of speech tagging is converted into a word vector representation for the application of models such as deep learning. Using sentiment analysis technology, the text is judged for sentiment polarity to obtain positive, negative or neutral sentiment tendencies. The brand text data is fused to extract word-of-mouth keywords, for example, noise data such as HTML tags and special characters are removed, and the text is split into a sequence of words using a Chinese word segmentation tool. Perform part-of-speech tagging on each word, use the word embedding model to convert the text after word segmentation and part-of-speech tagging into a word vector representation, and apply the sentiment analysis model to determine the sentiment polarity of the text. Use word-of-mouth keyword data to extract keywords from the fused brand text data. For example, extract word-of-mouth keywords such as "good quality" and "service attitude" from the fused brand text data. According to the extracted keywords, analyze their feature expressions in the text to form feature semantic analysis data. For example, identify "good quality" as a positive feature and "service attitude" as a positive feature. Use the sentiment analysis algorithm to evaluate the sentiment polarity of each feature in the semantic analysis data, and combine the sentiment analysis results of each feature to calculate the overall sentiment tendency score of the brand. According to the set threshold, classify the brand sentiment tendency score to generate product word-of-mouth prediction data. For example, according to the threshold, classify the brand sentiment tendency score of 0.8 as positive word-of-mouth, and generate product word-of-mouth prediction data.

[0188] Preferably, the sentiment analysis algorithm formula in step S46 is as follows:

[0189]

[0190] In the formula, S represents the brand sentiment tendency score, e represents the natural constant, m represents the total number of feature elements in the semantic analysis data, a represents the index value of the feature element, and w a is represented as the weight value of the i-th feature element, K is represented as the emotional sensitivity coefficient, P a It is expressed as the number of positive word-of-mouth of the a-th feature element, N a is represented by the number of negative word-of-mouth of the ath feature element, γ is represented by the sentiment polarity adjustment factor, and θ a Represented as the sentiment polarity angle value of the ath feature element, R a Represented as the relevance score of the a-th element.

[0191] The present invention utilizes a sentiment analysis algorithm, which fully considers the total number m of feature elements in the semantic analysis data, the index value a of the feature element, the weight value w of the i-th feature element, and the weight value m of the i-th feature element. a , emotional sensitivity coefficient K, the number of positive word-of-mouth of the ath characteristic element P a , the number of negative word-of-mouth of the ath feature element N a , sentiment polarity adjustment factor γ, sentiment polarity angle value θ of the ath feature element a , the relevance score R of the ath element a And the interaction between functions to form a functional relationship:

[0192] Right now, Through the sigmoid function The algorithm is based on x in the sigmoid function, which is the sum of the calculation results of all feature elements. The shape of this function is similar to an "S" shape, which can map any input value to an output value between 0 and 1. e is a natural constant used as the base of the exponential function. It plays the role of a smooth curve in the sigmoid function to ensure a smooth transition of the sentiment tendency score. During the calculation process, we first need to determine the total number of feature elements m. These feature elements are extracted from the semantic analysis data. Each feature element represents an aspect of the brand, such as product quality, customer service, price, etc. For each feature element a, we will have a weight value w a , this weight value determines the influence of the feature in the total score. The weight value is usually set based on the importance of the feature, which can be obtained through historical data analysis or expert opinion. The sentiment sensitivity coefficient K is a tuning parameter that affects the sensitivity of the sentiment score to the number of positive and negative word-of-mouth. The number of positive word-of-mouth P a and the number of negative word-of-mouth N a It is collected through online comments and feedback from consumers, and directly reflects the public's views on specific characteristics of the brand. a Together, they are used to adjust the direction and degree of sentiment. These two parameters help the algorithm distinguish between positive and negative sentiment and give them appropriate weights. The scores are added up and then converted into a sentiment tendency score S through the sigmoid function to calculate a comprehensive score that reflects the public's overall emotional attitude towards the brand.

[0193] The beneficial effect of the present application is that, in the data collection process, the target web page data is first obtained through HTTP request, and the web page content is analyzed. The response code is parsed, and information such as text, links, and pictures are extracted from the source code to generate an initial DOM tree snapshot. This helps to understand the overall structure of the web page and the layout of HTML elements. Through the analysis of inter-cluster structural features, the relationship and interaction between different DOM elements can be understood. Monitor the dynamic loading events of the page, record the dynamic changes of the page after the user operation, and obtain complete page information including asynchronous loading data. Perform node structure change detection on the dynamically loaded data and track the addition, deletion and change of page elements. Integrate these changes to form a high-level tag combination, perform brand word-of-mouth tag identification, and provide accurate and targeted information. Design a browser plug-in to process data in real time, and dynamically capture new elements such as comments, ratings, pictures, etc. according to user behavior and changes in web page content. Adjust the collection strategy according to the actual situation, dynamically adjust the crawling frequency, depth and method, and improve the efficiency and accuracy of data collection. Utilize adaptive collection strategies, combine multiple data modalities, such as text, images, videos, etc., and form comprehensive brand information through multimodal data fusion. Perform feature semantic analysis on the fused data, mine the semantic information of brand data, extract key features, and understand implicit information and emotional tendencies. Through semantic analysis of data, evaluate the overall tendency of brand reputation, predict the development trend of product reputation, and provide decision-making basis for brand management and market analysis.

[0194] Therefore, the embodiments should be regarded as illustrative and non-restrictive from all points, and the scope of the present invention is limited by the appended claims rather than the above description, and it is therefore intended that all changes falling within the meaning and range of equivalent elements of the application documents are included in the present invention.

[0195] The above description is only a specific embodiment of the present invention, so that those skilled in the art can understand or implement the present invention. Various modifications to these embodiments will be apparent to those skilled in the art, and the general principles defined herein may be implemented in other embodiments without departing from the spirit or scope of the present invention. Therefore, the present invention will not be limited to the embodiments shown herein, but should conform to the widest scope consistent with the principles and novel features invented herein.< / section> < / section>

Claims

1. A method for automatic data collection based on web page tag analysis, characterized in that: The following steps are involved: Step S1: Obtain target collection web page data; Parse the response code of the target collected web page data and generate the initial DOM tree snapshot data; The inter-cluster structural features are extracted according to the initial DOM tree snapshot data, thereby obtaining the initial DOM tree structural feature data; wherein step S1 is specifically as follows: Step S11: Acquire target collection web page data; Step S12: performing initial web page response processing according to the target collected web page data to generate initial page response data; Step S13: parsing the response code of the initial page response data to generate initial DOM tree snapshot data; Step S14: constructing a valid tag graph according to the initial DOM tree snapshot data, thereby obtaining valid tag relationship graph data; Step S15: Calculate the optimal number of clusters for the effective label relationship graph data using the preset internal cluster density data to generate optimal number of clusters data; Step S16: Perform hierarchical clustering analysis based on the optimal cluster quantity data to generate node hierarchical clustering data; Step S17: extracting inter-cluster structural features from the node hierarchical clustering data, thereby obtaining initial DOM tree structural feature data; Step S2: perform page dynamic loading monitoring according to the initial DOM tree structure feature data to generate page dynamic loading data; perform snapshot node structure change detection on the page dynamic loading data to generate page dynamic structure change data; perform dynamic tag combination processing according to the page dynamic structure change data to obtain dynamic tag combination data; perform brand word-of-mouth tag recognition according to the dynamic tag combination data to generate brand word-of-mouth tag data; wherein step S2 is specifically as follows: Step S21: Monitor page dynamic loading according to the initial DOM tree structure feature data to generate page dynamic loading data; Step S22: capturing a dynamic DOM tree snapshot of the page dynamically loaded data to generate dynamic DOM tree snapshot data; Step S23: Detecting snapshot node structure changes of the initial DOM tree structure feature data using the dynamic DOM tree snapshot data to generate page dynamic structure change data; Step S24: Timestamp the page dynamic structure change data to generate time series dynamic structure change data; Step S25: extracting the tag frequency from the time series dynamic structure change data to generate web page tag frequency data; Step S26: performing dynamic tag combination processing according to the web page tag frequency data to obtain dynamic tag combination data; performing tag annotation processing according to the dynamic tag combination data to generate dynamic tag annotation data; Step S27: Based on the preset graph neural network model, perform label recognition transfer learning on the dynamic label annotation data through the effective label relationship graph data to generate a dynamic label recognition model; Step S28: transmitting the time series dynamic structure change data to the dynamic tag recognition model to perform brand word-of-mouth tag recognition and generate brand word-of-mouth tag data; Step S3: Perform real-time element recognition on brand word-of-mouth tag data to generate real-time web page element data; perform adaptive collection optimization based on real-time web page element data to generate an adaptive page collection strategy; perform multimodal brand data collection based on the adaptive page collection strategy to generate multimodal brand data; wherein step S3 is specifically as follows: Step S31: performing real-time element recognition on brand word-of-mouth tag data through a preset browser plug-in architecture to generate real-time web page element data; Step S32: performing adaptive collection optimization according to real-time web page element data to generate an adaptive page collection strategy; Step S33: formulating a sampling task for the real-time web page element data according to a preset web page data collection rule to obtain sampling task data; Step S34: Perform distributed queue processing on the sampling task data to generate distributed sampling task data; Step S35: performing multimodal brand data collection based on the distributed sampling task data through an adaptive page collection strategy to generate multimodal brand data; Step S4: performing multimodal data fusion according to the multimodal brand data to obtain fused brand data; performing feature semantic analysis on the fused brand data to generate semantic analysis data; performing brand tendency evaluation according to the semantic analysis data, and performing product word-of-mouth prediction, thereby obtaining product word-of-mouth prediction data; wherein step S4 is specifically as follows: Step S41: performing data standardization processing on the multimodal brand data to generate standard multimodal brand data; Step S42: performing multimodal data fusion according to standard multimodal brand data, thereby obtaining fused brand data; Step S43: performing natural language processing on the fused brand data to generate fused brand text data; Step S44: extracting brand word-of-mouth keywords from the fused brand text data using a preset word-of-mouth keyword extraction rule to generate word-of-mouth keyword data; Step S45: performing feature semantic analysis on the fused brand text data through word-of-mouth keyword data; Step S46: using a sentiment analysis algorithm to evaluate the brand tendency of the semantic analysis data and generate a brand sentiment tendency score; Step S47: Predict product word-of-mouth based on the brand sentiment tendency score to generate product word-of-mouth prediction data.

2. The automatic data collection method based on web page tag analysis according to claim 1 is characterized in that: Step S14 includes the following steps: Step S141: performing tree-level traversal on the initial DOM tree snapshot data to generate tree-level traversal data; Step S142: performing node type classification according to the tree level traversal data to generate node type data; Step S143: performing a valid label node check on the node type data to obtain valid label node data; Step S144: recursively traverse the child nodes of the valid label node data to generate parent-child node relationship data; traverse the same-level nodes of the valid label node data to generate brother node relationship data; Step S145: constructing a graph data structure for the valid label node data through the parent-child node relationship data and the sibling node relationship data to generate node graph structure data; Step S146: Draw a directed label relationship graph according to the node graph structure data to generate valid label relationship graph data.

3. The automatic data collection method based on web page tag analysis according to claim 1 is characterized in that: Step S32 includes the following steps: Step S321: Perform user interaction monitoring based on real-time web page element data to generate user interaction monitoring data. Step S322: Generate a page heat map using the user interaction monitoring data, thereby obtaining modified heat map data; Step S323: Formulate a page collection strategy based on the modified heat map data to generate an initial page collection strategy; Step S324: extracting behavior key indicator data from the modified heat map data to generate behavior key indicator data; Step S325: performing importance evaluation processing according to the behavior key indicator data to generate key indicator evaluation data; Step S326: Adaptively update the acquisition weight according to the key indicator evaluation data to obtain real-time acquisition feature weight data; Step S327: dynamically adjusting the preset initial acquisition rate using the real-time acquisition feature weight data to obtain dynamic acquisition rate data; Step S328: Adaptively optimize the initial page acquisition strategy through dynamic acquisition rate data to generate an adaptive page acquisition strategy.

4. The automatic data collection method based on web page tag analysis according to claim 3 is characterized in that: Step S322 includes the following steps: Step S3221: performing interaction event classification processing on the user interaction monitoring data to obtain interaction event classification data, wherein the interaction event classification data includes user interaction behavior event data, mouse movement track event data and keyboard input event data; Step S3222: performing time series behavior processing according to the mouse movement trajectory event data to generate time series behavior point data; performing keyword extraction according to the keyboard input event data to generate input keyword data; Step S3223: performing cross-trajectory analysis on the time series behavior point data to generate interest area trajectory data; Step S3224: calculating the interest depth of the interest region trajectory data by using an interest depth algorithm, thereby obtaining interest depth data; Step S3225: Generate a page heat map for the interest depth data through the user interaction behavior event data to generate page heat map data; Step S3226: Use the input keyword data to perform heat map correction on the page heat map data to generate corrected heat map data.

5. The automatic data collection method based on web page tag analysis according to claim 4 is characterized in that: The interest depth algorithm formula in step S3224 is as follows: Where D(i) represents the interest depth data of the i-th interest area, i represents the index value of the interest area, t represents the residence time of the i-th interest area, T represents the total residence time of the user on the page, e represents a natural constant, k represents a time sensitivity parameter, and T i is represented as the attention time threshold of the i-th interest area, n is represented as the number of time series behavior points, j is represented as the index value of the time series behavior point, and w j It is expressed as the weight of the jth time series behavior point, x ij It is represented as the interaction intensity value of the user in the i-th interest area to the j-th behavior point, xˉ j It is represented as the average interaction strength value of all interest areas to the jth behavior point, and L is represented as the total depth value of the web page.

6. The automatic data collection method based on web page tag analysis according to claim 1 is characterized in that: The sentiment analysis algorithm formula in step S46 is as follows: In the formula, S represents the brand sentiment tendency score, e represents the natural constant, m represents the total number of feature elements in the semantic analysis data, a represents the index value of the feature element, and w a is represented as the weight value of the i-th feature element, K is represented as the emotional sensitivity coefficient, P a It is expressed as the number of positive word-of-mouth of the a-th feature element, N a is represented by the number of negative word-of-mouth of the ath feature element, γ is represented by the sentiment polarity adjustment factor, and θ a Represented as the sentiment polarity angle value of the ath feature element, R a Represented as the relevance score of the a-th element.

Images