Market information oriented automated data collection and decision support system

Abstract

The application discloses a market information-oriented automatic data collection and decision support system, and particularly relates to the field of artificial intelligence and information processing technology, and comprises four modules of structure rendering, semantic extraction, credibility evaluation and execution feedback. The system extracts structure anchor points through image rendering, generates semantic vectors in combination with context modeling, calculates structure consistency index and semantic effectiveness index, outputs five-level credibility level results through a fuzzy logic device, refreshes collection task scheduling priority, and updates a pre-training language model based on semantic offset feedback, so as to realize self-iterative semantic generalization and efficient decision support under cross-platform context. The application realizes stable identification of webpage structure through structure rendering, realizes multi-dimensional semantic feature generation through semantic extraction, outputs credibility level in combination with double-index evaluation and fuzzy logic, and then optimizes collection scheduling and model adaptive updating, so as to effectively improve the acquisition accuracy of market information data and decision reliability.

Description

Technical Field

[0001] This invention relates to the fields of artificial intelligence and information processing technology, and more specifically, to an automated data collection and decision support system for market intelligence. Background Technology

[0002] With the development of the digital economy, market intelligence is increasingly demonstrating its strategic value in corporate competition, policy response, product iteration, and user insights. In particular, complex information carriers such as multi-source heterogeneous web pages, news portals, e-commerce platforms, government websites, and social media contain a wealth of unstructured content regarding market trends, user preferences, product feedback, and competitor activities. Traditional methods relying on manual or rule-driven information extraction are no longer adequate for the intelligence environment characterized by high-frequency updates, multi-platform distribution, and significant semantic differences.

[0003] Existing web page information collection systems generally rely on structured template parsing, depending on fixed DOM paths, tag identifiers, or regular expression rules to locate and extract target content. However, in real-world applications, frequent updates to web page structures, significant changes in module arrangements, and a gradually increasing proportion of dynamically rendered content lead to template failures and insufficient stability in information extraction. Furthermore, existing systems generally lack a deep understanding of the semantic layer of web page content, often remaining at the stage of keyword matching or static rule judgment, making it difficult to address semantic challenges such as paraphrasing, changes in sentiment, and ambiguous intent.

[0004] The lack of an effective and reliable evaluation mechanism after information collection is a significant shortcoming of current technology. Existing methods often focus on the completeness and response speed of the captured results, while neglecting the quality evaluation of the data itself in multiple dimensions such as structural stability and semantic consistency, resulting in high noise levels in the collected content and high analysis costs. Therefore, this invention proposes an automated data collection and decision support system for market intelligence to address the above problems. Summary of the Invention

[0005] To achieve the above objectives, the present invention provides the following technical solution:

[0006] An automated data acquisition and decision support system for market intelligence, including a structure rendering module, a semantic extraction module, a reliability assessment module, and an execution feedback module;

[0007] The structure rendering module acquires the visual output of the target webpage in an image rendering environment, constructs a visual structure map based on pixel density and layout sequence, identifies content clustering areas through a deep perception network, and outputs a sequence of structural anchor points.

[0008] The semantic extraction module performs semantic vector encoding on the text content corresponding to the structural anchor sequence, extracts semantic representations using a context-pre-trained language modeling network, and generates topic vectors, sentiment vectors, and intent feature vectors by combining the domain corpus annotation results.

[0009] The credibility assessment module receives structural anchor sequence and semantic vector set as input. The former calculates structural consistency index based on structural symmetry index, historical version offset rate and path stability factor, while the latter calculates semantic validity index based on semantic cosine similarity, topic aggregation degree and sentiment certainty. The two indices are fused into fuzzy logic to output five-level credibility classification results, including Level 1: Definitely Credible, Level 2: Strongly Credible, Level 3: Suspicious, Level 4: Weakly Credible and Level 5: Rejection.

[0010] The execution feedback module refreshes the acquisition scheduling priority by updating the output results of the fuzzy logic device. It configures the data entry scheduling weight according to the five-level trust level, prioritizes the execution of first-level and second-level resource acquisition tasks, and performs online training feedback on the semantic features in the acquisition results to enhance the adaptability of the pre-trained semantic modeling network and realize self-iterative semantic generalization in cross-platform contexts.

[0011] In a preferred embodiment, after the structure rendering module obtains the visual output of the target webpage in the image rendering environment, it performs an image level normalization operation, divides the webpage visual output into grids according to a preset pixel region window, and constructs an initial pixel density distribution map based on the pixel density gradient change in each window.

[0012] The pixel density distribution map is mapped to a layout sequence, which includes the window arrangement order, density trend and visual symmetry features;

[0013] The deep perception network is used to identify content clustering areas in the layout sequence. A region candidate mechanism is used to generate candidate bounding boxes for the content clustering areas, and duplicate candidate boxes are eliminated based on the visual similarity between boxes and the aggregation threshold.

[0014] By matching the positions of candidate bounding boxes with the layout sequence one-to-one, a sequence of structural anchor points that satisfy the structural characteristics of a stable layout is extracted. This sequence of structural anchor points serves as one of the inputs to the reliability evaluation module.

[0015] In a preferred embodiment, the deep perception network in the structure rendering module includes at least one convolutional coding layer, a position attention mapping layer, and an output prediction layer. The convolutional coding layer receives the joint representation matrix of the pixel density distribution map and the layout sequence, and extracts the multi-scale feature map of the content region. The position attention mapping layer captures the content clustering pattern in the layout sequence through a cross-window comparison mechanism and assigns attention weights to the content region. The output prediction layer generates the final content clustering region based on the position of the maximum weight.

[0016] The training process of this deep perception network uses manually labeled web page content regions as supervision signals and employs a joint optimization of the cross-entropy loss function and a position offset penalty term.

[0017] The structure rendering module establishes a set of anchor point coordinate vectors through the content aggregation area output by the network, and performs an anchor point sequence sorting operation on the set. The sorting criterion is a combined scoring function that prioritizes visual weight and layout stability.

[0018] The sorted structural anchor sequence is used as the text location input for the semantic extraction module, and the path identifier of the structural anchor sequence is retained in the trust evaluation module for calculating the structural consistency index.

[0019] In a preferred embodiment, during the process of performing semantic vector encoding on the text content corresponding to the structural anchor sequence, the semantic extraction module first maps the position of the structural anchor sequence to the text distribution area of ​​the original webpage document, extracts the context text fragments corresponding to each anchor, and uses each text fragment as input to complete the multi-layer vector generation of semantic representation through a context pre-trained language modeling network.

[0020] The context-pretrained language modeling network uses a bidirectional attention mechanism to model the context linkage of text segments. The input embedding vector is encoded by position and then input into a multi-head attention block to obtain the context semantic dependency matrix. The semantic representation is then output through a feedforward network.

[0021] This semantic representation serves as the feature base, is aligned with the domain corpus annotation results, and obtains topic vectors, sentiment vectors, and intent feature vectors through a vector mapping transformation function. The three vectors maintain independent representations within the semantic extraction module and serve as the input source of the semantic vector set relied upon by the credibility evaluation module when calculating the semantic validity index.

[0022] In a preferred embodiment, the semantic extraction module pre-establishes a domain corpus annotation mapping table when generating topic vectors, sentiment vectors, and intent feature vectors by combining the domain corpus annotation results. The mapping table includes domain keyword groups, topic category codes, sentiment polarity templates, and intent template sets. Semantic representations are generated by mapping through the corresponding matching rules in the mapping table.

[0023] The topic vector is generated based on the inner product matching score of the semantic representation and the topic category encoding. The sentiment vector is generated by the similarity aggregation function of the semantic representation and the sentiment polarity template. The intent feature vector is output by the maximum relevance matching of the semantic representation to the intent template set. The three vectors are generated sequentially in the semantic extraction module and merged into the semantic vector set.

[0024] In a preferred embodiment, the structural consistency index is calculated as follows:

[0025] Each anchor point in the structural anchor point sequence is regarded as a graph node. Adjacency relationships are constructed according to the visual rendering order to form a local structural subgraph. Based on the topological graph mapping mechanism, the structural subgraph is matched with the corresponding webpage structure generated in the historical version. The topological consistency score is obtained by calculating the proportion of isomorphic nodes and the degree of preservation of node arrangement order. This score measures whether the structure has structural continuity in terms of content position, relative order and hierarchy.

[0026] For each anchor point, a path offset sequence is constructed in multiple historical webpage versions. Based on the time series, its multidimensional path fluctuation trajectory is constructed. The current anchor point path and trajectory mean are subjected to residual analysis. By comparing the trend direction and amplitude variation of path changes, the path stability factor is obtained. This score reflects whether the structure has long-term stability under dynamic changes.

[0027] The structural consistency index is obtained by nonlinearly combining the topology consistency score and the path stability score. The combination method adopts a suboptimal suppression mechanism: if one of the scores is extremely low, the overall index is set to a decreasing range to prevent structural drift from being masked by high topology. This mechanism comes from the suppression scoring strategy in graph neural networks, which ensures the robustness of the scoring results and the ability to detect anomalies.

[0028] In a preferred embodiment, the semantic validity index is calculated as follows:

[0029] The semantic cosine similarity score S1 is calculated by the cosine angle between the semantic vector and the semantic prototype vector. If S1 is greater than 0.8, it is recorded as 1; otherwise, it is linearly amplified by multiplying S1 by 1.25. The topic aggregation score S2 is calculated by the proportion of the number of the same topic codes in the set of semantic vectors. If the proportion exceeds 60%, S2 is equal to 1; otherwise, it is asymmetrically mapped by multiplying the actual proportion by 1.5. The sentiment certainty score is calculated based on the standard deviation σ of the sentiment dimension within the semantic vector. If σ is less than 0.1, it is recorded as 1. If σ is between 0.1 and 0.3, the score S3 is reduced by subtracting σ from 1. If σ is greater than 0.3, S3 is zero. The semantic validity index is finally generated by the minimum constraint function F of S1, S2, and S3, i.e., F = min(S1, S2, S3).

[0030] In a preferred embodiment, the trust assessment module inputs the structural consistency index and the semantic validity index into the fuzzy logic device to classify the trust level. The fuzzy logic device executes the output of the five trust levels in a rule-driven manner. The logic device uses a two-dimensional fuzzy interval mapping table to construct the index response matrix. The value ranges of the structural consistency index and the semantic validity index are divided into five segments: low, medium-low, medium, medium-high, and high, which correspond to the numerical intervals of 0 to 0.2, 0.2 to 0.4, 0.4 to 0.6, 0.6 to 0.8, and 0.8 to 1, respectively. The fuzzy logic device uses the cross interval combination of the two indices as the input state, forming a total of twenty-five mapping cases.

[0031] When both indices are in the high range, the output is Level 1: Definitely Trustworthy. When the structural consistency index is in the medium-high range and the semantic validity index is in the high range, the output is Level 2: Strongly Trustworthy. When either index is in the medium-low range and the other index is not higher than the medium-high range, the output is Level 3: Suspicious. When either of the two indices is below 0.2, the output is Level 4: Weakly Trustworthy. When both indices are below 0.2, the output is Level 5: Rejected.

[0032] In a preferred embodiment, when the execution feedback module refreshes the acquisition scheduling priority based on the output result of the five-level trust level of the fuzzy logic, the module presets the five-level scheduling weights as 0.9, 0.7, 0.5, 0.2 and 0, respectively, corresponding to the five levels of trust from level one to level five rejection.

[0033] In the task collection queue, the execution feedback module assigns a corresponding scheduling weight to each task to be collected based on its level label, and sorts them according to the weight. Tasks with weights of 0.9 and 0.7 are given priority to be pushed into the scheduling execution sequence.

[0034] After the collection is completed, the execution feedback module automatically parses the semantic feature vectors in the collection results and takes the samples that have a cosine distance greater than the preset feedback threshold with the current semantic prototype vector of the pre-trained semantic modeling network as valid feedback data.

[0035] Valid feedback data is input into the online training process to adjust the encoding parameters of the semantic modeling network. A training-while-inference mechanism is adopted to maintain continuous model updates and real-time adaptation. The update cycle can be configured to automatically trigger training once every m collection tasks.

[0036] The technical effects and advantages of this invention are as follows:

[0037] This invention acquires the visual output of a target webpage in an image rendering environment through a structure rendering module, constructs a visual structure map, and utilizes a deep perception network to identify content clustering regions, thereby generating a sequence of structural anchor points. By jointly modeling pixel density and layout sequences, and extracting visually salient regions using a deep perception network, the system can reliably identify key regions with information density and structural features in real-world environments with complex webpage structures and diverse content layouts. This processing significantly improves the accuracy of webpage content extraction, laying a structural foundation for subsequent semantic encoding and credibility assessment.

[0038] In its semantic extraction module, this invention encodes the text content corresponding to structural anchors using semantic vectors. It then extracts semantic representations using a context-pre-trained language modeling network and combines this with domain corpus annotation results to generate topic vectors, sentiment vectors, and intent feature vectors. Through this process, the system not only accurately acquires the semantic information corresponding to structural anchors but also transforms text content into vector expressions with multi-dimensional features, achieving a mapping from raw webpage information to a computable semantic structure. This approach enhances the ability to extract implicit semantics from webpage intelligence and is particularly suitable for market content recognition tasks with complex contexts and diverse expressions.

[0039] This invention uses a trust assessment module to jointly evaluate the structural anchor sequence and semantic vector set, calculating the structural consistency index and semantic validity index respectively. A fuzzy logic device then fuses the two indices, outputting a five-level trust classification result. Subsequently, an execution feedback module refreshes the data acquisition scheduling priority based on this level result and performs online training feedback based on high-level tasks, thereby dynamically adjusting the parameters of the semantic modeling network. This two-layer mechanism not only achieves dynamic evaluation and fine-grained classification of the acquired data quality but also enhances the model's generalization ability in cross-platform contexts through feedback loops, realizing bidirectional optimization of acquisition behavior and model adaptability. Attached Figure Description

[0040] To facilitate understanding by those skilled in the art, the present invention will be further described below with reference to the accompanying drawings;

[0041] Figure 1 This is a schematic diagram of the automated data collection and decision support system for market intelligence in this invention. Detailed Implementation

[0042] The technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those of ordinary skill in the art without creative effort are within the scope of protection of the present invention.

[0043] Reference Figure 1 The following examples were obtained: Example

[0044] An automated data acquisition and decision support system for market intelligence includes a structure rendering module, a semantic extraction module, a reliability assessment module, and an execution feedback module; the modules are interconnected.

[0045] The structure rendering module acquires the visual output of the target webpage in an image rendering environment, constructs a visual structure map based on pixel density and layout sequence, identifies content clustering regions through a deep perceptron network, and outputs a sequence of structural anchor points. The module's role is to achieve high-precision identification of the visual structure and extraction of content regions from the target webpage. This process is based on image rendering, acquiring the webpage's visual output within an image environment, effectively bypassing the sensitivity to content changes inherent in traditional HTML static paths. By performing joint modeling of pixel density and layout sequence at the image level, this module can construct a structure map that reflects the page's true visual structure. Based on this, a deep perceptron network is used to extract features from the layout sequence and locate content clustering regions within the page. Finally, a sequence of structural anchor points is generated, allowing downstream modules to directly extract corresponding content using these anchor points as indexes, achieving robust information acquisition under conditions of webpage heterogeneity and dynamic changes. The structural anchor point sequence not only has clear spatial location but also embeds layout stability weights, providing a precise foundation for subsequent semantic modeling and evaluation.

[0046] The semantic extraction module performs semantic vector encoding on the text content corresponding to the structural anchor sequence. It extracts semantic representations using a context-pre-trained language modeling network and generates topic vectors, sentiment vectors, and intent feature vectors based on domain corpus annotation results. This module is responsible for extracting deep semantic features from the text content located at structural anchors. Its core function is to extract contextual text fragments corresponding to the anchor positions from the original webpage document, using structural anchors as reference points, and then vectorizing them through a context-pre-trained language modeling network. This encoding process utilizes a bidirectional attention mechanism to achieve semantic linkage modeling between contexts, enabling the generated semantic vectors to capture both local information and overall contextual awareness. Subsequently, the module aligns with the domain corpus annotation results, mapping the semantic representations into three independent categories: topic vectors, sentiment vectors, and intent feature vectors. Each category carries a corresponding semantic judgment task. This module ensures the structured extraction of multi-dimensional intelligence information implicit in the original webpage text and serves as the central hub for the system's semantic recognition and intelligence discrimination.

[0047] The credibility assessment module receives a sequence of structural anchor points and a set of semantic vectors as input. The former calculates a structural consistency index based on structural symmetry indicators, historical version offset rates, and path stability factors, while the latter calculates a semantic validity index based on semantic cosine similarity, topic aggregation degree, and sentiment certainty. These two indices are fused into a fuzzy logic unit to output a five-level credibility classification: Level 1: Certainly Credible; Level 2: Strongly Credible; Level 3: Suspicious; Level 4: Weakly Credible; and Level 5: Rejected. The module's function is to integrate structural information and semantic features to construct a complete content credibility judgment system. The first part, targeting the structural anchor point sequence, constructs a structural consistency index based on three dimensions: structural symmetry indicators, historical version offset rates, and path stability factors. This index measures the degree of matching between the page structure and its historical stability. The second part, targeting the set of semantic vectors, constructs a semantic validity index based on three indicators: semantic cosine similarity, topic aggregation degree, and sentiment certainty. This index is used to determine whether the text possesses semantic focus, clarity, and target consistency. Finally, both indices are input into a fuzzy logic unit for comprehensive decision-making. The fuzzy logic generator outputs a five-level confidence level classification based on the combined range of two indices, providing clear confidence labels for subsequent data collection strategies and resource scheduling. This module enables trustworthy modeling of webpage content from both structural and semantic dimensions, and is the core of the entire system's intelligent and high-precision judgment.

[0048] The execution feedback module refreshes the collection scheduling priority based on the fuzzy logic generator output, configures data entry scheduling weights according to a five-level trust level, prioritizes the execution of level one and level two resource collection tasks, and performs online training feedback on the semantic features in the collection results. This enhances the adaptability of the pre-trained semantic modeling network and enables self-iterative semantic generalization in cross-platform contexts. The main task of the execution feedback module is to optimize the configuration of collection resources based on the trust level results and achieve adaptive evolution of the semantic model. This module maps the five-level trust level results output by the fuzzy logic generator to five-level scheduling weights and refreshes the execution priority of each task in the collection task queue accordingly. Higher weights are collected first, allowing the system to focus on high-value, high-confidence resources. Simultaneously, after collection, the module automatically analyzes the semantic feature vectors in the collection results, filters out samples with significant deviations from the current semantic modeling network as effective feedback, and inputs them into the online training mechanism for model fine-tuning. This training-and-inference mechanism enables the model to gradually build robust semantic generalization capabilities across multiple platforms, contexts, and expression forms, thereby significantly improving the overall system's adaptability and response speed to new intelligence sources and new webpage styles.

[0049] After acquiring the visual output of the target webpage in the image rendering environment, the structure rendering module first performs image layer normalization. The image rendering environment refers to the actual visual image generated after the webpage is loaded through the browser kernel or visualization rendering engine; its output is pixel-level image data, not traditional webpage structure code. Image layer normalization refers to standardizing this visual output under different display resolutions, scaling ratios, or browser environments, so that it expresses the page content distribution in a unified spatial coordinate system, facilitating subsequent region identification and comparison.

[0050] The normalized image is divided into multiple preset pixel region windows. A pixel region window is a fixed-size image slice unit, preset by the system (e.g., 64×64 pixels or 128×128 pixels), and is evenly distributed across the entire webpage image in a sliding or grid manner. Within each pixel region window, pixel density gradient change analysis is performed. This analysis evaluates the pixel brightness variation trend in different areas of the window, calculates density gradient statistics, and generates a measure of the content density of that window within the overall image.

[0051] Based on the pixel density gradient calculation results of all windows, the structure rendering module constructs an initial pixel density distribution map. This distribution map represents the degree of pixel aggregation in different areas of the entire page in the form of a two-dimensional matrix. The higher the value, the denser the content.

[0052] The pixel density distribution map is then mapped to a layout sequence. A layout sequence is an ordered chain of features that encodes the overall visual structure of a page. It consists of multiple layout descriptors arranged in the visual order of the page, with each descriptor corresponding to a pixel region window, reflecting the window's position, density attributes, and structural relationships within the page. This sequence serves as a crucial intermediate representation in the structure rendering module, carrying the structural cues required for subsequent region recognition tasks.

[0053] The three dimensions of information in the layout sequence are as follows:

[0054] Window arrangement order: This represents the visual rendering order of pixel region windows on the page from top to bottom and from left to right. It ensures that the model can understand the page structure according to the human visual flow pattern and adapt to the layout inertia of natural language or visual information from top to bottom and from left to right. This order is converted into a one-dimensional sequence index using two-dimensional image coordinates.

[0055] Density trend: Reflects the trajectory of pixel density changes within consecutive windows. By calculating the differences in density values ​​between adjacent windows and establishing a trend vector (e.g., +1 indicates increasing, -1 indicates decreasing, and 0 indicates stable), the system can identify structural signals in the image, such as obvious structural block boundaries, abrupt change points, and information hierarchy divisions. For example, when the density of multiple windows increases vertically, it usually indicates a visual organization of title—body text—key points.

[0056] Visual symmetry features: This assesses whether the page structure exhibits symmetry in the horizontal, vertical, or diagonal directions. Specifically, the page image is mirrored along the corresponding directions, and a symmetry score is calculated to form a quantifiable structural balance index. Symmetrical structures are commonly found on standardized pages such as news websites and government platforms, possessing high recognition value. The output of this layout sequence will serve as the dominant structural representation for subsequent deep learning network inputs, determining the basis for the model's focus area distribution.

[0057] The execution flow of deep learning network for identifying content clustering areas:

[0058] Next, a deep perceptron is used to analyze the layout sequence to identify potential content clustering regions. The deep perceptron is the core model in the structure rendering module for structure understanding and region focusing, and it comprises the following three functional levels:

[0059] Convolutional coding layer: Transforms the layout sequence into tensor input and extracts the local structural relationships between regions. During the sliding process of the convolutional kernel, it can capture patterns such as repeating blocks, aligned structures, and density abrupt changes, and output multi-scale feature maps, preserving spatial relationships.

[0060] Attention Mapping Layer: Based on the output of the convolutional coding layer, an attention weight matrix is ​​constructed across windows to measure the relative importance of each region. The attention mechanism guides the network to concentrate more computational resources on possible "content clustering regions" by calculating the feature similarity and positional relationship between regions.

[0061] Output prediction layer: Based on the attention distribution results, determine the set of regions with the most informational value, generate a preliminary content region heatmap, and then extract the coordinates of hotspot regions through a preset nonmaximum suppression algorithm as candidates for content clustering regions.

[0062] Content clustering areas refer to localized regions within web page images that have high information density and a high probability of user reading, such as the title area of ​​a news list page, the price and specifications area of ​​an e-commerce product page, and the introductory summary section of a policy announcement page. By effectively extracting these areas, the system can significantly improve the efficiency and accuracy of subsequent data collection and semantic analysis.

[0063] The structure rendering module performs a region candidate mechanism on the identified content cluster areas to form candidate bounding boxes. Candidate bounding boxes are image region localization boundaries constructed using rectangles as the basic unit, reflecting the focal location of specific suspicious information. The specific generation method is as follows: thresholding is performed on the heatmap of the content cluster area (e.g., areas with a heat value greater than 0.65 are marked as target areas); a minimum bounding rectangle is generated in the target area as the initial shape of the candidate bounding boxes; each candidate bounding box records its upper-left and lower-right coordinates, along with metadata such as heat value score, relative position index, and original window association number, forming the primary building block for subsequent structural anchor points.

[0064] Visual similarity aggregation and redundancy removal mechanism:

[0065] After candidate bounding boxes are generated, the structure rendering module performs visual similarity aggregation on all boxes to eliminate overlapping, redundant, or homogeneous content areas and improve anchor point quality. This aggregation process includes: color distribution comparison: statistically analyzing the RGB or HSV channel distribution of pixels within the box and using color histogram cross-validation to assess similarity; edge feature matching: extracting structural boundaries using edge detectors such as Canny and calculating similarity scores; texture consistency assessment: analyzing regional textures using gray-level co-occurrence matrix or local binary mode to assess differences; and spatial overlap rate assessment: using the IoU (Intersection over Union) metric to determine the degree of overlap between boxes. Box pairs that meet the above multiple similarity assessment dimensions are considered candidates of the same type if their aggregation score exceeds a set inter-box visual similarity aggregation threshold (e.g., 0.85). The pair with the higher score is retained, and the rest are discarded. This process ensures that only non-redundant and highly representative candidate regions are retained for subsequent structural anchor point extraction and further improves the accuracy of structural consistency assessment and semantic extraction.

[0066] The structure rendering module maps the retained candidate bounding boxes to window positions in the layout sequence, extracting anchor points that conform to stable layout structure characteristics and forming a structure anchor point sequence. The structure anchor point sequence is a uniquely located set of visual anchor points with three core attributes: image coordinates; the normalized position index within the page; and the pixel density label and structure confidence score of each anchor point. The structure anchor point sequence is not only the final output of this module but also one of the inputs to the credibility assessment module. It will be used to calculate the structure consistency index, which, together with the semantic vector set, determines the credibility level of the webpage information. The quality of this sequence directly affects the performance of subsequent semantic extraction and evaluation modules, making it the core structural entry point for the entire data acquisition process.

[0067] To achieve reliable quantification of anchor point selection, the structure rendering module assigns a structure confidence score to each anchor point. This score measures the stability and effectiveness of the anchor point as a structural representative area on the page. The score is calculated from the following two quantitative indicators: Pixel density intensity score, derived from the density value of the window containing the anchor point in the initial pixel density distribution map; using standard normalization, the density value is mapped to between zero and one; representing the visual focus of the anchor point. Layout position stability score, derived from whether the candidate marker box maintains a stable position in different historical versions of the webpage; statistically analyzing the frequency of occurrence and coordinate deviation of the area corresponding to the anchor point in the historical page structure; if the historical offset is less than a preset threshold, the score increases, indicating that it is a long-term stable structural component. The structure confidence score is synthesized as an unweighted product of the above two scores, defined as follows: Confidence score S = D × P, where D is the pixel density intensity score and P is the layout position stability score. This product method has a suboptimal constraint effect. When the score of any dimension is too low, the overall score is lowered to prevent the overall credibility from being masked by an excessively high score of a certain dimension.

[0068] After the aforementioned layout sequence and pixel density distribution map are constructed, the structure rendering module performs content region recognition on its joint representation (i.e., the structure map constructed above) through a deep perception network.

[0069] The depth-aware network in the structure rendering module comprises at least one convolutional coding layer, a positional attention mapping layer, and an output prediction layer. This network is used to identify content clustering areas on a webpage from the layout structure and pixel distribution. The network input is a joint representation matrix, which is composed of the pixel density distribution map and layout sequence constructed earlier. This matrix is ​​input into the network through matrix concatenation and channel fusion in the spatial dimension to ensure synchronous modeling of visual density and structural logic.

[0070] The convolutional coding layer performs multi-scale convolution operations, using multiple convolutional kernels of varying receptive field sizes to extract features from the input matrix, forming a multi-scale feature map containing high-density clustering, boundary variations, and local symmetry. These feature maps represent potential content region patterns at different resolution levels. The location attention mapping layer enhances the model's focus on key regions. This layer calculates the feature similarity between different windows, establishing a cross-window comparison mechanism to form an attention distribution weight map, indicating which locations are more likely to be content clustering regions within the overall page structure. Higher attention values ​​indicate greater structural saliency at that location. The output prediction layer, based on the aforementioned weight map, extracts the coordinates of the location with the highest weight value across the entire image and generates a set of bounding boxes for content clustering regions within their local neighborhoods. The output is a set of rectangular box coordinates, representing the regions in the image with the highest information density and structural probability.

[0071] The training process of this deep perception network uses manually labeled webpage content regions as supervision signals. Specifically, regions such as titles, summaries, and body text in webpages are manually labeled to form the training set. During training, the cross-entropy loss function is used to measure the consistency between the model output and the manually labeled regions, and a positional offset penalty term is introduced to constrain the distance between the model's output region and the ground truth labeled region in coordinate space. This penalty term ensures that the model maintains localization accuracy while accurately predicting content regions, forming a joint optimization objective. This joint optimization strategy includes two loss calculation mechanisms: classification accuracy loss and the positional offset penalty term, both of which work together to adjust the network parameters during training.

[0072] The classification accuracy loss is constructed using a discriminative loss method based on the log-likelihood principle to measure the network's accuracy in identifying content clusters. In each training sample image, the actual content regions on the page are manually labeled, and these labeled regions are designated as positive target regions, while the remaining unlabeled regions are designated as negative regions. During the prediction phase, the network outputs a probability value for a positive region. The greater the difference between this value and the manually labeled value, the higher the loss value. This approach, by penalizing the class bias between the prediction and the label, guides the network to learn to accurately distinguish between content regions during training, thereby improving the recognition rate.

[0073] The positional offset penalty term is used to evaluate the degree of deviation between the content region location predicted by the network and the manually labeled region at the image coordinate level. Specifically, for each predicted bounding box and the ground truth labeled box, the difference between the top-left and bottom-right pixel coordinates is calculated, and the cumulative error of this difference in both the horizontal and vertical axes is measured. When the predicted bounding box is perfectly aligned with the labeled box, the error is zero; however, when the predicted bounding box is significantly offset, the error will increase significantly. The introduction of this penalty term can effectively suppress the network's tendency to misjudge specific position coordinates while correctly identifying the region category, thus enabling the network to not only "know where it is," but also "know exactly where it is."

[0074] During training, the two loss terms are integrated into a joint optimization objective. The training algorithm updates its parameters based on this joint objective, ensuring that the network improves simultaneously in both region recognition accuracy and region localization precision. This optimization strategy enhances the network's adaptability to different webpage styles and layout structures, and is one of the key mechanisms for achieving high robustness of the structure rendering module at the visual modeling level.

[0075] The structure rendering module establishes a set of anchor point coordinate vectors based on the content aggregation regions output by the network. This set records the coordinates of the top-left and bottom-right corners of each bounding box, along with the attention weight value output by the network and its corresponding layout window index. Each element in this set is considered a candidate structure anchor point.

[0076] The module then performs an anchor sequence sorting operation on the set, using a combined scoring function that prioritizes visual weight and layout stability. Visual weight priority refers to scoring each candidate anchor based on the attention value (i.e., content saliency) generated by the deep perception network in the output prediction layer. The scoring formula is defined as: A1 = w_attn, where w_attn is the maximum attention value of the network for that anchor, normalized to the interval [0,1]. Layout stability priority refers to the degree of fluctuation in the anchor's position across multiple historical webpage versions. It is calculated as: A2 = 1 - σ_pos, where σ_pos represents the standard deviation of the anchor's position in historical versions, indicating the fluctuation range; a smaller value indicates greater stability, and a higher A2 value. The combined scoring function is defined as: S_total = α × A1 + (1 - α) × A2; where α is an adjustable weight ratio parameter (e.g., α = 0.6), representing the priority configuration of visual saliency relative to layout stability. The sorting is performed from high to low based on the S_total value, ensuring that anchors with both high significance and high stability are prioritized in the structural anchor sequence. The sorted structural anchor sequence not only serves as the text localization input for the semantic extraction module but also as path identifiers for the structural consistency index in the reliability evaluation module, thus maintaining the structural semantic linkage consistency across the entire system.

[0077] In the process of semantic vector encoding the text content corresponding to the structural anchor point sequence, the semantic extraction module first maps the positions of the structural anchor point sequence to the text distribution area of ​​the original webpage document. The structural anchor point sequence is a set of positional anchor coordinates extracted by the structural rendering module based on image density, layout information, and content clustering areas. Each anchor point corresponds to a high-confidence content area on the rendered webpage image. The text distribution area refers to the actual text data area in the webpage source code corresponding to the rendering position. By mapping image coordinates to document structure markers, it is possible to trace back from visual anchor points to the original text.

[0078] This module then extracts the contextual text fragments corresponding to each anchor point. The scope typically includes the paragraph corresponding to the anchor point and several paragraphs above and below it, ensuring the completeness of semantic understanding. For example, on a news page, the contextual text fragments of a title anchor might include the paragraphs before and after the news section containing the title; on an e-commerce page, the context of a product parameter anchor might include a combination of the product title, price description, and specifications.

[0079] Each text segment serves as an independent input, generating semantic representations through a context-pre-trained language modeling network. This language modeling network is a language expression comprehension model pre-trained on a large-scale corpus, possessing strong contextual understanding capabilities. The network structure employs a bidirectional attention mechanism, simultaneously considering both preceding and following information within the text segment to construct semantic linkage modeling.

[0080] Internally, the input text fragments are first transformed into a set of embedding vectors, converting the original text into fixed-dimensional numerical vector representations, and applying positional encoding based on the relative positions of words within the sentence. These embedding vectors are then fed into a multi-head attention block, where the system learns attention relationships between words in parallel across multiple subspaces, constructing a contextual semantic dependency matrix. This matrix reflects the strength of the association between the current word and other words in the context. Finally, a feedforward network processes this semantic dependency matrix, outputting a set of vectors representing the comprehensive semantic relationships, constituting the final semantic representation.

[0081] This semantic representation, serving as the feature base, will be aligned with the domain corpus annotation results. The domain corpus annotation results refer to pre-established knowledge templates for specific industries or usage scenarios, used to identify attributes such as the theme, sentiment, and behavioral intent of the text within that context. This alignment process is implemented through a vector mapping transformation function, which maps the general semantic representation to a structured domain feature space, outputting three specific vectors: a theme vector, a sentiment vector, and an intent feature vector.

[0082] The rules for generating the three vectors are as follows:

[0083] The generation of topic vectors is based on the vector inner product matching score between semantic representation and preset topic category codes, reflecting the relative proximity of the current text content within a predefined set of topics. For example, in financial industry applications, topic category codes might include "interest rate changes," "industry dynamics," and "policy interpretation."

[0084] The generation of the sentiment vector is accomplished through a similarity aggregation function between the semantic representation and the sentiment polarity template. The sentiment polarity template is a set of vector representations with emotional tendencies, typically including "positive sentiment," "negative sentiment," and "neutral sentiment." The function calculates the similarity between the semantic representation and each template, and then weights and aggregates the results to obtain a multi-dimensional sentiment intensity expression vector.

[0085] The generation of intent feature vectors involves using semantic representation as the query vector, matching it within the intent template set, and selecting the one with the highest relevance as the output. The intent template set is a sample set of intent expressions constructed for operational behaviors or textual goals, such as "complaint," "purchase," "inquiry," and "praise," and its composition varies in different application scenarios.

[0086] Three vectors are generated sequentially within the semantic extraction module, maintaining independent representations to support subsequent calculations. Ultimately, these three vectors are merged into a semantic vector set, serving as the foundational input for the credibility assessment module when calculating the semantic validity index. This semantic vector set ensures a closed loop from structural anchor point localization to semantic content extraction and establishes a stable mapping relationship between content fragments and semantic structures, representing the core output of the entire semantic recognition process.

[0087] The structural consistency index is calculated as follows:

[0088] First, each anchor point in the structural anchor point sequence is treated as a graph node. The structural anchor point sequence is a set of visual anchor points extracted at the image level by the structural rendering module through a deep perceptron network to identify content clustering areas. Each anchor point includes its coordinates, pixel density, and layout weight on the page. Graph nodes represent each anchor point as a unit node in a graph structure to support subsequent structural modeling and analysis. Next, the adjacency relationships between anchor points are constructed according to the visual rendering order. The visual rendering order refers to the natural visual flow of page content from top to bottom and from left to right on the user's end, often used to express the hierarchy of information presentation and reading order. Based on this order, adjacency edges are established between each anchor point and its adjacent anchor points, forming a local structural subgraph with directionality and positional order. This subgraph reflects the relative position and structural connectivity of content blocks on the current page. After obtaining this local structural subgraph, it is matched with the corresponding webpage structure generated in historical versions using a topological graph mapping mechanism. Topological graph mapping is a method in graph theory used to determine whether two graph structures are equivalent in terms of the number of nodes, connectivity, and arrangement. Subgraph isomorphism matching, on the other hand, determines whether a current subgraph can be embedded into the overall structure graph of a historical version of a webpage while preserving its node connectivity.

[0089] The specific execution process includes: classifying nodes in the structure diagrams generated from historical webpage versions, retaining the corresponding sub-graphs for each version; comparing node labels and verifying edge relationships between the node set of the current sub-graph and the sub-graphs of historical versions; calculating the isomorphic node ratio, i.e., the ratio of the number of nodes in the current sub-graph that can be accurately matched in historical versions to the total number of nodes; and calculating the node arrangement order retention rate, measuring the consistency between the visual order of nodes in the current anchor sequence and the corresponding order in historical versions, which can be represented by normalized sum of position index differences. A weighted expression is constructed using the above two dimensions to obtain a topological consistency score. This score measures whether the current page structure maintains structural continuity with historical pages in terms of content position, relative order, and hierarchy. For example, if the three layers of content "title—price—specifications" in an e-commerce page are always arranged in a top-to-bottom order, the topological consistency score can reach a high range; if the order is reversed or missing, the score decreases.

[0090] The path stability factor is a component of the structural consistency index. It measures whether a structural anchor point remains stable in the page structure across multiple historical webpage versions. Its calculation is based on a path fluctuation characteristic sequence constructed from path change data. Combined with trend and residual analysis, it yields a score reflecting "long-term stability." The calculation process includes the following four steps:

[0091] Constructing a path offset sequence: Historical versions of the target webpage are backsampled, with the structural rendering module extracting corresponding structural anchor points for each version. For the current anchor point, its corresponding anchor point record is found in historical versions, recording its visual position in each version, including its vertical coordinate, logical block number, or node level position in the document structure. These position values ​​are arranged in chronological order according to version, forming a path offset sequence. For example, if the vertical coordinates of a title anchor point in five historical versions are 80, 85, 87, 100, and 105 respectively, then this sequence reflects a "gradual downward shift" over time.

[0092] Constructing a multidimensional path fluctuation trajectory: The path offset sequence is encoded and analyzed in three different dimensions: Location center offset: The average position value of the historical path sequence is calculated. The difference between the current version anchor point's position and this average value is used to obtain the absolute offset distance, which is used to determine whether the current position deviates from the historical center. Time trend direction: Using a sliding window method (e.g., three versions per window), the direction of change is calculated for each historical position sequence. For example, if the position value shows a continuous upward trend, the trend direction is "upward"; if there are large fluctuations, the trend is "no direction". Fluctuation amplitude variability: The standard deviation of the differences between historical paths is calculated, reflecting whether the path change is stable. If each change value is close, the fluctuation is small and the variability is low; if the path value fluctuates drastically, the variability is high. A multidimensional fluctuation trajectory of the anchor point is constructed using these three dimensions.

[0093] Perform residual analysis: Compare the current anchor point's position with the historical average to obtain the residual value. The smaller the residual value, the more the current page structure remains near the historical average; the larger the residual value, the more significant the shift in the anchor point. Combined with trend direction and variability, determine if the residual is "normal": if the trend is stable and the fluctuation is small, then a small residual is considered "stable"; if the trend is volatile and the fluctuation is large, then a small residual may still be considered "abnormal". An evaluation logic can be set here, for example: residual less than a set threshold and low variability → judged as high stability; moderate residual, simple trend but small fluctuation → judged as medium stability; large residual or chaotic trend and high fluctuation → judged as low stability.

[0094] Generate the path stability factor score: According to the determination result of the previous step, the path stability factor is classified into different intervals: If the current anchor position is close to the historical mean, the path change direction is consistent, and the fluctuation range is small, the path stability factor is recorded as a high-confidence value and is recorded in the high interval (such as can be defined as one or close to one); If the current position has a moderate deviation, the change trend is尚可预测, and the fluctuation is moderate, the path stability factor is recorded as a medium-confidence value and falls into the middle interval; If the current position drifts significantly, the fluctuation is strong, and the trend is incoherent, the path stability factor is recorded as a low-confidence value and is classified into the low interval or the lowest interval. For example: In the web page of the government affairs platform, the title anchor of the "Notice and Announcement" area has been in the top left area for eight consecutive versions. If the offset position of the current anchor is in the middle and the fluctuation is obvious, the path stability factor is marked as a low value; On the contrary, if the position change is within ten pixels and the trend is stable, the score can be close to full marks.

[0095] Finally, the structural consistency index is obtained by non-linearly combining the topological consistency score and the path stability score, for example, in a multiplicative form. The combination method adopts a sub-optimal suppression mechanism, that is, if either of the two scores is lower than the set threshold (such as zero point three), the overall index does not enter the high-confidence level interval. This mechanism is derived from a suppression-based scoring strategy in graph neural networks, aiming to prevent the structural drift from being masked by a too-high score in a certain dimension and to improve the robustness and anomaly detection ability of the scoring model. For example, in the page of a government affairs information platform, if the "Announcement Title" area is always stable and in reasonable order, but its path has jumped frequently in multiple versions, resulting in a low path stability score, then even if the topological score is very high, the overall structural consistency index should be reduced to the medium-confidence or low-confidence interval to prevent misjudgment.

[0096] After the structure rendering module and the semantic extraction module complete the linkage operation, the system obtains a set of semantic vector sets corresponding to the structure anchor sequence. The semantic vector set is a composite expression set composed of topic vectors, sentiment vectors, and intention feature vectors generated by semantic encoding of the text segments corresponding to multiple structure anchors through a context pre-trained language modeling network and combined with the domain corpus annotation results. The three vectors generated by each structure anchor are independently represented within the module and jointly form the semantic vector set. In order to measure whether the semantic features contained in the semantic vector set are concentrated, consistent, and credible, the system calculates the semantic validity index for this set in the credibility assessment module. This index is used to reflect whether the identified web page semantics in this batch of acquisition tasks meet the requirements of the target topic and emotional stability, and to guide the subsequent data screening and feedback training mechanism. The calculation method of the semantic validity index is as follows:

[0097] It should be noted that there is an unclear expression "尚可预测" in the original text which is translated as "尚可预测" in a literal way. You may need to check and correct it according to the actual situation.The similarity of each semantic vector in the semantic vector set is compared with the system's preset semantic prototype vector. This semantic prototype vector represents the standard semantic expression pattern generated by the system based on the predetermined acquisition target, typically containing the keywords, structural themes, and contextual features expected by the current task. The first scoring metric is derived by calculating the semantic cosine similarity between the semantic vector and the semantic prototype vector. This score, called semantic cosine similarity, measures whether the current semantic expression is close to the acquisition target; the score is recorded as the first metric. If the similarity value is greater than 0.8, the first metric is marked as 1; if it is less than 0.8, it is linearly amplified by multiplying the similarity value by 1.25. This strategy provides partial confidence compensation when the semantic expression deviates slightly, but no correction is made if the deviation is severe.

[0098] Clustering statistics are performed on all topic vectors in the semantic vector set to determine the proportion of vectors belonging to the same topic code. The topic code is a classification identifier generated by the system in the semantic extraction module by matching the vector inner product score between the semantic representation and the topic category code, used to express the topic category to which the current text belongs. The most frequent category among the topic codes is counted, and its proportion in the overall set is calculated to obtain the second scoring indicator, namely the topic aggregation degree. If this proportion exceeds 60%, the second indicator is set to 1; if it does not exceed 60%, an asymmetric mapping is performed by multiplying the actual proportion by 1.5 to avoid severely weakening semantic judgment due to slight topic dispersion. This score reflects the focus of the system's collected results on topic expression; a high degree of topic concentration indicates a clear and consistent data structure.

[0099] The system performs fluctuation analysis on the sentiment dimension within all sentiment vectors in the semantic vector set. Sentiment vectors are generated by matching semantic representations with sentiment polarity templates, and dimensions include positive, negative, and neutral sentiments. The system statistically analyzes the numerical distribution of all sentiment vectors in the set across each dimension and calculates their standard deviation, which serves as the third scoring metric, called sentiment certainty.

[0100] If the standard deviation is less than 0.1, it indicates that the emotional expression in the set is highly consistent, and this indicator is marked as 1; if the standard deviation is between 0.1 and 0.3, the third indicator is weighted by subtracting the standard deviation from 1, reflecting some ambiguous but acceptable emotional shifts; if the standard deviation is greater than 0.3, it indicates that there are mixed emotional expressions in the set, and the information is ambiguous, and this indicator is marked as zero.

[0101] The system ultimately summarizes the three scoring indicators and fuses them using a minimum-value constraint function to generate a semantic validity index. This function calculates the minimum value among the three scores as the final result, ensuring that any anomaly in any of the three dimensions is sufficient to trigger a decrease in overall credibility, preventing some high values ​​from masking semantic bias risks. That is, if any indicator is low, the overall semantics are deemed unreliable; if all indicators are in the high range, the semantics are considered focused, clear, and reliable. For example, if the theme of a set of structural anchors corresponds to "macroeconomic forecasts," the sentiment tends to be neutral to positive, and the similarity value is consistently above 0.9, then the semantic validity index is highly reliable. Conversely, if the theme distribution is chaotic, some texts express anxiety or negative emotions, or the semantics significantly deviate from the collection objective, the index will drop to the low-to-medium or rejection range. This index will serve as one of the inputs to the fuzzy logic unit, further participating in the subsequent five-level credibility classification, thus providing a decision-making basis for the scheduling priority and model feedback strategy in the execution feedback module.

[0102] After feature extraction and scoring of the structural anchor sequence and semantic vector set, the credibility assessment module calculates the structural consistency index and semantic validity index. The former measures the stability of the page structure in terms of spatial arrangement and historical evolution, while the latter measures the credibility of the text content in terms of semantic theme, emotional expression, and task fit. These two indices are simultaneously input into the fuzzy logic engine to determine the overall credibility level of the collected results. The fuzzy logic engine is a fuzzy rule-driven inference engine. Its core mechanism is to determine the state of the input index through preset interval rules and output a discrete credibility level accordingly. The fuzzy logic engine does not directly perform precise numerical calculations but divides the input signal into several overlapping intervals and defines response rules for each pair of input states. In its implementation, the logic engine uses a two-dimensional fuzzy interval mapping table to construct the index response matrix. The matrix uses the structural consistency index and semantic validity index as the horizontal and vertical axes, respectively. Each axis is divided into five fuzzy segments, named as: low, low-medium, medium, high-medium, and high, with corresponding numerical ranges as follows: low: 0 to 0.2; low-medium: 0.2 to 0.4; medium: 0.4 to 0.6; high-medium: 0.6 to 0.8; high: 0.8 to 1.

[0103] Each pair of input indices is mapped to a segment combination of the two axes mentioned above, forming an input state point. The system then looks up the corresponding confidence level output rule based on the position of this state point in the two-dimensional table. Since each of the two index intervals has five levels, the combinations result in twenty-five different input state scenarios.

[0104] The fuzzy logic generator sets the following five confidence levels for each state:

[0105] Level 1 Definitely Trustworthy: When both the structural consistency index and the semantic validity index are in the high range (i.e., both indices are greater than 0.8), the overall data structure is considered stable and the semantic expression is clear and accurate. The system outputs a Level 1 Definitely Trustworthy level. Data collection results at this level have the highest priority and are prioritized in subsequent data collection tasks and included in the training feedback process.

[0106] Level 2 Strong Trustworthiness: When the structural consistency index is in the mid-to-high range (greater than 0.6 and no higher than 0.8) and the semantic validity index is in the high range, this state is judged as having basically trustworthy structure and excellent semantics, and a Level 2 Strong Trustworthiness level is output. The results at this level are suitable for mid-frequency task scheduling and have model enhancement value.

[0107] Level 3 Suspicious: When any index is in the low-to-medium range (greater than 0.2 and no higher than 0.4), and the other index does not exceed the high-to-medium range (no higher than 0.8), the combination is considered to have uncertainties such as content heterogeneity, structural fluctuations, or semantic shifts, and the system outputs a Level 3 Suspicious rating. This level of data is only scheduled when there are no high-confidence tasks, or used for training low-weight models.

[0108] Level 4 Weakly Trustworthy: When either of the two indices is below 0.2 (i.e., in the low range), but the other is still above 0.2, it indicates that at least one indicator has seriously deviated from the normal range, and the system outputs a Level 4 Weakly Trustworthy rating. Data at this level can be scheduled when no available resources are available, but a stronger data verification mechanism is required.

[0109] Level 5 rejection: When both the structural consistency index and the semantic validity index are below 0.2, it indicates that the structure is extremely unstable, the semantics are unclear, and the task is mismatched. The system directly judges this state as rejection and outputs a Level 5 rejection level. The data will not participate in subsequent processing.

[0110] Taking a specific implementation as an example, if a webpage exhibits a stable modular layout with minimal path fluctuations, but its content semantics are overly fragmented and its emotional expression highly inconsistent (e.g., one part contains neutral policy information, while another part contains negative user comments), the structural consistency index may be in the high range, while the semantic validity index falls only into the low to medium range. In this case, the fuzzy logic device will output a level three suspicious result. Conversely, when the structure is extremely poor and the emotional expression is mixed, making the intent indiscernible, both indices fall into the lowest range simultaneously, directly outputting a level five rejection. This fuzzy logic device ensures that the credibility assessment module, in scenarios involving complex webpage structures and content, can flexibly classify credibility levels through joint evaluation of the two indices, thereby achieving dynamic control over data quality. It is a key module for ensuring the credibility of data acceptance and the stability of subsequent decisions.

[0111] The execution feedback module is a key component in automated data acquisition and decision support systems for market intelligence, responsible for task scheduling control and adaptive model training. This module uses the five-level trust level output by the trust assessment module as the decision-making basis, dynamically updates the priority of data acquisition tasks, and executes an online feedback learning strategy based on semantic offset analysis after data acquisition is completed, thereby achieving iterative optimization and cross-platform adaptability of the model.

[0112] During operation, the execution feedback module first receives the five-level trust level results from the fuzzy logic unit. This five-level classification includes Level 1: Definitely Trustworthy; Level 2: Strongly Trustworthy; Level 3: Suspicious; Level 4: Weakly Trustworthy; and Level 5: Rejection, representing the current data acquisition task's level in the dual-dimensional evaluation of structural consistency and semantic validity. To achieve differentiated scheduling, the execution feedback module presets a corresponding scheduling weight value for each level. Specifically, the weight is set as follows: Level 1: Definitely Trustworthy - 0.9; Level 2: Strongly Trustworthy - 0.7; Level 3: Suspicious - 0.5; Level 4: Weakly Trustworthy - 0.2; and Level 5: Rejection - 0.05. The scheduling weight indicates the system's processing priority for data of different levels in the current acquisition cycle; a higher weight means the task contributes significantly to the quality of the system results and has a higher processing priority.

[0113] During the task queue initialization phase, the system assigns a corresponding trust level label to each task and binds it with a corresponding scheduling weight. The execution feedback module sorts the task queue based on these weights. The sorting rule is to arrange tasks from highest to lowest weight value; tasks with equal weight values ​​can be sorted by arrival time or a custom strategy. The system prioritizes scheduling tasks with weights of 0.9 and 0.7. These tasks correspond to level 1 and level 2 trust levels, have clear structural stability and semantic target fit, have high collection value, and are suitable as corpus input for subsequent analysis or model updates.

[0114] After the task is completed, the collected results are recorded by the system and automatically parsed by the execution feedback module. The parsing primarily focuses on the semantic feature vectors generated by the semantic extraction module, including topic vectors, sentiment vectors, and intent feature vectors. These vectors are then matched against the semantic prototype vectors stored internally by the system, using a cosine angle calculation to measure the degree of semantic shift between them. When the system detects that the cosine distance between the semantic feature vector and the semantic prototype vector in a collected result is greater than a preset feedback threshold (e.g., 0.3), it indicates that the sample has a significant difference in semantic expression and belongs to a potential semantic expansion sample or contextual mutation sample. Such samples will be determined as valid feedback data.

[0115] Valid feedback data is automatically input into the online training process. This process is used to adjust the encoding parameters of the currently used context-pre-trained language modeling network. The training process employs a train-while-inferring mechanism, meaning that the system dynamically introduces new samples and gradually updates the model weight parameters while continuously running, without interrupting the real-time inference task, thus maintaining the model's continuity and responsiveness. To avoid frequent updates causing system fluctuations, the execution feedback module sets a training trigger cycle. This cycle can be configured to automatically execute one training round after completing a predetermined number of data collection tasks (e.g., every ten or twenty executions). Through this control method, the system can gradually accumulate cross-context and cross-platform semantic generalization capabilities while maintaining responsiveness, improving the model's adaptability to different webpage structures, language styles, and content domains.

[0116] For example, during a market monitoring process, the system collected text from the "Product Parameter Description" section of a third-party platform. Its sentiment expression significantly deviated from the "neutral description" defined by the semantic prototype, exhibiting a strong "negative complaint" tone, while its structural anchors remained stable and the content was credible. The execution feedback module then marked this result as a valid feedback sample to improve the model's ability to identify "non-standard product sentiment expressions." The execution feedback module established a closed-loop process between data collection scheduling and model updates—task evaluation, priority scheduling, semantic filtering, and model feedback. This not only ensured the focus and efficiency of the system's data collection but also enhanced the model's evolutionary ability in complex market contexts through a continuous feedback mechanism.

[0117] The above formulas are all dimensionless calculations. The formulas are derived from software simulations based on a large amount of collected data to obtain the most recent real-world results. The preset parameters in the formulas are set by those skilled in the art according to the actual situation.

[0118] It should be understood that in the various embodiments of this application, the order of the above-mentioned processes does not imply the order of execution. The execution order of each process should be determined by its function and internal logic, and should not constitute any limitation on the implementation process of the embodiments of this application.

[0119] Those skilled in the art will recognize that the units and algorithm steps of the various examples described in conjunction with the embodiments disclosed herein can be implemented in electronic hardware, or a combination of computer software and electronic hardware. Whether these functions are implemented in hardware or software depends on the specific application and design constraints of the technical solution. Those skilled in the art can use different methods to implement the described functions for each specific application, but such implementation should not be considered beyond the scope of this application.

[0120] Those skilled in the art will clearly understand that, for the sake of convenience and brevity, the specific working processes of the systems, devices, and units described above can be referred to the corresponding processes in the foregoing method embodiments, and will not be repeated here.

[0121] The above description is merely a specific embodiment of this application, but the scope of protection of this application is not limited thereto. Any variations or substitutions that can be easily conceived by those skilled in the art within the scope of the technology disclosed in this application should be included within the scope of protection of this application. Therefore, the scope of protection of this application should be determined by the scope of the claims.

Claims

1. A market intelligence oriented automated data collection and decision support system, characterized in that, The structure rendering module, the semantic extraction module, the credibility evaluation module and the execution feedback module are included. The structure rendering module obtains the visual output of the target webpage in the image rendering environment, constructs a visual structure map based on pixel density and layout sequence, identifies the content aggregation area through a deep perception network, and outputs a structure anchor point sequence. The semantic extraction module performs semantic vector coding on the text content corresponding to the structure anchor point sequence, extracts semantic representation using a context pre-training language modeling network, and generates theme vectors, sentiment vectors and intent feature vectors in combination with field corpus annotation results. The credibility evaluation module receives the structure anchor point sequence and the semantic vector set as input, calculates the structure consistency index based on the structure symmetry index, the historical version offset rate and the path stability factor, calculates the semantic effectiveness index based on the semantic cosine similarity, the theme aggregation degree and the sentiment certainty, and outputs a five-level credibility grade division result including first-level determined credibility, second-level strong credibility, third-level suspicious, fourth-level weak credibility and fifth-level rejection through the fuzzy logic device. The execution feedback module refreshes the collection and scheduling priority according to the output result of the fuzzy logic device, configures the data entry scheduling weight according to the five-level credibility grade, and preferentially executes the first-level and second-level resource collection tasks, and performs online training feedback on the semantic features in the collection result. The structure anchor point sequence is a set of position anchor coordinates extracted by the structure rendering module based on image density, layout information and content aggregation area, and each anchor point corresponds to a high-confidence content area on the webpage rendering image. The generation of the theme vector is based on the vector inner product matching score of the semantic representation and the theme category coding, the generation of the sentiment vector is completed through the similarity aggregation function of the semantic representation and the sentiment polarity template, and the intent feature vector is output by the maximum relevance matching of the semantic representation to the intent template set; the three vectors are generated in the semantic extraction module and merged into the semantic vector set.

2. The market intelligence oriented automated data collection and decision support system as claimed in claim 1, wherein, After obtaining the visual output of the target webpage in the image rendering environment, the structure rendering module performs image level normalization, divides the webpage visual output into a grid according to a preset pixel area window, and constructs an initial pixel density distribution map based on the pixel density gradient change in each window. The pixel density distribution map is mapped to a layout sequence, which includes window arrangement order, density trend and visual symmetry feature. The content aggregation area in the layout sequence is identified through a deep perception network, and candidate bounding boxes are generated for the content aggregation area using a region candidate mechanism, and repeated candidate boxes are removed based on the visual similarity aggregation threshold between the boxes. The structure anchor point sequence that meets the stable layout structure feature is extracted through one-to-one correspondence between the position of the candidate bounding box and the layout sequence, and the structure anchor point sequence is used as one of the inputs of the credibility evaluation module.

3. The market intelligence oriented automated data collection and decision support system as claimed in claim 2, wherein, The depth perception network in the structure rendering module comprises at least one convolutional coding layer, a position attention mapping layer and an output prediction layer, the convolutional coding layer receives a joint expression matrix of the pixel density distribution map and the layout sequence, extracts a multi-scale feature atlas of the content area, the position attention mapping layer captures a content aggregation mode in the layout sequence through a cross-window comparison mechanism, and assigns a weight to the content area, and the output prediction layer generates a final content aggregation area based on a maximum weight position; The depth perception network training process uses a manually labeled web content area as a supervision signal, and jointly optimizes a cross-entropy loss function and a position offset penalty term; The structure rendering module establishes an anchor point coordinate vector set based on the content aggregation area output by the network, and performs an anchor point sequence sorting operation on the set, and the sorting standard is a combined scoring function of visual weight priority and layout stability priority.

4. The market intelligence oriented automated data collection and decision support system of claim 3, wherein, In the process of performing semantic vector coding on the text content corresponding to the structure anchor point sequence, the structure anchor point sequence is first mapped to the text distribution area of the original web document, the context text segment corresponding to each anchor point is extracted, and the multi-layer vector generation of the semantic representation is completed through the context pre-training language modeling network by taking each text segment as input; The context pre-training language modeling network uses a bidirectional attention mechanism to model the context of the text segment, wherein the input embedding vector is input to the multi-head attention block after position encoding, and the context semantic dependency matrix is obtained, and then the semantic representation is output through the feedforward network; The semantic representation is used as a feature basis, and is aligned with the domain corpus annotation result to obtain a topic vector, a sentiment vector and an intent feature vector. The three vectors are independently represented in the semantic extraction module and are used as the input source of the semantic vector set for the semantic validity index calculation of the credible evaluation module.

5. The market intelligence oriented automated data collection and decision support system as claimed in claim 4, wherein, In the process of generating the topic vector, the sentiment vector and the intent feature vector in combination with the domain corpus annotation result, a domain corpus annotation mapping table is established in advance, the mapping table includes a domain keyword group, a topic category code, a sentiment polarity template and an intent template set, and the semantic representation is mapped to generate by the corresponding matching rule in the mapping table.

6. The market intelligence oriented automated data collection and decision support system as claimed in claim 5, wherein, The structure consistency index calculation method is as follows: Each anchor point in the structure anchor point sequence is regarded as a graph node, an adjacency relationship is constructed according to the visual rendering order to form a local structure subgraph, and according to a topological graph mapping mechanism, the structure subgraph is matched with a comparison graph generated by the corresponding web page structure in the historical version; the topological consistency score is obtained by calculating the isomorphic node proportion and the node arrangement order retention degree; For each anchor point, a path offset sequence is constructed in multiple historical web page versions, a multi-dimensional path fluctuation trajectory is constructed based on a time sequence, and a residual error analysis is performed on the current anchor point path and the trajectory mean value, and the trend direction and amplitude variation of the path change are compared to obtain a path stability factor; The structure consistency index is obtained by nonlinear combination of the topological consistency score and the path stability factor, and the combination method adopts a suboptimal suppression mechanism.

7. The market intelligence oriented automated data collection and decision support system of claim 6, wherein, The semantic validity index calculation method is as follows: The semantic cosine similarity is calculated by the cosine angle between the semantic vector and the semantic prototype vector, and if S1 is greater than 0.8, it is recorded as 1, otherwise it is multiplied by 0.25 to linearly enlarge S1; the theme aggregation degree is calculated by the proportion of the number of semantic vectors belonging to the same theme code in the semantic vector set, and if the proportion is more than 60%, S2 is equal to 1, otherwise the actual proportion is multiplied by 0.5 to perform asymmetric mapping; the emotion certainty is calculated according to the standard deviation σ of the emotion dimension in the semantic vector, if the standard deviation σ of the emotion dimension is less than 0.1, the score S3 is recorded as 1, if the standard deviation σ of the emotion dimension is between 0.1 and 0.3, the score S3 is reduced by the standard deviation σ of the emotion dimension, and if the standard deviation σ of the emotion dimension is greater than 0.3, S3 is zero; the semantic effectiveness index is finally generated by S1, S2 and S3 through the minimum value limiting function F, that is, F = min (S1, S2, S3).

8. The market intelligence oriented automated data collection and decision support system of claim 7, wherein, The structure consistency index and the semantic effectiveness index are input into the fuzzy logic device to divide the confidence level, the fuzzy logic device executes the output of the five-level confidence level through the rule-driven mode, the logic device uses a two-dimensional fuzzy interval mapping table to construct an index response matrix, the value range of the structure consistency index and the semantic effectiveness index is divided into five segments of low, medium-low, medium-high, medium and high, respectively corresponding to the numerical interval of 0-0.2, 0.2-0.4, 0.4-0.6, 0.6-0.8 and 0.8-1, the fuzzy logic device takes the cross interval combination of the two indexes as the input state; When both indexes are in the high interval, the first level of determined confidence is output, when the structure consistency index is in the medium-high interval and the semantic effectiveness index is in the high interval, the second level of strong confidence is output, when any index is in the medium-low interval and the other index is not higher than the medium-high interval, the third level of suspicious is output, when any index is lower than 0.2, the fourth level of weak confidence is output, and when both indexes are lower than 0.2, the fifth level of rejection is output.

9. The market intelligence oriented automated data collection and decision support system of claim 8, wherein, When the fuzzy logic device refreshes the acquisition scheduling priority based on the five-level confidence level output result, the module presets five-level scheduling weights of 0.9, 0.7, 0.5, 0.2 and 0 for the five levels of determined confidence to rejection; In the acquisition task queue, the execution feedback module assigns corresponding scheduling weights to each task according to the level label, and sorts the priorities according to the weights, wherein the tasks with weights of 0.9 and 0.7 are preferentially pushed into the scheduling execution sequence; After the acquisition is completed, the execution feedback module automatically analyzes the semantic feature vector in the acquisition result, and takes the samples with cosine distance greater than the preset feedback threshold from the current semantic prototype vector of the pre-trained semantic modeling network as effective feedback data; The effective feedback data is input into the online training process to adjust the coding parameters of the semantic modeling network, and the edge training and reasoning mechanism is adopted to maintain continuous updating and real-time adaptation of the model, and the update period can be configured to trigger training once every m acquisition tasks.

Images