Automatic data acquisition and decision support system for market intelligence

Abstract

The invention discloses a market intelligence-oriented automatic data acquisition and decision support system, particularly relates to the technical field of artificial intelligence and information processing, and comprises a structure rendering module, a semantic extraction module, and a credible evaluation and execution feedback module. The system extracts structural anchor points through image rendering, generates semantic vectors in combination with context modeling, calculates a structural consistency index and a semantic validity index, outputs a five-level credibility level result through a fuzzy logic device, refreshes a collection task scheduling priority, and updates a pre-training language model based on semantic offset feedback. Self-iteration semantic generalization and efficient decision support under a cross-platform context are realized; according to the method, stable recognition of the webpage structure is achieved through structure rendering, multi-dimensional semantic feature generation is achieved by means of semantic extraction, the credibility level is output in combination with double-index evaluation and fuzzy logic, then collection scheduling and model self-adaptive updating are optimized, and the obtaining precision and decision reliability of market information data are effectively improved.

Description

Technical Field

[0001] The present invention relates to the field of artificial intelligence and information processing technology, and more specifically, to an automated data collection and decision support system for market intelligence. Background Art

[0002] With the development of the digital economy, the strategic value of market intelligence in corporate competition, policy response, product iteration, and user insights has become increasingly prominent. This is particularly true in complex information platforms such as multi-source, heterogeneous web pages, news portals, e-commerce platforms, government websites, and social media, which contain vast amounts of unstructured content on market trends, user preferences, product feedback, and competitive trends. Traditional methods that rely on manual or rule-based information extraction are no longer adaptable to the intelligence environment characterized by high-frequency updates, multi-platform distribution, and significant semantic differences.

[0003] Existing web information collection systems generally focus on structured template parsing, relying on fixed DOM paths, tag identifiers, or regular rules to locate and extract target content. However, in real-world applications, web page structures are frequently updated, module arrangements change significantly, and the proportion of dynamically rendered content gradually increases, resulting in structural templates being easily invalidated and information extraction being unstable. Furthermore, existing systems generally lack a deep understanding of web content at the semantic level, often remaining at the keyword matching or static rule judgment stage, making it difficult to address semantic challenges such as synonymous rephrasing, shifting sentiment, and ambiguous intent.

[0004] After information collection is complete, the lack of an effective trustworthy evaluation mechanism is a significant shortcoming of current technology. Existing methods often focus on the completeness and response speed of captured results, while neglecting the quality assessment of the data itself in multiple dimensions, such as structural stability and semantic consistency. This results in noisy collected content and high analysis costs. Therefore, this paper proposes an automated data collection and decision support system for market intelligence to address this issue. Summary of the Invention

[0005] To achieve the above object, the present invention provides the following technical solutions: An automated data collection and decision support system for market intelligence, including a structure rendering module, a semantic extraction module, a trustworthy assessment module, and an execution feedback module; The structural rendering module obtains the visual output of the target webpage in an image rendering environment, constructs a visual structural map based on pixel density and layout sequence, identifies content cluster areas through a depth perception network, and outputs a structural anchor sequence; The semantic extraction module performs semantic vector encoding on the text content corresponding to the structural anchor sequence, extracts semantic representations using a context-pretrained language modeling network, and generates topic vectors, sentiment vectors, and intent feature vectors based on the domain corpus annotation results. The trust evaluation module receives a sequence of structural anchor points and a set of semantic vectors as input. The former calculates a structural consistency index based on structural symmetry, historical version deviation rate, and path stability factor. The latter calculates a semantic validity index based on semantic cosine similarity, topic aggregation, and sentiment certainty. These two indices are integrated into a fuzzy logic controller to output a five-level trust rating result, including level 1: certain trust, level 2: strong trust, level 3: suspicious, level 4: weak trust, and level 5: rejection. In the execution feedback module, the fuzzy logic output results refresh the collection scheduling priority, configure the data entry scheduling weight according to the five-level trust level, give priority to the execution of the first and second level resource collection tasks, and perform online training feedback on the semantic features in the collection results, enhance the adaptability of the pre-trained semantic modeling network, and realize self-iterative semantic generalization in a cross-platform context.

[0006] In a preferred embodiment, after obtaining the visual output of the target webpage in the image rendering environment, the structure rendering module performs an image level normalization operation, divides the webpage visual output into grids according to preset pixel area windows, and constructs an initial pixel density distribution map based on the pixel density gradient change within each window; Mapping the pixel density distribution map into a layout sequence, where the layout sequence includes window arrangement order, density trend, and visual symmetry features; The deep perception network is used to identify content clusters in the layout sequence, and a region candidate mechanism is used to generate candidate marker boxes for the content clusters. Duplicate candidate boxes are then removed based on the visual similarity aggregation threshold between the boxes. By making one-to-one correspondence between the positions of candidate marker boxes and layout sequences, a structural anchor sequence that meets the stable layout structural characteristics is extracted, and the structural anchor sequence is used as one of the inputs of the trustworthy assessment module.

[0007] In a preferred embodiment, the depth perception network in the structure rendering module includes at least one convolutional coding layer, a position attention mapping layer, and an output prediction layer. The convolutional coding layer receives a joint expression matrix of a pixel density distribution map and a layout sequence, extracts a multi-scale feature map of the content area, and the position attention mapping layer captures the content clustering pattern in the layout sequence through a cross-window comparison mechanism and assigns attention weights to the content area. The output prediction layer generates the final content clustering area based on the position of the maximum weight. The training process of the depth perception network uses manually annotated web page content areas as supervision signals and adopts a cross entropy loss function and a position offset penalty term for joint optimization. The structure rendering module establishes a set of anchor coordinate vectors based on the content cluster area output by the network, and performs an anchor sequence sorting operation on the set. The sorting criterion is a combined scoring function of visual weight priority and layout stability priority. The sorted structural anchor sequence is used as the text positioning input of the semantic extraction module, and the path identification of the structural anchor sequence is retained for calculating the structural consistency index in the trust evaluation module.

[0008] In a preferred embodiment, the semantic extraction module, in the process of performing semantic vector encoding on the text content corresponding to the structural anchor sequence, first maps the position of the structural anchor sequence to the text distribution area of ​​the original webpage document, extracts the context text segment corresponding to each anchor point, and uses each text segment as input to complete the generation of a multi-layer vector of semantic representation through a context pre-trained language modeling network; The contextual pre-trained language modeling network uses a bidirectional attention mechanism to model the contextual linkage of text fragments. The input embedding vector is positionally encoded and then input into a multi-head attention block to obtain a contextual semantic dependency matrix, which is then output through a feedforward network. This semantic representation serves as the feature basis and is aligned with the domain corpus annotation results. The topic vector, sentiment vector, and intention feature vector are obtained through the vector mapping conversion function. The three vectors maintain independent representations within the semantic extraction module and serve as the input source of the semantic vector set that the trusted assessment module relies on when calculating the semantic validity index.

[0009] In a preferred embodiment, the semantic extraction module pre-establishes a domain corpus annotation mapping table in the process of generating topic vectors, sentiment vectors, and intention feature vectors based on the domain corpus annotation results. The mapping table includes domain keyword groups, topic category codes, sentiment polarity templates, and intention template sets. The semantic representation is generated by mapping the corresponding matching rules in the mapping table. The generation of topic vectors is based on the vector inner product matching score of the semantic representation and the topic category encoding. The generation of sentiment vectors is completed through the similarity aggregation function of the semantic representation and the sentiment polarity template. The intention feature vector is output by the maximum relevance matching of the semantic representation to the intention template set; the three vectors are generated sequentially in the semantic extraction module and merged into the semantic vector set.

[0010] In a preferred embodiment, the structural consistency index is calculated as follows: Each anchor point in the structural anchor point sequence is considered a graph node, and adjacency relationships are constructed according to the visual rendering order to form a local structural subgraph. Based on the topological graph mapping mechanism, this structural subgraph is matched with the corresponding web page structure generated in the historical version for subgraph isomorphism. The topological consistency score is obtained by calculating the isomorphic node ratio and the degree of node arrangement order preservation. This score measures whether the structure has structural continuity in terms of content position, relative order and hierarchy. For each anchor point, a path offset sequence is constructed across multiple historical webpage versions. A multidimensional path fluctuation trajectory is constructed based on the time series. Residual analysis is performed between the current anchor point path and the trajectory mean. The trend direction and amplitude variability of path changes are compared to obtain a path stability factor, which reflects the long-term stability of the structure under dynamic changes. The structural consistency index is obtained by nonlinearly combining the topological consistency score and the path stability score. The combination adopts a suboptimal suppression mechanism: if one of the scores is extremely low, the overall index is set to a descending range to prevent structural drift from being obscured by high topology. This mechanism is derived from the inhibitory scoring strategy in graph neural networks, which ensures the robustness of the scoring results and the ability to detect anomalies.

[0011] In a preferred embodiment, the semantic validity index is calculated as follows: The semantic cosine similarity is calculated by the cosine angle between the semantic vector and the semantic prototype vector, score S1. If S1 is greater than 0.8, it is recorded as one, otherwise it is linearly amplified by multiplying S1 by 1.25; the topic aggregation is scored S2 by counting the proportion of the number of codes belonging to the same topic in the set of semantic vectors. If the proportion exceeds 60%, S2 is equal to one, otherwise it is asymmetric mapped by multiplying the actual proportion by 1.5; the emotional certainty is calculated based on the standard deviation σ of the emotional dimension within the semantic vector. If σ is less than 0.1, it is recorded as one. If σ is between 0.1 and 0.3, the score S3 is weighted by subtracting σ from one. If σ is greater than 0.3, S3 is returned to zero; the semantic validity index is finally generated by S1, S2 and S3 through the minimum limit function F, that is, F = min (S1, S2, S3).

[0012] In a preferred embodiment, the trust evaluation module inputs the structural consistency index and the semantic validity index into a fuzzy logic device for trust level classification. The fuzzy logic device outputs a five-level trust level in a rule-driven manner. The logic device uses a two-dimensional fuzzy interval mapping table to construct an index response matrix. The value ranges of the structural consistency index and the semantic validity index are divided into five segments: low, medium-low, medium, medium-high, and high, corresponding to the numerical intervals of 0 to 0.2, 0.2 to 0.4, 0.4 to 0.6, 0.6 to 0.8, and 0.8 to 1, respectively. The fuzzy logic device uses the intersection interval combination of the two indices as the input state, forming a total of 25 mapping situations. When both indexes are in the high range, the output is level one, certain credibility; when the structural consistency index is in the medium-high range and the semantic validity index is in the high range, the output is level two, strong credibility; when either index is in the medium-low range and the other index is not higher than the medium-high range, the output is level three, suspicious; when either index is lower than 0.2, the output is level four, weak credibility; when both indexes are lower than 0.2, the output is level five, rejection.

[0013] In a preferred embodiment, when the execution feedback module refreshes the acquisition scheduling priority based on the five-level credibility output results of the fuzzy logic device, the module presets the five-level scheduling weights as 0.9, 0.7, 0.5, 0.2 and 0, corresponding to the five levels from level 1 confidence to level 5 rejection; In the collection task queue, the execution feedback module assigns a corresponding scheduling weight based on the level label of each task to be collected, and sorts the priorities according to the weight. Tasks with weights of 0.9 and 0.7 are pushed into the scheduling execution sequence first. After the collection is completed, the execution feedback module automatically analyzes the semantic feature vectors in the collection results and selects the samples whose cosine distance with the current semantic prototype vector of the pre-trained semantic modeling network is greater than the preset feedback threshold as valid feedback data; Effective feedback data is input into the online training process to adjust the encoding parameters of the semantic modeling network. A training-while-inference mechanism is adopted to maintain continuous model updates and real-time adaptation. The update cycle can be configured to automatically trigger training every m acquisition tasks.

[0014] Technical effects and advantages of the present invention: The present invention uses a structural rendering module to obtain the visual output of the target webpage in an image rendering environment, construct a visual structural map, and utilize a depth perception network to identify content clusters, thereby generating a sequence of structural anchor points. By jointly modeling pixel density and layout sequences and extracting visually significant regions using a depth perception network, the system is able to stably identify key areas with information density and structural characteristics in real-world environments with complex webpage structures and diverse content layouts. This process significantly improves the accuracy of webpage content extraction and lays a structural foundation for subsequent semantic encoding and credibility assessment.

[0015] In the semantic extraction module of the present invention, the text content corresponding to the structural anchor point is encoded into a semantic vector, the semantic representation is extracted with the help of a context-pretrained language modeling network, and the topic vector, sentiment vector, and intention feature vector are generated in combination with the domain corpus annotation results. Through this process, the system can not only accurately obtain the semantic information corresponding to the structural anchor point, but also convert the text content into a vector expression with multi-dimensional features, realizing the mapping from the original web page information to the computable semantic structure. This method enhances the ability to extract implicit semantics from web page intelligence, and is particularly suitable for market content recognition tasks with complex contexts and diverse expressions.

[0016] The present invention uses a trustworthy assessment module to jointly assess the structural anchor sequence and semantic vector set, calculating the structural consistency index and semantic validity index, respectively. Fuzzy logic is then used to fuse these two indices, outputting a five-level trustworthiness classification. Subsequently, the execution feedback module uses this classification result to refresh the acquisition scheduling priority and dynamically adjusts the parameters of the semantic modeling network based on online training feedback from high-level tasks. This two-tier mechanism not only enables dynamic assessment and fine-grained grading of acquired data quality, but also enhances the model's generalization capabilities in cross-platform contexts through feedback loops, achieving a two-way optimization of acquisition behavior and model adaptability. BRIEF DESCRIPTION OF THE DRAWINGS

[0017] In order to facilitate understanding by those skilled in the art, the present invention will be further described below with reference to the accompanying drawings; Figure 1 This is a schematic diagram of the automated data collection and decision support system for market intelligence in the present invention. DETAILED DESCRIPTION

[0018] The following will provide a clear and complete description of the technical solutions in the embodiments of the present invention in conjunction with the accompanying drawings. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. All other embodiments obtained by ordinary technicians in this field based on the embodiments of the present invention without making any creative efforts shall fall within the scope of protection of the present invention.

[0019] Reference Figure 1 The following examples were obtained: Example

[0020] An automated data collection and decision support system for market intelligence, including a structure rendering module, a semantic extraction module, a trustworthy assessment module, and an execution feedback module; each module is connected to each other through communication.

[0021] The structural rendering module acquires the visual output of the target webpage in an image rendering environment, constructs a visual structural graph based on pixel density and layout sequence, identifies content clusters using a deep perception network, and outputs a sequence of structural anchors. The structural rendering module's purpose is to achieve high-precision identification of the target webpage's visual structure and content region extraction. This process, based on image rendering, captures the webpage's visual output within an image environment, effectively circumventing the sensitivity of traditional HTML static paths to content changes. By jointly modeling pixel density and layout sequence at the image level, the module constructs a structural graph that reflects the page's true visual structure. Based on this, the deep perception network extracts features from the layout sequence and locates content clusters within the page. Finally, a sequence of structural anchors is generated, enabling downstream modules to directly extract corresponding content using these anchors as indices, achieving robust information acquisition despite the heterogeneity and dynamic nature of the webpage. The structural anchor sequence not only has spatially explicit location information but also embeds layout stability weights, providing a precise foundation for subsequent semantic modeling and evaluation.

[0022] The semantic extraction module performs semantic vector encoding on the text content corresponding to the structural anchor sequence, extracting semantic representations using a contextually pre-trained language modeling network. Combined with the domain corpus annotation results, it generates topic vectors, sentiment vectors, and intent feature vectors. The semantic extraction module is responsible for extracting deep semantic features from the text content located at the structural anchors. Its core is to use the structural anchors as reference points, extract contextual text fragments corresponding to the anchor positions from the original webpage document, and then vectorize and encode them using a contextually pre-trained language modeling network. This encoding process uses a bidirectional attention mechanism to model the semantic linkage between contexts, enabling the generated semantic vectors to capture both local information and global contextual awareness. Subsequently, the module aligns the semantic representations with the domain corpus annotation results and maps them into three independent representations: topic vectors, sentiment vectors, and intent feature vectors. Each vector class carries a corresponding semantic judgment task. This module ensures the structured extraction of multi-dimensional intelligence implicit in the original webpage text and serves as the core of the system's semantic recognition and intelligence judgment.

[0023] The trustworthiness assessment module receives a sequence of structural anchor points and a set of semantic vectors as input. The former calculates a structural consistency index based on structural symmetry, historical version drift rate, and path stability factor. The latter calculates a semantic validity index based on semantic cosine similarity, topic convergence, and sentiment certainty. These two indices are integrated into a fuzzy logic controller to output a five-level trustworthiness classification: level 1 (definitely trustworthy), level 2 (strongly trustworthy), level 3 (doubtful), level 4 (weakly trustworthy), and level 5 (rejected). The trustworthiness assessment module integrates structural information and semantic features to construct a comprehensive content trustworthiness assessment system. The first half of the module constructs a structural consistency index for the structural anchor point sequence based on structural symmetry, historical version drift rate, and path stability factor. This index measures the degree of alignment between the spatial layout and historical stability of the page structure. The second half constructs a semantic validity index for the set of semantic vectors based on semantic cosine similarity, topic convergence, and sentiment certainty. This index is used to determine the semantic focus, clarity, and goal consistency of the text. Ultimately, both indices are input into the fuzzy logic controller for comprehensive decision-making. The fuzzy logic analyzer outputs a five-level credibility classification based on the combined range of the two indices, providing clear credibility labels for subsequent acquisition strategies and resource scheduling. This module implements trustworthy modeling of web content from both structural and semantic dimensions, and is the core of the entire system's intelligent and high-precision judgment.

[0024] In the execution feedback module, the fuzzy logic output refreshes the collection scheduling priority. Data entry scheduling weights are configured based on the five levels of trustworthiness, prioritizing the execution of first- and second-level resource collection tasks. Online training feedback is also provided for the semantic features in the collection results, enhancing the adaptability of the pre-trained semantic modeling network and enabling self-iterative semantic generalization across platforms. The execution feedback module's primary task is to optimize collection resource allocation based on the trustworthiness results and enable adaptive evolution of the semantic model. This module maps the five levels of trustworthiness output by the fuzzy logic output into five levels of scheduling weights, which it then uses to refresh the execution priority of each task in the collection task queue. Tasks with higher weights are prioritized for collection, allowing the system to focus on high-value, high-confidence resources. Furthermore, after collection is complete, the module automatically analyzes the semantic feature vectors in the collection results, selecting samples with significant deviations from the current semantic modeling network as valid feedback. These samples are then fed into the online training mechanism for model fine-tuning. This simultaneous training and inference mechanism enables the model to gradually build robust semantic generalization capabilities across multiple platforms, contexts, and representations, significantly improving the system's adaptability and responsiveness to new intelligence sources and web page formats.

[0025] After obtaining the target webpage's visual output in an image rendering environment, the structural rendering module first performs image-level normalization. An image rendering environment refers to the actual visual image generated by loading a webpage via a browser kernel or visual rendering engine. The output is pixel-level image data, not traditional webpage structure code. Image-level normalization standardizes this visual output across different display resolutions, zoom ratios, or browser environments, expressing the page content distribution in a unified spatial coordinate system to facilitate subsequent region identification and comparison.

[0026] The normalized image is divided into multiple preset pixel region windows. A pixel region window is a fixed-size image slice unit with a preset size (e.g., 64×64 pixels or 128×128 pixels) that is evenly distributed across the entire webpage image in a sliding or grid-like manner. Within each pixel region window, a pixel density gradient change analysis is performed. This analysis evaluates the pixel brightness variation trends across different regions within the window, calculates density gradient statistics, and generates a measure of the content density of that window within the overall image.

[0027] Based on the pixel density gradient calculation results of all windows, the structure rendering module constructs an initial pixel density distribution map. This distribution map represents the degree of pixel aggregation in different areas of the entire page in the form of a two-dimensional matrix. The higher the value, the denser the content.

[0028] This pixel density distribution map is then mapped into a layout sequence. A layout sequence is an ordered feature chain that encodes the overall visual structure of a page. It consists of multiple layout descriptors arranged in the visual order of the page. Each descriptor corresponds to a pixel region window, reflecting the window's position, density properties, and structural relationships within the page. This sequence serves as a key intermediate representation in the structure rendering module, carrying the structural cues required for subsequent region recognition tasks.

[0029] The three-dimensional information in the layout sequence is as follows: Window Arrangement Order: This represents the visual rendering order of pixel area windows on the page, from top to bottom and left to right. This ensures that the model can understand the page structure according to the human visual flow pattern and adapt to the top-to-bottom and left-to-right layout inertia of natural language or visual information. This order is converted into a one-dimensional sequence index via two-dimensional image coordinates.

[0030] Density Trend: This reflects the changing trajectory of pixel density within consecutive windows. By calculating the difference in density between adjacent windows and establishing a trend vector (e.g., +1 indicates increasing density, -1 indicates decreasing density, and 0 indicates stability), the system can identify structural signals within the image, such as distinct structural block boundaries, breakpoints, and information hierarchy. For example, when multiple windows have increasing density vertically, this typically indicates a visual organization of title, body text, and key points.

[0031] Visual symmetry features assess whether the page structure exhibits horizontal, vertical, or diagonal symmetry. This is accomplished by mirroring the page image in corresponding directions and calculating a symmetry score, creating a quantifiable indicator of structural balance. Symmetrical structures are often found on standard pages such as news websites and government platforms, and are highly recognizable. The output of this layout sequence serves as the dominant structural representation of the subsequent depth perception network input, determining the distribution of the model's attention areas.

[0032] The execution process of the deep perception network to identify content cluster areas: Next, the layout sequence is analyzed using a depth perception network to identify possible content clusters. The depth perception network is the core model for structural understanding and regional focus in the structure rendering module, and consists of the following three functional layers: Convolutional Encoding Layer: This layer converts the layout sequence into tensor inputs to extract local structural relationships between regions. As the convolution kernel slides, it captures patterns such as repeated blocks, aligned structures, and density mutations, and outputs a multi-scale feature map that preserves spatial relationships.

[0033] Attention Mapping Layer: Based on the output of the convolutional coding layer, it constructs an attention weight matrix across windows to measure the relative importance of each region. The attention mechanism calculates feature similarity and positional relationships between regions, guiding the network to focus more computing resources on areas with high content density.

[0034] Output prediction layer: Based on the attention distribution results, the set of regions with the most information value is determined, and a preliminary content area heat map is generated. The coordinates of the hot spots are then extracted through a preset non-maximum suppression algorithm as candidates for content cluster areas.

[0035] Content clusters are areas within webpage images with high information density and a high probability of being read by users. Examples include the title area of ​​a news listing, the price and specifications section of an e-commerce product page, and the introduction and summary section of a policy announcement page. By effectively extracting these areas, the system can significantly improve the efficiency and accuracy of subsequent data collection and semantic analysis.

[0036] The structural rendering module applies a region candidate mechanism to the identified content clusters to form candidate marker boxes. The candidate marker boxes are image region positioning boundaries constructed using rectangles as basic units, reflecting the specific focus location of suspicious information. The specific generation method is as follows: threshold cutting is performed on the heat map in the content cluster area (for example, areas with a heat greater than 0.65 are marked as target areas); a minimum bounding rectangle is generated in the target area as the initial form of the candidate marker boxes; each candidate marker box records its top-left and bottom-right coordinate points, and is accompanied by metadata such as heat score, relative position index, and original window association number, forming the primary building block of the subsequent structural anchor points.

[0037] Visual similarity aggregation and redundancy elimination mechanism: After candidate marker boxes are generated, the structure rendering module performs visual similarity aggregation on all boxes to eliminate overlapping, redundant, or homogeneous content areas and improve anchor quality. This aggregation process includes: color distribution comparison: Counting the RGB or HSV channel distribution of pixels within the box and assessing similarity using color histogram intersection; edge feature matching: Using edge detectors such as Canny to extract structural boundaries and calculate similarity scores; texture consistency: Analyzing regional textures using gray-level co-occurrence matrices or local binary patterns to assess differences; and spatial overlap assessment: Using the Intersection over Union (IoU) metric to determine the degree of overlap between boxes. Box pairs that meet these multiple similarity criteria are classified as similar if their aggregation score exceeds a set inter-box visual similarity aggregation threshold (e.g., 0.85). Those with higher scores are retained, while others are discarded. This process ensures that only non-redundant, representative candidate regions are retained for subsequent structural anchor extraction, further improving the accuracy of structural consistency assessment and semantic extraction.

[0038] The structural rendering module maps the retained candidate marker boxes to the window positions in the layout sequence in a one-to-one correspondence, extracts anchor points that meet the characteristics of a stable layout structure, and forms a structural anchor sequence. The structural anchor sequence is a set of visual anchor points that can be uniquely located and has three core properties: the first is the image coordinate position; the second is the position index within the page after normalization; and the third is the pixel density label and structural confidence score of the anchor point. The structural anchor sequence is not only the final output of this module, but also one of the inputs of the trust evaluation module. It will be used to calculate the structural consistency index in the future and jointly determine the trust level of the web page information with the semantic vector set. The quality of this sequence will directly affect the performance of the subsequent semantic extraction and evaluation modules, and is the core of the structural entry of the entire data collection process.

[0039] To achieve reliable quantification of anchor point screening, the structural rendering module assigns a structural confidence score to each anchor point. This score is used to measure the stability and effectiveness of the anchor point as a structural representative area on the page. This score is calculated from the following two quantitative indicators: a pixel density strength score, which is derived from the density value of the window where the anchor point is located in the initial pixel density distribution map; a standard normalization method is used to map the density value to a range between zero and one; and a score that indicates the visual prominence of the anchor point. A layout position stability score is derived from whether the candidate marker box maintains a stable position in different historical versions of the web page; the frequency of occurrence and coordinate deviation of the area corresponding to the anchor point in the historical page structure are counted; if the historical deviation is less than a preset threshold, the score is increased, indicating that it has a stable structural composition in the long term. The structural confidence score is synthesized as the unweighted product of the above two scores, defined as follows: confidence score S = D × P, where D is the pixel density strength score and P is the layout position stability score. This multiplication method has a suboptimal restriction effect. When the score of any dimension is low, the overall score will be lowered to prevent the situation where the overall credibility is insufficient due to a high score of one dimension.

[0040] After the aforementioned layout sequence and pixel density distribution map are constructed, the structure rendering module performs the content area recognition task on their joint representation (i.e., the structure map constructed above) through the depth perception network.

[0041] The depth perception network in the structure rendering module consists of at least one convolutional encoding layer, a positional attention mapping layer, and an output prediction layer. This depth perception network is used to identify areas of webpage content concentration based on the layout structure and pixel distribution. The network input is a joint expression matrix, which is composed of the pixel density distribution map constructed above and the layout sequence. This matrix is ​​input into the network through matrix splicing and channel fusion in the spatial dimension, ensuring the simultaneous modeling of visual density and structural logic.

[0042] The convolutional coding layer performs multi-scale convolution operations, extracting features from the input matrix using convolution kernels of multiple receptive field sizes. This creates multi-scale feature maps that include high-density clustering, boundary variation, and local symmetry. These feature maps represent underlying content region patterns at different resolution levels. The position attention mapping layer is used to enhance the model's focus on key regions. This layer calculates feature similarity between different windows, establishes a cross-window comparison mechanism, and forms an attention distribution weight map that indicates which locations within the overall page structure are most likely to be content clusters. Higher attention values ​​indicate greater structural significance. Based on this weight map, the output prediction layer extracts the coordinates of the locations with the highest weights in the entire image and generates a set of bounding boxes for content clusters within the local neighborhood. The output is a set of rectangular box coordinates representing the regions with the highest information density and structural potential in the image.

[0043] The training process of this deep perception network uses manually annotated web page content regions as supervisory signals. Specifically, regions such as the title, abstract, and body of the webpage are manually labeled to form a training set. During training, a cross-entropy loss function is used to measure the consistency of the model output with the manually annotated regions. A position offset penalty term is introduced to constrain the distance between the model output region and the true annotated region in coordinate space. This penalty term ensures that the model accurately predicts the content region while maintaining positioning accuracy, forming a joint optimization goal. This joint optimization strategy includes two loss calculation mechanisms: classification accuracy loss and position offset penalty term, which jointly influence the adjustment direction of network parameters during training.

[0044] The classification accuracy loss is constructed using a discriminative loss method based on the log-likelihood principle to measure the network's accuracy in identifying content-intensive areas. In each training sample image, the actual content area on the page is manually annotated and used as the positive target area, while the remaining unannotated areas are considered negative. During the prediction phase, the network outputs a probability value for a region being positive. The greater the difference between this value and the manual annotation, the higher the loss value. This method penalizes the category deviation between predictions and annotations, guiding the network to learn to accurately distinguish between content areas during training, thereby improving recognition rate.

[0045] The position offset penalty is used to evaluate the degree of deviation between the position of the content area predicted by the network and the manually annotated area at the image coordinate level. Specifically, for each predicted box and the true annotated box, the difference in the pixel coordinates of the upper left corner and the lower right corner is calculated, and the cumulative error of the difference in both the horizontal and vertical directions is measured. When the predicted box is completely aligned with the annotated box, the error is zero; when the predicted box is significantly offset, the error will increase significantly. The introduction of this penalty can effectively suppress the network's tendency to misjudge the specific position coordinates while determining that the region category is correct, thereby prompting the network to not only "know where it is" but also "know where it is."

[0046] During training, the two loss terms are combined into a joint optimization objective. The training algorithm updates parameters based on this joint objective, ensuring simultaneous improvements in both region recognition accuracy and localization precision. This optimization strategy enhances the network's adaptability to diverse web page styles and layout structures, and is a key mechanism for achieving high robustness in the structural rendering module at the visual modeling level.

[0047] The structure rendering module creates a set of anchor coordinate vectors based on the content clusters output by the network. This set records the coordinates of the top-left and bottom-right corners of each marked box, along with the attention weights output by the network and their corresponding layout window index. Each element in this set is considered a candidate structure anchor.

[0048] The module then sorts the anchor sequence on this set, using a combined scoring function that prioritizes visual weight and layout stability. Visual weight prioritization is a scoring function based on the attention value (i.e., content salience) generated by the deep perception network in the output prediction layer for each candidate anchor point. The scoring formula can be defined as: A1 = w_attn, where w_attn is the maximum attention value of the network for that anchor point, normalized to the range [0, 1]. Layout stability prioritization refers to the degree of fluctuation in the position of the anchor point across multiple historical webpage versions. It is calculated as: A2 = 1 - σ_pos, where σ_pos represents the standard deviation of the anchor point's position across historical versions, indicating the magnitude of fluctuation. Smaller values ​​indicate greater stability and higher A2. The combined scoring function is defined as: S_total = α × A1 + (1 - α) × A2; α is an adjustable weighting parameter (e.g., α = 0.6), indicating the priority of visual salience over layout stability. Sorting is performed from high to low S_total values, ensuring that anchors with both high significance and high stability are prioritized within the structural anchor sequence. The sorted structural anchor sequence not only serves as text positioning input for the semantic extraction module but also serves as path identification for the structural consistency index in the trustworthiness assessment module, maintaining system-wide structural semantic linkage consistency.

[0049] When performing semantic vector encoding on the text content corresponding to the structural anchor sequence, the semantic extraction module first maps the locations of the structural anchor sequence to the text distribution area of ​​the original webpage document. The structural anchor sequence is a set of positional anchor coordinates extracted by the structural rendering module based on image density, layout information, and content clustering areas. Each anchor corresponds to a high-confidence content area on the rendered webpage image. The text distribution area refers to the actual text data area corresponding to the rendering location in the webpage source code. By mapping image coordinates with document structure markers, it is possible to trace back from the visual anchor point to the original text.

[0050] The module then extracts the contextual text snippets corresponding to each anchor point. This context typically includes the paragraph corresponding to the anchor point and several paragraphs above and below it, ensuring complete semantic understanding. For example, on a news page, the contextual text snippets for a title anchor point might include the preceding and following paragraphs of the news module where the title appears; on an e-commerce page, the context for a product parameter anchor point might include the product title, price description, and specifications.

[0051] Each text segment is used as an independent input, and a contextually pre-trained language modeling network is used to generate a semantic representation. This language modeling network is a language expression understanding model pre-trained using a large-scale corpus, and has strong contextual understanding capabilities. The network structure uses a bidirectional attention mechanism, which simultaneously considers the preceding and following context of the text segment to construct a semantic linkage model.

[0052] Within the model, the input text fragment is first converted into a set of embedding vectors. This converts the original text into a fixed-dimensional numerical vector representation, and applies positional encoding based on the relative position of the words in the sentence. These embedding vectors are then fed into a multi-head attention block. The system learns the attention relationships between words in parallel across multiple subspaces, constructing a contextual semantic dependency matrix that reflects the strength of the current word's association with other words in the context. Finally, this semantic dependency matrix is ​​processed by a feedforward network, outputting a set of expression vectors that comprehensively represent the semantic relationships, forming the final semantic representation.

[0053] This semantic representation, serving as a feature foundation, is aligned with the domain corpus annotation results. Domain corpus annotation results refer to pre-established knowledge templates for specific industries or usage scenarios, which are used to identify attributes such as themes, sentiment, and behavioral intent within the context of the text. This alignment is achieved through a vector mapping transformation function, which maps the universal semantic representation into a structured domain feature space and outputs three specific vectors: a topic vector, a sentiment vector, and an intent feature vector.

[0054] The generation rules of the three vectors are as follows: Topic vectors are generated based on the inner product match score between the semantic representation and the pre-defined topic category codes, reflecting the relative proximity of the current text content to the pre-defined topic set. For example, in financial industry applications, topic category codes may include "interest rate changes," "industry dynamics," and "policy interpretation."

[0055] Sentiment vectors are generated using a similarity aggregation function between semantic representations and sentiment polarity templates. Sentiment polarity templates are a set of vector representations of emotional tendencies, typically including "positive sentiment," "negative sentiment," and "neutral sentiment." This function calculates the similarity between the semantic representation and each template and then weights and aggregates the results to produce a multi-dimensional sentiment intensity expression vector.

[0056] To generate an intent feature vector, semantic representation is used as a query vector. This vector is matched against a set of intent templates, and the most relevant one is selected as the output. An intent template is a set of sample intent expressions constructed for action actions or textual targets, such as "complaint," "purchase," "consult," and "compliment." These templates vary in composition across different application scenarios.

[0057] The three vectors are generated sequentially within the semantic extraction module and maintained as independent representations to support subsequent computations. Ultimately, these three vectors are combined into a semantic vector set, which serves as the basic input for the semantic validity index calculation in the trustworthiness assessment module. This semantic vector set ensures a closed loop from structural anchor point positioning to semantic content extraction, establishing a stable mapping between content segments and semantic structures. It represents the core outcome of the entire semantic recognition process.

[0058] The structural consistency index is calculated as follows: First, each anchor point in the structural anchor point sequence is considered a graph node. The structural anchor point sequence is a set of visual anchor points extracted at the image level by the structural rendering module using a depth perception network to identify content clusters. Each anchor point contains its coordinates, pixel density, and layout weight within the page. A graph node represents each anchor point as a unit node within a graph structure to support subsequent structural modeling and analysis. Next, adjacency relationships between anchor points are established according to the visual rendering order. The visual rendering order refers to the natural top-to-bottom and left-to-right visual flow of page content on the user's end, and is often used to express the hierarchy and reading order of information presentation. Based on this order, adjacency edges are established between each anchor point and its adjacent anchor points, forming a local structural subgraph with directional and positional order. This subgraph reflects the relative position and structural connectivity of content blocks within the current page. After obtaining this local structural subgraph, it is then subjected to subgraph isomorphism matching with a comparison graph generated from the corresponding web page structure in historical versions using a topological graph mapping mechanism. Topological mapping is a method used in graph theory to determine whether two graph structures are equivalent in terms of node number, connectivity, and arrangement. Subgraph isomorphism matching determines whether a subgraph of the current structure can be embedded into the overall structure of a historical webpage while preserving its node connectivity.

[0059] The specific execution process includes: classifying the nodes in the structural graph generated by historical web page versions, retaining the structural subgraph of the corresponding area in each version; comparing the node labels and edge relationships of the node set of the current structural subgraph with the historical version subgraph; calculating the proportion of isomorphic nodes, that is, the ratio of the number of nodes in the current subgraph that can be accurately matched in the historical version to the total number of nodes; calculating the node arrangement order preservation degree, which measures the consistency of the visual order of the nodes in the current anchor sequence with the corresponding order in the historical version, and can be expressed by the normalized sum of the position index differences. A weighted expression is constructed using the above two dimensions to obtain a topological consistency score. This score is used to measure whether the current page structure maintains structural continuity with the historical page in terms of content position, relative order, and hierarchy. For example, if the three layers of content "title-price-specifications" on an e-commerce page are always arranged in order from top to bottom, the topological consistency score can reach a high range; if the order is reversed or missing, the score will decrease.

[0060] The path stability factor is a component of the structural consistency index, used to measure whether a structural anchor point remains stable in the page structure across multiple historical web page versions. Its calculation process is based on the path fluctuation characteristic sequence constructed from path change data, combined with trend and residual analysis to obtain a score reflecting "long-term stability." The calculation process includes the following four steps: Constructing a path offset sequence: The target webpage's historical versions are retroactively sampled, and the structural rendering module extracts the corresponding structural anchors for each version. For the current anchor, its corresponding anchor record is searched in the historical versions, and its visual position in each version is recorded, including the vertical coordinate, the logical block number, or the node hierarchical position in the document structure. These position values ​​are arranged in chronological order according to the version, forming a path offset sequence. For example, if the vertical coordinates of a title anchor in five historical versions are 80, 85, 87, 100, and 105, respectively, then the sequence reflects a "gradual downward shift" over time.

[0061] Constructing a multidimensional path fluctuation trajectory: Path deviation sequences are encoded and analyzed along three different dimensions: Position Center Offset: Calculates the average position value of the historical path sequence and subtracts the current version's anchor point position from this average value to obtain the absolute offset distance, which is used to determine whether the current position deviates from the historical center. Temporal Trend Direction: Using a sliding window approach (e.g., three versions as one window), calculates the direction of change for each segment of the historical position sequence. For example, if the position value shows a continuous upward trend, the trend direction is "upward"; if there are large fluctuations, the trend is "no direction." Fluctuation Amplitude Variation: Calculates the standard deviation of the differences between historical paths to reflect the stability of path changes. If the change values ​​are close, the fluctuation is small and the variation is low; if the path values ​​fluctuate significantly, the variation is high. Using these three dimensions, a multidimensional fluctuation trajectory for the anchor point is constructed.

[0062] Perform residual analysis: Compare the current anchor point position with the historical mean to obtain the residual value. Smaller residual values ​​indicate that the current page structure remains close to the historical average structure; larger residual values ​​indicate that the anchor point has shifted significantly. Combined with the trend direction and variability, determine whether the residual is "normal": If the trend is stable with small fluctuations, a small residual is "stable"; if the trend is intense with large fluctuations, a small residual may still be considered "abnormal." A set of evaluation logic can be set here. For example, if the residual is less than the set threshold and the fluctuation variability is low, it is judged as highly stable; if the residual is medium, the trend is simple but the fluctuation is small, it is judged as medium stable; if the residual is large or the trend is chaotic with high fluctuations, it is judged as low stable.

[0063] Generating a Path Stability Factor Score: Based on the results of the previous step, the Path Stability Factor is categorized into different ranges: If the current anchor point position is close to the historical mean, the path change direction is consistent, and the fluctuation range is small, the Path Stability Factor is scored as a high confidence value and placed in the high range (e.g., one or close to one). If the current position has a moderate deviation, the trend is predictable, and the fluctuation is moderate, the Path Stability Factor is scored as a medium confidence value and placed in the middle range. If the current position drifts significantly, the fluctuation is strong, and the trend is inconsistent, the Path Stability Factor is scored as a low confidence value and placed in the low or lowest range. For example, on the government platform website, the title anchor of the "Notices and Announcements" section has been located in the top left area for eight consecutive versions. If the anchor point shifts to the middle position this time, indicating significant fluctuation, the Path Stability Factor is scored as a low value. Conversely, if the position change is within ten pixels and the trend is stable, the score can be close to the full score.

[0064] Finally, the structural consistency index is obtained by nonlinearly combining the topological consistency score and the path stability score, for example, in the form of a product. The combination method adopts a suboptimal suppression mechanism, that is, if either of the two scores is lower than the set threshold (for example, 0.3), the overall index will not enter the high confidence level interval. This mechanism is derived from an inhibitory scoring strategy in graph neural networks. The purpose is to prevent structural drift from being masked by an excessively high score in a certain dimension, and to improve the robustness and anomaly detection capabilities of the scoring model. For example, in a government information platform page, if the "Announcement Title" area is always stable and the order is reasonable, but its path has frequently jumped in multiple versions, resulting in a low path stability score, then even if the topological score is very high, the overall structural consistency index should be reduced to a medium or low confidence interval to prevent misjudgment.

[0065] After the structural rendering module and the semantic extraction module complete the linkage operation, the system obtains a set of semantic vectors corresponding to the structural anchor sequence. The semantic vector set is a composite expression set consisting of topic vectors, sentiment vectors, and intention feature vectors generated by semantically encoding text fragments corresponding to multiple structural anchors through a contextual pre-trained language modeling network and combining the domain corpus annotation results. The three vectors generated by each structural anchor maintain independent representations within the module and together constitute a semantic vector set. In order to measure whether the semantic features contained in the semantic vector set are centralized, consistent, and credible, the system calculates the semantic validity index for the set in the credible evaluation module. This index is used to reflect whether the semantics of the identified web page in this batch acquisition task meet the target theme and emotional stability requirements, and to guide subsequent data screening and feedback training mechanisms. The semantic validity index calculation method is as follows: A similarity comparison is performed between each semantic vector in the semantic vector set and the semantic prototype vector preset by the system. The semantic prototype vector represents the standard semantic expression pattern generated by the system according to the established acquisition target, which usually contains the key words, structural themes and contextual features expected by the current task. The first scoring indicator is obtained by calculating the semantic cosine angle between the semantic vector and the semantic prototype vector. This score is called semantic cosine similarity, which is used to measure whether the current semantic expression is close to the acquisition target, and the score is recorded as the first indicator. If the similarity value is greater than 0.8, the first indicator is marked as one; if it is lower than 0.8, it is linearly amplified at a ratio of 1.25 times the similarity value. This strategy is used to provide partial confidence compensation when the semantic expression deviates slightly, but no correction is made if the deviation is serious.

[0066] Perform cluster statistics on all topic vectors in the semantic vector set to determine the proportion of the number of vectors that belong to the same topic code. The topic code is a classification identifier generated by the system in the semantic extraction module by matching the vector inner product of the semantic representation and the topic category code. It is used to express the topic category to which the current text belongs. Count the category with the highest frequency in the topic code and calculate its proportion in the overall set to obtain the second scoring indicator, the topic aggregation degree. If the proportion exceeds 60%, the second indicator is marked as 1; if it does not exceed, an asymmetric mapping is performed by multiplying the actual proportion by 1.5 to avoid seriously weakening the semantic judgment due to slight dispersion of the topic. This score reflects the degree of focus of the system's collection results on the topic expression. If the topic is highly concentrated, it means that the data structure is clear and consistent.

[0067] The system analyzes the fluctuations of the emotional dimensions within all sentiment vectors in the semantic vector set. Sentiment vectors are generated by matching semantic representations with sentiment polarity templates, and their dimensions include positive, negative, and neutral sentiment. The system then statistically analyzes the numerical distribution of all sentiment vectors in the set along each dimension and calculates their standard deviation, which serves as a third scoring metric, called sentiment certainty.

[0068] If the standard deviation is less than 0.1, it means that the emotional expressions in the set are highly consistent, and the index is marked as one; if the standard deviation is between 0.1 and 0.3, the third indicator is weighted by the value of one minus the standard deviation, reflecting some ambiguous but acceptable emotional deviations; if the standard deviation is greater than 0.3, it means that there are mixed expressions of multiple emotions in the set and the information is ambiguous, and the index is marked as zero.

[0069] Ultimately, the system aggregates these three scoring metrics and fuses them using a minimum constraint function to generate a semantic validity index. This function's calculation logic takes the minimum of the three scores as the final result, ensuring that any anomaly in any of the three dimensions is sufficient to trigger a decrease in overall credibility, preventing the risk of semantic deviations being masked by a few high values. Specifically, if any metric is low, the overall semantics are deemed unreliable; if all metrics are high, the semantics are considered focused, clear, and reliable. For example, if the texts corresponding to a group of structural anchors all share the theme of "macroeconomic forecasting," the sentiment tends to be neutral to positive, and the similarity value remains stable above 0.9, the semantic validity index is considered high. Conversely, if the topic distribution is disorganized, some texts express anxiety and negativity, or the semantics clearly deviate from the collection objectives, the index will drop to the medium-low or rejection range. This index serves as one of the inputs to the fuzzy logic controller and is subsequently used in the five-level credibility classification, providing a decision-making basis for scheduling priorities and model feedback strategies in the execution feedback module.

[0070] After extracting and scoring the features of the structural anchor sequence and semantic vector set, the trustworthiness assessment module calculates a structural consistency index and a semantic validity index, respectively. The former measures the stability of the page structure in terms of spatial arrangement and historical evolution, while the latter measures the credibility of the text content in terms of semantic themes, emotional expression, and task fit. These two indices are simultaneously input into the fuzzy logic controller to determine the overall credibility of the collected results. The fuzzy logic controller is a fuzzy rule-driven inference engine. Its core mechanism is to determine the state of the input index using preset interval rules and output a discrete credibility level accordingly. The fuzzy logic controller does not directly perform precise numerical calculations. Instead, it divides the input signal into several overlapping intervals and defines response rules for each pair of input states. In its implementation, the logic controller uses a two-dimensional fuzzy interval mapping table to construct an index response matrix. The matrix uses the structural consistency index and semantic validity index as the horizontal and vertical axes. Each axis is divided into five fuzzy segments, named: low, medium-low, medium, medium-high and high. The corresponding numerical ranges are: low: 0 to 0.2; medium-low: 0.2 to 0.4; medium: 0.4 to 0.6; medium-high: 0.6 to 0.8; high: 0.8 to 1.

[0071] Each pair of input index values ​​is mapped to a combination of segments along the two axes, forming an input state point. The system then searches for the corresponding trust level output rule based on the state point's position in the two-dimensional table. Since each of the two index intervals has five levels, this combination creates 25 different input state scenarios.

[0072] The fuzzy logic controller sets the following five levels of credibility output for each state: Level 1 Trust: When both the Structural Consistency Index and the Semantic Validity Index are in the high range (i.e., both indices are greater than 0.8), the data structure is considered stable and the semantic expression is clear and accurate, and the system outputs a Level 1 Trust rating. Collection results at this level have the highest priority and are prioritized in subsequent collection tasks and enter the training feedback process.

[0073] Level 2 Strong Trust: When the structural consistency index is in the medium-high range (greater than 0.6 and no higher than 0.8) and the semantic validity index is in the high range, the state is judged to be basically structurally trustworthy and semantically excellent, and the output is a Level 2 Strong Trust rating. Results at this level are suitable for medium-frequency task scheduling and have model enhancement value.

[0074] Level 3 Suspicious: When any index is in the medium-low range (greater than 0.2 and no higher than 0.4), and the other index is within the medium-high range (no higher than 0.8), the combination is considered to have uncertainties such as content heterogeneity, structural fluctuation, or semantic drift, and the system outputs a Level 3 Suspicious rating. Data at this level is only scheduled when there are no high-confidence tasks or used for low-weight model training.

[0075] Level 4 Weak Trust: When either index is below 0.2 (i.e., in the low range), but the other index is still above 0.2, indicating that at least one indicator is significantly deviating from the normal range, the system outputs a Level 4 Weak Trust rating. Data at this level can be dispatched when there are no available resources, but the data review mechanism must be strengthened.

[0076] Level 5 rejection: When the structural consistency index and semantic validity index are both lower than 0.2, it means that the structure is extremely unstable, the semantics are unclear, and the task does not match. The system directly determines this state as rejection and outputs a level 5 rejection level. The data will not be included in subsequent processing.

[0077] For example, if a webpage's structure exhibits a stable modular layout with minimal path fluctuations, but its content is overly fragmented and its emotional expression is highly inconsistent (e.g., one section contains neutral policy information and another contains negative user comments), the structural consistency index may be high, while the semantic validity index falls only into the medium-low range. In this case, the fuzzy logic controller will output a Level 3 suspicious result. If the structure is extremely poor, the emotional expression is mixed, and the intention is undiscernible, both indices will fall into the lowest range, resulting in a direct Level 5 rejection. This fuzzy logic controller ensures that the trust evaluation module can flexibly classify trust levels through dual-index joint evaluation in complex webpage structures and content scenarios, thereby achieving dynamic control of data quality. It is a key module for ensuring the credibility of data adoption and the stability of subsequent decision-making.

[0078] The execution feedback module is a key component responsible for task scheduling and model adaptive training within the automated data collection and decision support system for market intelligence. This module uses the five-level trustworthiness rating output by the trustworthiness assessment module as the basis for decision-making, dynamically updates the scheduling priority of collection tasks, and implements an online feedback learning strategy based on semantic drift analysis after collection is complete, enabling iterative model optimization and cross-platform adaptability.

[0079] During operation, the execution feedback module first receives the five-level trust level results output by the fuzzy logic device. The five-level division includes the first level of definite trust, the second level of strong trust, the third level of doubt, the fourth level of weak trust and the fifth level of rejection, which respectively represent the level of the current acquisition task in the two-dimensional evaluation of structural consistency and semantic validity. In order to achieve differentiated scheduling, the execution feedback module presets a corresponding scheduling weight value for each level. The specific settings are: the first level of definite trust corresponds to a weight of 0.9; the second level of strong trust corresponds to a weight of 0.7; the third level of doubt corresponds to a weight of 0.5; the fourth level of weak trust corresponds to a weight of 0.2; the fifth level of rejection corresponds to a weight of zero. The scheduling weight is used to indicate the system's processing priority for different levels of data in the current acquisition cycle. A high weight means that the task contributes greatly to the quality of the system results and has a high processing priority.

[0080] During the initialization phase of the collection task queue, the system assigns a corresponding trust level label to each pending task and associates it with a corresponding scheduling weight. The execution feedback module sorts the task queue based on these weights. The sorting rule is to sort by weight from highest to lowest. Tasks with equal weights can be sorted by arrival time or a custom strategy. The system prioritizes tasks with weights of 0.9 and 0.7. These tasks correspond to the first and second trust levels, have clear structural stability and semantic target alignment, and have high collection value, making them suitable as corpus input for subsequent analysis or model updates.

[0081] After the task is completed, the collection results are recorded by the system and automatically parsed by the execution feedback module. The parsing content is mainly aimed at the semantic feature vectors generated by the semantic extraction module, including topic vectors, sentiment vectors, and intention feature vectors. These vectors will be matched with the semantic prototype vectors retained within the system for similarity, and the degree of semantic deviation between the two will be measured using the cosine angle calculation method. When the system finds that the cosine distance between the semantic feature vector and the semantic prototype vector in a certain collection result is greater than the preset feedback threshold (for example, 0.3), it means that the sample has significant differences in semantic expression and is a potential semantic expansion sample or context mutation sample. Such samples will be judged as valid feedback data.

[0082] Valid feedback data is automatically input into the online training process. This process is used to adjust the encoding parameters of the currently used contextual pre-trained language modeling network. The training process adopts a training-while-inference mechanism, that is, the system dynamically introduces new samples in a continuous running state and gradually updates the model weight parameters without interrupting the real-time inference task to maintain the continuity and responsiveness of the model. In order to avoid fluctuations in the system caused by frequent updates, the execution feedback module sets a training trigger cycle. This cycle can be configured to automatically execute a training round after completing a predetermined number of collection tasks (for example, every ten or twenty executions). Through this control method, the system can gradually accumulate cross-context and cross-platform semantic generalization capabilities while maintaining response speed, thereby improving the adaptability of the model to different web page structures, language styles, and content fields.

[0083] For example, during a market monitoring session, the system collected text from the "Product Parameter Description" section of a third-party platform. The text's emotional expression significantly deviated from the "neutral description" defined by the semantic prototype, exhibiting a strong "negative complaint" tone. Furthermore, the text's structural anchors were stable and the content was credible. The execution feedback module then marked this result as a valid feedback sample to improve the model's ability to identify "non-standard product emotional expressions." The execution feedback module established a closed-loop process between collection scheduling and model updating: task evaluation—priority scheduling—semantic screening—model feedback. This not only ensures the focus and efficiency of the system's collection behavior, but also enhances the model's ability to evolve in complex market contexts through a continuous feedback mechanism.

[0084] The above formulas are all dimensionless and numerical calculations. The formulas are obtained by collecting a large amount of data and performing software simulation to obtain the most recent real situation. The preset parameters in the formulas are set by technicians in this field according to actual conditions.

[0085] It should be understood that in the various embodiments of the present application, the size of the serial numbers of the above-mentioned processes does not mean the order of execution. The execution order of each process should be determined by its function and internal logic, and should not constitute any limitation on the implementation process of the embodiments of the present application.

[0086] Those skilled in the art will appreciate that the units and algorithm steps of each example described in conjunction with the embodiments disclosed herein can be implemented in electronic hardware, or a combination of computer software and electronic hardware. Whether these functions are performed in hardware or software depends on the specific application and design constraints of the technical solution. Professional and technical personnel can use different methods to implement the described functions for each specific application, but such implementation should not be considered beyond the scope of this application.

[0087] Those skilled in the art will clearly understand that, for the convenience and brevity of description, the specific working processes of the systems, devices and units described above can refer to the corresponding processes in the aforementioned method embodiments and will not be repeated here.

[0088] The above description is merely a specific embodiment of the present application, but the scope of protection of the present application is not limited thereto. Any changes or substitutions that can be easily conceived by a person skilled in the art within the technical scope disclosed in this application should be included in the scope of protection of this application. Therefore, the scope of protection of this application should be based on the scope of protection of the claims.

Claims

1. Automated data collection and decision support system for market intelligence, characterized by: It includes structure rendering module, semantic extraction module, trust evaluation module and execution feedback module; The structural rendering module obtains the visual output of the target webpage in an image rendering environment, constructs a visual structural map based on pixel density and layout sequence, identifies content cluster areas through a depth perception network, and outputs a structural anchor sequence; The semantic extraction module performs semantic vector encoding on the text content corresponding to the structural anchor sequence, extracts semantic representations using a context-pretrained language modeling network, and generates topic vectors, sentiment vectors, and intent feature vectors based on the domain corpus annotation results. The trust evaluation module receives a sequence of structural anchor points and a set of semantic vectors as input. The former calculates a structural consistency index based on structural symmetry, historical version deviation rate, and path stability factor. The latter calculates a semantic validity index based on semantic cosine similarity, topic aggregation, and sentiment certainty. These two indices are integrated into a fuzzy logic controller to output a five-level trust rating result, including level 1: certain trust, level 2: strong trust, level 3: suspicious, level 4: weak trust, and level 5: rejection. In the execution feedback module, the fuzzy logic output results refresh the collection scheduling priority, configure the data entry scheduling weight according to the five-level trust level, give priority to the execution of the first and second level resource collection tasks, and perform online training feedback on the semantic features in the collection results.

2. The automated data collection and decision support system for market intelligence according to claim 1, characterized in that: After obtaining the visual output of the target webpage in the image rendering environment, the structure rendering module performs image level normalization, divides the webpage visual output into grids according to preset pixel area windows, and constructs an initial pixel density distribution map based on the pixel density gradient change within each window; Mapping the pixel density distribution map into a layout sequence, where the layout sequence includes window arrangement order, density trend, and visual symmetry features; The deep perception network is used to identify content clusters in the layout sequence, and a region candidate mechanism is used to generate candidate marker boxes for the content clusters. Duplicate candidate boxes are then removed based on the visual similarity aggregation threshold between the boxes. By making one-to-one correspondence between the positions of candidate marker boxes and layout sequences, a structural anchor sequence that meets the stable layout structural characteristics is extracted, and the structural anchor sequence is used as one of the inputs of the trustworthy assessment module.

3. The automated data collection and decision support system for market intelligence according to claim 2, characterized in that: The depth perception network in the structure rendering module includes at least one convolutional coding layer, a position attention mapping layer, and an output prediction layer. The convolutional coding layer receives the joint expression matrix of the pixel density distribution map and the layout sequence, extracts the multi-scale feature map of the content area, and the position attention mapping layer captures the content clustering pattern in the layout sequence through a cross-window comparison mechanism and assigns attention weights to the content area. The output prediction layer generates the final content clustering area based on the position of the maximum weight. The training process of the depth perception network uses manually annotated web page content areas as supervision signals and adopts a cross entropy loss function and a position offset penalty term for joint optimization. The structure rendering module establishes a set of anchor coordinate vectors through the content cluster area output by the network, and performs an anchor sequence sorting operation on the set. The sorting standard is a combined scoring function of visual weight priority and layout stability priority.

4. The automated data collection and decision support system for market intelligence according to claim 3, characterized in that: The semantic extraction module performs semantic vector encoding on the text content corresponding to the structural anchor sequence. It first maps the position of the structural anchor sequence to the text distribution area of ​​the original webpage document, extracts the context text fragment corresponding to each anchor point, and uses each text fragment as input to complete the generation of multi-layer vectors of semantic representation through the context pre-trained language modeling network. The contextual pre-trained language modeling network uses a bidirectional attention mechanism to model the contextual linkage of text fragments. The input embedding vector is positionally encoded and then input into a multi-head attention block to obtain a contextual semantic dependency matrix, which is then output through a feedforward network. This semantic representation serves as the feature basis and is aligned with the domain corpus annotation results. The topic vector, sentiment vector, and intention feature vector are obtained through the vector mapping conversion function. The three vectors maintain independent representations within the semantic extraction module and serve as the input source of the semantic vector set that the trusted assessment module relies on when calculating the semantic validity index.

5. The automated data collection and decision support system for market intelligence according to claim 4, characterized in that: In the process of generating topic vectors, sentiment vectors, and intention feature vectors based on the domain corpus annotation results, the semantic extraction module pre-establishes a domain corpus annotation mapping table. This mapping table includes domain keyword groups, topic category codes, sentiment polarity templates, and intention template sets. The semantic representation is generated by mapping the corresponding matching rules in the mapping table. The generation of topic vectors is based on the vector inner product matching score of the semantic representation and the topic category encoding. The generation of sentiment vectors is completed through the similarity aggregation function of the semantic representation and the sentiment polarity template. The intention feature vector is output by the maximum relevance matching of the semantic representation to the intention template set; the three vectors are generated sequentially in the semantic extraction module and merged into the semantic vector set.

6. The automated data collection and decision support system for market intelligence according to claim 5, characterized in that: The structural consistency index is calculated as follows: Each anchor point in the structural anchor point sequence is considered as a graph node, and the adjacency relationship is constructed according to the visual rendering order to form a local structural subgraph. Based on the topological graph mapping mechanism, this structural subgraph is matched with the comparison graph generated by the corresponding web page structure in the historical version for subgraph isomorphism. The topological consistency score is obtained by calculating the isomorphic node ratio and the node arrangement order preservation degree. For each anchor point, a path offset sequence is constructed across multiple historical webpage versions. A multidimensional path fluctuation trajectory is constructed based on the time series. A residual analysis is performed between the current anchor point path and the trajectory mean. The trend direction and amplitude variability of the path change are compared to obtain the path stability factor. The structural consistency index is obtained by nonlinearly combining the topological consistency score and the path stability score, and the combination adopts the suboptimal suppression mechanism.

7. The automated data collection and decision support system for market intelligence according to claim 6, characterized in that: The semantic validity index is calculated as follows: The semantic cosine similarity is calculated by the cosine angle between the semantic vector and the semantic prototype vector, score S1. If S1 is greater than 0.8, it is recorded as one, otherwise it is linearly amplified by multiplying S1 by 1.25; the topic aggregation is scored S2 by counting the proportion of the number of codes belonging to the same topic in the set of semantic vectors. If the proportion exceeds 60%, S2 is equal to one, otherwise it is asymmetric mapped by multiplying the actual proportion by 1.5; the emotional certainty is calculated based on the standard deviation σ of the emotional dimension within the semantic vector. If σ is less than 0.1, it is recorded as one. If σ is between 0.1 and 0.3, the score S3 is weighted by subtracting σ from one. If σ is greater than 0.3, S3 is returned to zero; the semantic validity index is finally generated by S1, S2 and S3 through the minimum limit function F, that is, F = min (S1, S2, S3).

8. The automated data collection and decision support system for market intelligence according to claim 7, characterized in that: The trust evaluation module inputs the structural consistency index and the semantic validity index into the fuzzy logic device for trust level classification. The fuzzy logic device outputs a five-level trust level in a rule-driven manner. The logic device uses a two-dimensional fuzzy interval mapping table to construct an index response matrix. The value ranges of the structural consistency index and the semantic validity index are divided into five segments: low, medium-low, medium, medium-high, and high. The corresponding numerical ranges are 0 to 0.2, 0.2 to 0.4, 0.4 to 0.6, 0.6 to 0.8, and 0.8 to 1, respectively. The fuzzy logic device uses the intersection interval combination of the two indices as the input state. When both indexes are in the high range, the output is level one, certain credibility; when the structural consistency index is in the medium-high range and the semantic validity index is in the high range, the output is level two, strong credibility; when either index is in the medium-low range and the other index is not higher than the medium-high range, the output is level three, suspicious; when either index is lower than 0.2, the output is level four, weak credibility; when both indexes are lower than 0.2, the output is level five, rejection.

9. The automated data collection and decision support system for market intelligence according to claim 8, characterized in that: When the execution feedback module refreshes the acquisition scheduling priority based on the five-level trust level output results of the fuzzy logic device, the module presets the five-level scheduling weights as 0.9, 0.7, 0.5, 0.2 and 0, corresponding to the five levels from level 1 confidence to level 5 rejection; In the collection task queue, the execution feedback module assigns a corresponding scheduling weight based on the level label of each task to be collected, and sorts the priorities according to the weight. Tasks with weights of 0.9 and 0.7 are pushed into the scheduling execution sequence first. After the collection is completed, the execution feedback module automatically analyzes the semantic feature vectors in the collection results and selects the samples whose cosine distance with the current semantic prototype vector of the pre-trained semantic modeling network is greater than the preset feedback threshold as valid feedback data; Effective feedback data is input into the online training process to adjust the encoding parameters of the semantic modeling network. A training-while-inference mechanism is adopted to maintain continuous model updates and real-time adaptation. The update cycle can be configured to automatically trigger training every m acquisition tasks.

Images