An internet data intelligent collection and semantic understanding method and system based on a large language model

Abstract

The application discloses an internet data intelligent collection and semantic understanding method and system based on a large language model, belongs to the technical field of data processing, and generates a task target and progress constraint through acquisition of a task description instruction, and filters candidate paths in combination with a path noise proportion filtering mechanism. The semantic correlation degree is calculated based on multi-dimensional semantic representation parameters, and the correlation degree threshold value and the collection depth are dynamically adjusted according to the task progress stage, so that adaptive data collection is realized. After the collected internet data is cleaned and standardized, a large language model is used for information extraction, entity alignment and co-reference resolution, and an external knowledge graph is fused to construct a semantic understanding graph. The understanding graph and the preset task knowledge graph are compared in multiple dimensions to generate a semantic deviation value, and the noise filtering threshold value is adjusted in a closed loop according to the deviation value, so that the collection direction and the data quality are continuously optimized, and efficient and accurate internet information mining is realized.

Description

Technical Field

[0001] This invention relates to the field of data processing technology, and in particular to a method and system for intelligent collection and semantic understanding of Internet data based on a large language model. Background Technology

[0002] Existing internet data collection and semantic understanding solutions typically employ a path of "keyword matching and static rule-triggered collection, shallow parsing and rule template extraction for understanding." In the data collection phase, webpage crawling is triggered by pre-defined fixed keyword combinations or URL pattern templates. Page content is extracted using regular expressions or DOM parsing, and the collection path is filtered once based on a static noise ratio threshold. In the semantic understanding phase, after cleaning and segmenting the collected text, entities are extracted using dictionary-based or rule-based named entity recognition methods. Relationships between entities are extracted using predefined syntactic templates or pattern matching rules. Entity alignment is performed using simple string similarity, and rule-based coreference resolution addresses pronoun referencing issues. Finally, the extracted results are organized into a preliminary entity relationship graph.

[0003] In practical applications, two main bottlenecks are encountered. First, the data collection process relies on fixed keyword matching or static crawling rules for triggering and filtering, failing to understand the inherent relationship between the task objective and the page content at the semantic level. This leads to a disconnect between the collection strategy and the task objective, resulting in the indiscriminate acquisition of a large amount of irrelevant data. Simultaneously, static noise filtering and data collection depth control mechanisms lack the means to dynamically adjust the breadth and focus of data collection based on the actual progress of the task. Second, the semantic understanding stage often remains at a shallow parsing level of webpage text. Rule-based and string matching methods struggle to effectively handle ambiguities in entity names, the resolution of pronoun and indicator terms, and other issues. This prevents the integration of fragmented information into structured knowledge representation, resulting in severe entity redundancy and relationship fragmentation in the extraction results. These technical bottlenecks prevent existing technologies from effectively transforming massive amounts of unstructured internet data into high-precision structured knowledge while ensuring data collection purity, making it difficult to meet the increasingly complex needs of in-depth intelligence analysis and knowledge construction. Therefore, how to ensure that the data acquisition process is semantically aligned with the task objectives, how to endow the system with dynamic focusing capabilities, and how to improve the depth of semantic understanding to integrate fragmented information into structured knowledge have become technical problems that urgently need to be solved in this field. Summary of the Invention

[0004] To address the technical problems of existing technologies, such as the disconnect between the data collection process and the task objectives, lack of dynamic focusing capabilities, insufficient semantic understanding depth, and difficulty in integrating fragmented information into structured knowledge, this invention provides a method and system for intelligent internet data collection and semantic understanding based on a large language model. The technical solution is as follows: On the one hand, a method for intelligent collection and semantic understanding of Internet data based on a large language model is provided. This method includes: acquiring task description instructions; extracting task description instructions to generate task objectives and task progress constraints; acquiring the noise ratio of each path; determining whether it reaches a preset noise ratio threshold; filtering the path if it does, otherwise marking it as a candidate path; acquiring the semantic relevance of each candidate path based on the semantic representation parameters of the task objectives and each candidate path; dynamically adjusting the semantic relevance threshold of each task progress stage based on the task progress constraints; dynamically filtering data collection paths based on the semantic relevance of each candidate path and the adjusted semantic relevance threshold of each task progress stage; dynamically adjusting the data collection depth of each collection path based on the semantic relevance of each collection path and the task progress stage; dynamically collecting Internet data based on the data collection depth; performing semantic understanding using a large language model based on the collected Internet data to generate a semantic understanding graph; comparing the semantic understanding graph with a preset task knowledge graph to generate a semantic deviation value; and dynamically adjusting the preset noise ratio threshold based on the semantic deviation value.

[0005] On the other hand, a large language model-based intelligent internet data acquisition and semantic understanding system is provided. This system is applied to a large language model-based intelligent internet data acquisition and semantic understanding method. The system includes: a task parsing and path preprocessing module: used to obtain task description instructions, extract and generate task objectives and task progress constraints; obtain the noise ratio of each data acquisition path, compare it with a preset noise ratio threshold, filter paths exceeding the threshold, and mark the remaining paths as candidate paths; and a dynamic path filtering and threshold adjustment module: used to obtain the semantic relevance of each candidate path based on the task objective and the content semantic representation parameters of each candidate path, and dynamically adjust the threshold based on task progress constraints. The system sets semantic relevance thresholds for each task progress stage and dynamically selects data acquisition paths based on the semantic relevance of each candidate path and the adjusted semantic relevance thresholds for each task progress stage. An adaptive depth acquisition module dynamically adjusts the data acquisition depth of each acquisition path based on its semantic relevance and task progress stage, and dynamically acquires internet data based on the data acquisition depth. A semantic understanding and closed-loop feedback module performs semantic understanding using a large language model based on the acquired internet data, generating a semantic understanding graph. It then compares the semantic understanding graph with a preset task knowledge graph to generate a semantic deviation value, and dynamically adjusts a preset noise ratio threshold based on the semantic deviation value.

[0006] Beneficial effects The beneficial effects of the technical solutions provided in the embodiments of the present invention include at least the following: 1. By coupling multi-dimensional semantic representation parameters to calculate the semantic relevance of paths, dynamically and adaptively adjusting the semantic filtering threshold in different task stages, and providing a path backup and supplement mechanism, high-value collection paths and low-value paths to be activated can be accurately distinguished. This avoids the rigidity of path filtering caused by a single fixed threshold, thereby achieving dynamic and accurate filtering of Internet collection paths throughout the entire task cycle and reducing the overhead of invalid page collection.

[0007] 2. By integrating path semantic relevance and task mining stage requirements into a dual-variable adaptive adjustment of the collection depth level, and dynamically downgrading and correcting the collection configuration based on real-time data, while relying on a large language model to complete text cleaning, multi-dimensional entity matching, coreference resolution and entity normalization to construct a standardized semantic graph, the system can adapt to the information mining needs of different data sources in a layered manner, unify the structured semantic expression of heterogeneous Internet text, and thus achieve bidirectional adaptive optimization of collection depth and semantic extraction accuracy, improving the completeness and consistency of information extraction from unstructured Internet data.

[0008] 3. By extracting topological and semantic features from the graph, a multi-level feature vector is generated. The deviations of entity, relation, and attribute dimensions are calculated layer by layer and coupled to obtain a quantified semantic deviation value. Then, the path noise filtering threshold is updated in reverse based on the deviation interval. This establishes a closed-loop control logic for the entire chain from data acquisition and semantic modeling to parameter feedback, continuously corrects the previous path filtering standard, and enables the acquisition strategy to be autonomously iterated and optimized according to the semantic matching effect of the task, thereby reducing the overall noise of the acquired data and narrowing the semantic deviation between the graph and the task target. Attached Figure Description

[0009] To more clearly illustrate the technical solutions in the embodiments of the present invention, the accompanying drawings used in the description of the embodiments will be briefly introduced below. Obviously, the accompanying drawings described below are only some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0010] Figure 1 A flowchart of an intelligent Internet data acquisition and semantic understanding method based on a large language model provided in this application embodiment; Figure 2 This is a schematic diagram of the structure of the Internet data intelligent acquisition and semantic understanding system based on a large language model provided in the embodiments of this application. Detailed Implementation

[0011] The following provides explanations for some of the terms used in this application. It should be noted that these explanations are for the convenience of those skilled in the art and do not constitute a limitation on the scope of protection claimed in this application.

[0012] To make the technical problems, technical solutions and advantages of the present invention clearer, a detailed description will be given below in conjunction with the accompanying drawings and specific embodiments.

[0013] like Figure 1 The diagram shows a flowchart of an intelligent internet data acquisition and semantic understanding method based on a large language model provided in this application. The method includes the following steps: obtaining task description instructions; extracting task description instructions to generate task objectives and task progress constraints; obtaining the noise ratio of each path; determining whether a preset noise ratio threshold is reached; filtering the path if it is, otherwise marking the path as a candidate path; the noise ratio refers to the ratio of the amount of noisy data (or the number of noisy entries) to the total amount of acquireable data (or the number of entries) under that path. The noise ratio is obtained as follows: for historical acquisition paths, the noise ratio is based on the ratio of invalid data in historical acquisition records to the total amount of data. The ratio of data volume is obtained through statistical analysis. For new paths that appear for the first time, sample data is obtained through pre-collection and the noise ratio is calculated, or the noise ratio is temporarily set to a preset default value. Each path refers to a set of predefined or dynamically generated data collection sources / channels used to obtain information from the Internet, including different websites or sites (such as news portals, forums, blogs), different columns or sections of the same website (such as "technology", "finance", "sports"), specific URL patterns or templates (such as links on search results pages and tag aggregation pages), query paths generated by combinations of search keywords, social media topic streams, timelines, user homepages, etc. The semantic relevance of each candidate path is obtained based on the semantic representation parameters of the task objectives and each candidate path. The semantic relevance threshold of each task progress stage is dynamically adjusted based on the task progress constraints. Data collection paths are dynamically selected based on the semantic relevance of each candidate path and the adjusted semantic relevance threshold of each task progress stage. The data collection depth of each collection path is dynamically adjusted based on the semantic relevance of each collection path and the task progress stage. Internet data is dynamically collected based on the data collection depth. Information extraction, entity alignment and coreference resolution are performed using a large language model based on the collected Internet data to generate a semantic understanding graph. The semantic understanding graph is compared with the preset task knowledge graph to generate a semantic deviation value. The preset noise ratio threshold is dynamically adjusted based on the semantic deviation value.

[0014] Large language models refer to massive artificial neural network models based on the Transformer architecture and pre-trained on massive amounts of text data. They typically contain billions or even hundreds of billions of parameters and can capture the deep syntax, semantics, and contextual relationships of natural language, possessing comprehensive language capabilities such as text generation, logical reasoning, and knowledge-based question answering. This invention, through the construction of a fully closed-loop, dynamically adaptive overall process for internet data acquisition and semantic understanding, breaks through the technical limitations of traditional fixed-rule acquisition and static semantic analysis. It solves industry pain points such as high noise levels, fixed acquisition paths, rigid semantic matching, inability to adapt to changes in task progress, and the inability to automatically correct deviations between acquisition results and task objectives in traditional network data acquisition. Simultaneously, it does not rely on manual real-time parameter tuning, automatically parsing user task instructions, selecting high-quality acquisition paths, and dynamically adjusting selection criteria and acquisition depth according to task progress. Furthermore, it combines large language models to complete deep semantic analysis and uses the deviation results from graph comparison to iteratively optimize path filtering thresholds. This achieves a fully automated closed loop of data acquisition, semantic analysis, error feedback, and parameter iteration, significantly improving the accuracy and adaptability of heterogeneous internet data acquisition, effectively reducing the amount of invalid data acquisition, and ensuring that the final semantic understanding results align with the preset task objectives.

[0015] Preferably, the steps for obtaining the semantic relevance of each candidate path based on the task objective and the content semantic representation parameters of each candidate path include: the content semantic representation parameters include path topic similarity, path core entity overlap, and path historical information effectiveness; obtaining predefined content semantic representation critical parameters, including topic similarity critical value, entity overlap critical value, and effectiveness critical value; obtaining predefined content semantic representation correction parameters, including topic matching correction factor, entity overlap correction factor, and effectiveness correction factor; combining the ratio of path topic similarity to the topic similarity critical value with the topic matching correction factor to obtain the topic matching influence component; combining the ratio of path core entity overlap to the entity overlap critical value with the entity overlap correction factor to obtain the entity overlap influence component; combining the ratio of path historical information effectiveness to the effectiveness critical value with the effectiveness correction factor to obtain the historical effective influence component; and coupling the topic matching influence component, entity overlap influence component, and historical effective influence component to obtain the semantic relevance of each candidate path.

[0016] Path topic similarity is obtained by extracting the content topic semantic feature vectors of candidate paths and calculating their similarity with the topic semantic feature vectors of the task target. Path core entity overlap is obtained by statistically analyzing the proportion of candidate path content that overlaps with the task target's core entities to the total number of the task target's core entities. Path historical information effectiveness is obtained by statistically analyzing the proportion of effective data that meets the task semantic requirements in the path's historical data collection to the total historical data collection volume. Combination processing refers to combining a ratio (such as the ratio of similarity to the critical value) with a corresponding correction factor through a product operation. Coupling processing refers to combining multiple specific parameter values ​​(such as the topic matching influence component, entity overlap influence component, and historical effective influence component) through a summation operation. For example, let the path topic similarity be... The topic similarity threshold is The topic matching correction factor is The formula for calculating the topic matching influence component is: ; Let the overlap of the core entities along the path be... The critical value for entity overlap is The entity overlap correction factor is The components affecting the overlap of entities are: ; Assume the validity rate of historical path information is The efficiency threshold is The efficiency correction factor is The effective historical impact is: ; Then the semantic relevance of the candidate path for: ; in, , , where represents the weighting coefficient of each influencing component. The same principle applies to subsequent components.

[0017] This invention addresses the problems of traditional single-dimensional path relevance determination, such as biased results, low accuracy, and inability to quantify path acquisition value, by integrating three multi-dimensional semantic representation parameters: path topic similarity, core entity overlap, and historical information effectiveness. It also quantifies path quality from three core dimensions: topic matching, core entity matching, and historical acquisition effectiveness, obtaining precise semantic relevance values ​​through multi-component weighted coupling. This enables refined and quantifiable determination of the semantic value of acquired paths, accurately distinguishing between high-quality, effective paths and low-quality, ineffective paths, providing precise data support for subsequent path selection and acquisition depth adjustment, and improving data acquisition quality from the source.

[0018] Preferably, the step of dynamically adjusting the semantic relevance threshold of each task progress stage based on task progress constraints includes: predefining the entire task cycle into an exploration stage, a convergence stage, and a refinement stage according to the task progress constraints; setting corresponding initial semantic relevance thresholds and progress completion rate ranges for each progress stage; specifically obtaining the preset total allowed time for the task in the task progress constraints, setting two progress completion rate dividing points P1 and P2 (e.g., 30% and 70%), and dividing the entire cycle into three stages: the task progress completion rate in [0, P1) is the exploration stage, [P1, P2) is the convergence stage, and [P2, 100%] is the refinement stage; at the same time, setting an initial semantic relevance threshold for each stage, using a lower threshold in the exploration stage to broadly discover possible paths, using a medium threshold in the convergence stage to focus on high-quality sources, and using a high threshold in the refinement stage to retain only strongly relevant paths. Real-time statistics are collected on task progress status parameters, including task completion rate, remaining task duration percentage, and coverage of collected valid data. The current task completion rate is matched with the progress completion rate intervals corresponding to each progress stage to obtain the current progress stage, and the initial semantic relevance threshold corresponding to the current progress stage is obtained. The ratio of the remaining task duration percentage to the duration percentage threshold is combined with a duration percentage correction factor to obtain the duration constraint influence component. The ratio of the collected valid data coverage to the coverage threshold is combined with a coverage correction factor to obtain the information coverage influence component. The duration constraint influence component and the information coverage influence component are weighted and coupled to obtain the task progress convergence benchmark value. The task progress convergence benchmark value is input into a predefined progress convergence benchmark value-threshold dynamic correction coefficient mapping relationship to generate a threshold dynamic correction coefficient. The threshold dynamic correction coefficient is multiplied with the initial semantic relevance threshold of the corresponding stage to obtain the adjusted semantic relevance threshold for each task progress stage.

[0019] The mapping relationship between the progress convergence baseline value and the threshold dynamic correction coefficient adopts a piecewise linear function form, specifically as follows: ; in, The lower limit threshold of the convergence benchmark value, The upper limit threshold of the convergence benchmark value, The maximum value of the convergence benchmark. This is the threshold dynamic correction coefficient. A preferred value is... , , .

[0020] The task progress completion rate is obtained by real-time statistics of the ratio of the execution time from the task start time to the current time to the total allowed time of the task as defined in the task progress constraints. The remaining task time percentage is obtained by calculating the ratio of the remaining time after deducting the execution time from the total allowed time to the total allowed time of the task. The effective data coverage rate is obtained by statistically analyzing the ratio of the number of matches between the effective entities, relations, and attributes that have been collected and completed semantic parsing and the core elements in the preset task knowledge graph to the total number of core elements. This invention divides the entire task lifecycle into three core stages: exploration, convergence, and refinement. It combines three dynamic parameters—task completion rate, remaining time percentage, and effective data coverage—to adjust the semantic relevance thresholds for each stage in real time. This solves the problems of traditional technologies that use fixed thresholds throughout the process, failing to adapt to the collection needs of different task stages, leading to incomplete collection in the early stages, data redundancy in the later stages, and a disconnect between task progress and collection rhythm. Simultaneously, it automatically calculates threshold correction coefficients based on the task's progress, dynamically adapting to the information mining needs of different stages. This achieves adaptive dynamic iteration of semantic filtering thresholds, ensuring that path selection criteria align with task progress. It allows for ample data expansion in the early stages and precise refinement in the later stages, balancing the comprehensiveness and accuracy of data collection and significantly improving the overall collection efficiency.

[0021] Preferably, the step of dynamically selecting data acquisition paths based on the semantic relevance of each candidate path and the adjusted semantic relevance threshold of each task progress stage includes: comparing the semantic relevance of each candidate path with the adjusted semantic relevance threshold of the current progress stage; if the semantic relevance of a candidate path is greater than or equal to the current stage threshold, then the path is included in the formal data acquisition path set; if the semantic relevance of a candidate path is less than the current stage threshold, then the path is moved to the waiting-to-activate path pool; if the current number of formal acquisition paths is less than the preset minimum path size, then paths are added from the waiting-to-activate path pool in descending order of semantic relevance until the minimum path size requirement is met; if the current number of formal acquisition paths is greater than or equal to the preset minimum path size, then candidate paths in the formal data acquisition path set are directly marked as acquisition paths.

[0022] This invention addresses the problems of insufficient effective paths, data gaps, task interruptions, or wasted computing power caused by excessive paths in traditional dynamic filtering by establishing a hierarchical path filtering mechanism, combined with dynamic threshold comparison, a cache pool of paths to be activated, and a minimum path size fallback strategy. It strictly adheres to real-time thresholds for selecting high-quality paths while supplementing the number of paths through a cache pool to ensure a basic collection scale and prevent extreme situations from affecting task progress. This balances the accuracy of path filtering with task stability, eliminating low-value invalid paths and reducing computing power consumption while ensuring a sufficient number of collection paths and stable data collection continuity, making it suitable for various complex internet data collection scenarios.

[0023] Preferably, the step of dynamically adjusting the data acquisition depth of each acquisition path based on the semantic relevance of each acquisition path and the task progress stage includes: inputting the semantic relevance of each acquisition path into a predefined semantic relevance-acquisition depth level mapping relationship to obtain the corresponding data acquisition depth level for each acquisition path. The semantic relevance-acquisition depth level mapping relationship includes a preset semantic relevance value range and matching judgment rules for three types of acquisition depth levels. The data acquisition depth levels include deep acquisition level, balanced acquisition level, and shallow acquisition level. The initial acquisition depth benchmark value corresponding to the current data acquisition depth level is obtained. The semantic relevance-acquisition depth level mapping relationship is specifically as follows:

[0024] Based on the information mining depth requirements corresponding to the current task progress stage, the initial collection depth benchmark value is multiplied by a preset stage depth adjustment factor to obtain the target collection depth benchmark value. The stage depth adjustment factors include shallow adjustment factors, equilibrium adjustment factors, and deep adjustment factors. Specifically, the information mining depth requirements correspond to: shallow adjustment factors for the exploration stage, equilibrium adjustment factors for the convergence stage, and deep adjustment factors for the refinement stage. The target collection depth benchmark value is input into a predefined data collection depth benchmark value-data collection depth level mapping relationship to obtain the corresponding data collection depth level as the data collection depth for each collection path. The data collection depth benchmark value-page crawling layer mapping relationship includes a one-to-one correspondence rule between preset data collection depth benchmark value ranges and corresponding data collection depth level levels. The data collection depth benchmark value-data collection depth level mapping relationship is as follows:

[0025] This invention establishes a mapping relationship between semantic relevance and collection depth levels, and configures differentiated depth adjustment factors based on the information mining needs of each stage of the task. This enables adaptive adjustment of the collection path depth, solving the resource waste problems of traditional methods such as uniform collection depth for all paths, insufficient mining of high-value paths, and excessive collection of low-value paths. Furthermore, it matches different collection levels based on the semantic value of the path, and fine-tunes the collection depth according to the needs of each task stage, achieving refined allocation of collection resources. High-relevance paths deeply mine core information, while low-relevance paths shallowly filter redundant content, significantly improving the effective information output rate and reducing ineffective computing power consumption.

[0026] Preferably, the steps for dynamically collecting internet data based on data collection depth include: inputting the data collection depth level of each collection path into a predefined data collection depth level-collection task configuration mapping relationship to obtain the collection task configuration for the corresponding collection path. The collection task configuration includes page crawling depth, parsed field range, and duplicate data filtering rules. The data collection depth level-collection task configuration mapping relationship includes a one-to-one matching rule between each preset data collection depth level and the corresponding collection task configuration; dynamically collecting internet data based on the page crawling depth and parsed field range of each collection path, and performing data preprocessing through duplicate data filtering rules, and then calculating the data effectiveness rate of the returned data in real time; for collection paths with a data effectiveness rate lower than a preset effectiveness threshold, triggering downgrade processing, lowering the data collection depth level of the collection path to the next lower level, and adjusting its collection task configuration accordingly; for collection paths with a data effectiveness rate higher than or equal to the preset effectiveness threshold, maintaining the collection task configuration corresponding to the current data collection depth level.

[0027] This invention addresses the problems of fixed collection configurations failing to adapt to different data quality paths, uncontrollable collection noise, and the continuous output of invalid data from inferior paths by matching dedicated collection configurations to different collection depth levels and dynamically downgrading and optimizing collection strategies based on real-time data. It also enables differentiated configuration of crawling depth, parsing range, and deduplication rules, and allows for real-time monitoring and adaptive optimization of data quality. Furthermore, it achieves dynamic quality control during the data collection process, accurately filtering duplicate and invalid data, promptly optimizing inferior collection path strategies, and continuously improving the purity and effectiveness of the collected data.

[0028] Preferably, the steps for generating a semantic understanding graph based on collected internet data using a large language model include: cleaning, segmenting, and reorganizing the collected internet data; firstly, removing non-textual noise such as HTML tags, navigation bars, advertisements, and scripts from web pages, and uniformly encoding the text to UTF-8; then, performing preliminary sentence segmentation based on sentence-ending punctuation (., !, ?) and line breaks, and correcting erroneous segmentation using semantic boundary rules; finally, merging adjacent sentences with consistent themes into logically complete paragraphs based on the original paragraph structure or semantic coherence, and removing blank lines and other non-textual elements. The garbled text fragment is processed to obtain normalized text. The normalized text is then processed using a predefined information extraction prompt template to extract entities, entity attributes, and relationships between entities, constructing an initial graph structure. Each extracted entity mention is treated as an independent node in the graph structure, with relationships between entities serving as edges connecting the corresponding entity nodes. Attribute information for each entity node is appended to its corresponding node, forming the initial graph structure to be aligned and disambiguated. The information extraction prompt template is a carefully designed natural language instruction embedded in the input of the large language model to standardize the model's behavior in extracting structured information.It typically includes a task description (e.g., "Extract entities, attributes, and relationships from the following text"), an output format definition (e.g., JSON mode), few-shot examples, and constraints (e.g., entity type limitations) to ensure the model can stably output triples that meet expectations. A large language model is used to semantically encode each extracted entity and its context in the normalized text, generating an entity context semantic vector. Based on string similarity of entity names, feature similarity of entity attributes, and semantic similarity between entity context semantic vectors, a comprehensive matching degree between entities is calculated. The string similarity of entity names is calculated by... The edit distance (Levenshtein distance) between entity name texts is obtained by normalization. The feature similarity of entity attributes is obtained by concatenating entity attribute key-value pairs into attribute description text, encoding it into fixed-dimensional attribute feature vectors through a pre-trained language model, and then calculating the cosine similarity between two vectors. The semantic similarity between entity context semantic vectors is obtained by encoding the context window (including the sentence in which it appears and the surrounding text) of each entity using a large language model, extracting the hidden layer vector corresponding to that entity as the context semantic vector, calculating the cosine similarity between each pair of vectors, and then normalizing all three. Set corresponding weight coefficients w1, w2, and w3 (w1+w2+w3=1), and sum them to obtain the comprehensive matching score = w1 name similarity + w2 attribute similarity + w3 semantic similarity; then, based on the comprehensive matching score, filter candidate aligned entity pairs from entity pairs that reach the preset comprehensive matching score threshold. An entity pair consists of any two entities from different sources or different mentions; perform referential relationship detection on the normalized text, and identify sentences containing pronouns (he, it, its), indicator words (this, that, that), or noun phrases that may refer to others, along with the necessary context, and extract text fragments containing referential relationships; then, match the text fragments with... The entities extracted from the context of this segment are identified using a pre-defined coreference resolution prompt template. This process identifies the antecedent entities pointed to by pronouns, indicator words, and noun phrases in the text, establishing a mapping relationship between pronouns and antecedent entities. All pronouns pointing to the same antecedent entity are grouped into the same reference chain, constructing a coreference chain. The coreference resolution prompt template is a pre-defined natural language instruction template used to standardize the execution of coreference resolution tasks by large language models. It typically includes a task description (e.g., "identify the referents of pronouns, indicator words, and noun phrases in the text"), an output format (e.g., JSON, specifying the pronoun, reference type, and antecedent entity ID), and a few-sample example. The coreference chains are fused with candidate aligned entity pairs, and standard entities are matched to the entities in the candidate aligned entity pairs based on external knowledge graph links to obtain candidate standard entities. The coreference chain results obtained from coreference resolution are parsed to obtain multiple coreference clusters, each containing several entity mentions pointing to the same real-world object.Simultaneously, candidate aligned entity pairs generated in the information extraction step are obtained. These alignment pairs record the preliminary link relationships between entity mentions and standard entities in the external knowledge graph. For each core-index cluster, all external standard entities matched by candidate aligned entity pairs within the cluster are collected, forming a candidate standard entity set. A voting or maximum confidence strategy is applied to select the optimal standard entity from this set as the candidate standard entity for that core-index cluster. For isolated mentions not belonging to any core-index cluster, the link result with the highest confidence among their candidate aligned entity pairs is directly taken as their candidate standard entity. Finally, a unified mapping from each entity mention to candidate standard entities is formed. Pre-defined entity merging rules are applied to eliminate conflicting alignment relationships, and a clustering algorithm is used to merge all entity mentions pointing to the same real-world object into a unique entity node. For entity mentions that have not successfully linked to standard entities in the external knowledge graph, a hierarchical clustering algorithm is used for merging. The specific steps are as follows: 1. Encode each entity mention and its context using a large language model to obtain a context semantic vector. .

[0029] 2. Calculate mentions for any two entities. and semantic similarity between .

[0030] 3. A bottom-up agglomerative hierarchical clustering algorithm is adopted. Initially, each mention is in a separate cluster, and the two closest clusters are merged in each iteration. The distance metric used is average-linkage. .

[0031] 4. Set a similarity threshold When the distance between the two nearest clusters is greater than The merging process stops at a certain point, and the resulting clusters are treated as a set of entity references for the same real-world object.

[0032] First, pre-defined entity merging conflict rules are loaded, such as "if different mentions within the same coreference cluster are linked to different standard entities through candidate aligned entity pairs, then links consistent with the majority of mentions within the cluster are prioritized, and a few conflicting alignments are eliminated," or "if a single mention has multiple high-confidence links that conflict with each other, then the best-matching one is retained based on entity type consistency." These rules are used to resolve contradictions in alignment relationships. Then, merge clustering is performed: all mentions linked to the same candidate standard entity are directly merged into one entity node; for mentions that are not successfully linked to a standard entity, their similarity is calculated using their context semantic vectors, and mentions with similarity higher than a set threshold are grouped into one cluster using hierarchical clustering or connected component analysis. Finally, a unique entity node ID is assigned to each merge result, and the attributes and relationships of all mentions under that node are aggregated to form a unique entity node, thus completing the normalization from multiple mentions to a single graph node. The merged entity nodes replace the corresponding entities in the initial graph structure to generate a semantic understanding graph.

[0033] This invention addresses the core problems of traditional semantic understanding, such as its inability to handle heterogeneous web page data, chaotic entity recognition, ambiguous referencing, duplicate entities from multiple sources, and chaotic graph structure, through a full-process semantic processing involving multi-step data normalization, large-scale model structured information extraction, multi-dimensional entity similarity calculation, coreference resolution, and entity alignment, clustering, and normalization. Furthermore, it standardizes and cleans data to unify formats, standardizes model extraction behavior based on prompt templates, and accurately matches entities by integrating string, attribute, and semantic multi-dimensional similarity. Simultaneously, it resolves referential ambiguity, merges duplicate entities, and generates a standardized semantic understanding graph with a regular structure, unique entities, accurate relationships, and unified semantics. This completely solves the problems of semantic ambiguity and entity redundancy in messy internet data, providing a high-quality data foundation for subsequent graph comparison and deviation analysis.

[0034] Preferably, the steps for comparing the semantic understanding graph with the preset task knowledge graph to generate semantic deviation values ​​include: extracting the topological and semantic features of the semantic understanding graph to generate a first graph feature vector, and extracting the topological and semantic features of the preset task knowledge graph to generate a second graph feature vector; firstly, encoding the graph structure using a graph neural network (e.g., using GraphSAGE or GAT), using the pre-trained semantic embeddings of each entity and relation as initial node features, and obtaining node representations containing topological information through multi-layer neighbor aggregation; subsequently, using global average pooling or attention pooling, aggregating all node representations into a fixed-length graph-level semantic feature vector; simultaneously, statistical features of the graph (e.g., centrality, density, clustering coefficient) can be added as supplementary dimensions to the topological feature vector, and finally concatenating them to obtain a complete graph feature vector. The above operations are performed on the semantic understanding graph and the preset task knowledge graph respectively to obtain the first and second graph feature vectors; the first graph feature vector and the second graph feature vector are compared according to preset comparison dimensions, and the two vectors are segmented according to preset subspaces corresponding to the entity layer, relation layer, and attribute layer. Entity layer comparison: Extract entity-related feature segments, calculate the cosine distance or Euclidean distance between two vectors in the entity space, and map it to the proportion of entity missing, redundant, and ambiguous deviation values; Relationship layer comparison: Extract relationship feature segments, calculate the distance, and map it to the proportion of relationship missing, mismatch, and type deviation; Attribute layer comparison: Extract attribute feature segments, calculate the deviation, and map it to the proportion of attribute missing, value deviation, and redundancy. The deviation values ​​of each layer are uniformly normalized to the [0,1] interval for subsequent generation of semantic deviation values, thus obtaining the deviation values ​​corresponding to each comparison dimension, including the entity layer, relation layer, and attribute layer. Predefined critical parameters for the comparison dimensions are obtained, including entity deviation critical values, relation deviation critical values, and attribute deviation critical values. Predefined correction parameters for the comparison dimensions are also obtained, including entity deviation correction factors, relation deviation correction factors, and attribute deviation correction factors. The ratio of the entity layer deviation value to the entity deviation critical value is combined using the entity deviation correction factor to obtain the entity deviation impact component. The ratio of the relation layer deviation value to the relation deviation critical value is combined using the relation deviation correction factor to obtain the relation deviation impact component. The ratio of the attribute layer deviation value to the attribute deviation critical value is combined using the attribute deviation correction factor to obtain the attribute deviation impact component. The entity deviation impact component, relation deviation impact component, and attribute deviation impact component are coupled to obtain the final semantic deviation value.

[0035] Entity layer bias is obtained by matching the core entity set extracted from the semantic understanding graph with the baseline entity set of the pre-defined task knowledge graph one by one, and counting the proportion of missing entities, redundant entities, and ambiguous entities to the total number of baseline entities. Relationship layer bias is obtained by comparing the semantic relationships between corresponding entities in the two graphs one by one, and counting the proportion of missing relationships, mismatched relationships, and relationship type deviations to the total number of baseline relationships. Attribute layer bias is obtained by comparing the attribute fields and attribute values ​​corresponding to the same entity in the two graphs one by one, and counting the proportion of missing attributes, value deviations, and redundant attributes to the total number of baseline attributes. This invention utilizes graph neural network encoding to fuse graph topological and semantic features, enabling refined comparison of dual graph deviations at three levels: entities, relationships, and attributes. It combines multi-dimensional correction factors to couple and calculate the global semantic deviation value, solving the problems of traditional graph comparisons that only compare surface text, ignore topological structure and deep semantics, have one-sided deviation judgments, and cannot quantify the overall degree of deviation. Simultaneously, it can accurately identify various problems in the graph, such as missing or redundant entities, mismatched relationships, and attribute deviations, quantifying the global deviation level. This achieves a comprehensive, refined, and quantifiable assessment of semantic deviation, accurately locating deficiencies in data collection and semantic understanding, and providing accurate and reliable deviation data for subsequent closed-loop feedback and parameter tuning.

[0036] Preferably, the step of dynamically adjusting the preset noise ratio threshold based on the semantic deviation value includes: predefining a semantic deviation threshold; comparing the semantic deviation value with the semantic deviation threshold; if the semantic deviation value is greater than or equal to the semantic deviation threshold, it is determined that the current acquisition direction deviates from the task target, and threshold feedback control is performed; if the semantic deviation value is less than the semantic deviation threshold, it is determined that the current acquisition effect meets the requirements, and no additional adjustment is performed; performing threshold feedback control includes: inputting the semantic deviation value into a predefined semantic deviation-noise ratio threshold adjustment coefficient mapping relationship to obtain the noise ratio threshold adjustment coefficient; the semantic deviation-noise ratio threshold adjustment coefficient mapping relationship includes a one-to-one correspondence rule between the preset semantic deviation value range and the corresponding noise ratio threshold adjustment coefficient; the semantic deviation-noise ratio threshold adjustment coefficient mapping relationship is specifically as follows: ; In the formula, This is the current semantic deviation value. This is the semantic deviation threshold. This represents the maximum semantic deviation. For the maximum adjustment coefficient, when hour, This means that no adjustment will be made.

[0037] The current preset noise ratio threshold is multiplied by the noise ratio threshold adjustment coefficient to obtain the adjusted preset noise ratio threshold. The adjusted preset noise ratio threshold is then used as the updated preset noise ratio threshold for noise filtering in subsequent paths.

[0038] This invention addresses the problem of traditional fixed noise thresholds failing to adapt to changes in task deviation, leading to the continuous collection of invalid noise data and task deviation when the deviation is too large. Furthermore, by establishing a closed-loop control logic of deviation detection, parameter adjustment, and optimized collection, the entire collection system achieves autonomous iterative optimization. The greater the task deviation, the stricter the noise filtering standard, continuously correcting the collection direction and ensuring that the data collection and semantic understanding results consistently align with the preset task objectives, thus enhancing the system's adaptability and robustness.

[0039] like Figure 2 The diagram shown is a structural schematic of the Internet data intelligent acquisition and semantic understanding system based on a large language model provided in this application embodiment. It includes: a task parsing and path preprocessing module: used to obtain task description instructions, extract and generate task objectives and task progress constraints; obtain the noise ratio of each data acquisition path, compare it with a preset noise ratio threshold, filter paths exceeding the threshold, and mark the remaining paths as candidate paths; and a dynamic path filtering and threshold adjustment module: used to obtain the semantic relevance of each candidate path based on the task objective and the content semantic representation parameters of each candidate path, and dynamically adjust the semantic relevance of each task progress stage based on task progress constraints. The system employs several key technologies: a semantic relevance threshold and a dynamic selection of data acquisition paths based on the semantic relevance of each candidate path and the adjusted semantic relevance threshold for each task progress stage; an adaptive depth acquisition module that dynamically adjusts the data acquisition depth of each acquisition path based on its semantic relevance and task progress stage, and dynamically acquires internet data based on the data acquisition depth; and a semantic understanding and closed-loop feedback module that performs semantic understanding using a large language model based on the acquired internet data, generating a semantic understanding graph; comparing the semantic understanding graph with a preset task knowledge graph to generate a semantic deviation value, and dynamically adjusting a preset noise ratio threshold based on the semantic deviation value.

[0040] Obviously, those skilled in the art can make various modifications and variations to this application without departing from the scope and intent of this application. Therefore, if such modifications and variations fall within the scope of the claims of this application and their equivalents, this application is also intended to include such modifications and variations.

Claims

1. A method for intelligent collection and semantic understanding of internet data based on a large language model, characterized in that, Includes the following steps: Obtain the task description instructions, extract the task description instructions to generate task objectives and task progress constraints; Obtain the noise ratio of each path, determine whether it reaches the preset noise ratio threshold, filter the path if it does, otherwise mark the path as a candidate path. The semantic relevance of each candidate path is obtained based on the semantic representation parameters of the task objectives and each candidate path. The semantic relevance threshold of each task progress stage is dynamically adjusted based on the task progress constraints. The data collection path is dynamically selected based on the semantic relevance of each candidate path and the adjusted semantic relevance threshold of each task progress stage. The data acquisition depth of each acquisition path is dynamically adjusted based on the semantic relevance of each acquisition path and the task progress stage. Internet data is dynamically acquired based on the data acquisition depth, and semantic understanding is performed using a large language model based on the acquired Internet data to generate a semantic understanding graph. The semantic understanding graph is compared with the preset task knowledge graph to generate a semantic deviation value, and the preset noise ratio threshold is dynamically adjusted based on the semantic deviation value.

2. The method for intelligent collection and semantic understanding of Internet data based on a large language model as described in claim 1, characterized in that: The steps for obtaining the semantic relevance of each candidate path based on the task objective and the content semantic representation parameters of each candidate path include: The content semantic representation parameters include path topic similarity, path core entity overlap, and path historical information effectiveness. Obtain predefined content semantic representation critical parameters, including topic similarity critical value, entity overlap critical value, and efficiency critical value; obtain predefined content semantic representation correction parameters, including topic matching correction factor, entity overlap correction factor, and efficiency correction factor. The ratio of path topic similarity to topic similarity threshold is combined with the topic matching correction factor to obtain the topic matching influence component. The ratio of the overlap degree of the core entity of the path to the critical value of the overlap degree is combined with the entity overlap correction factor to obtain the entity overlap influence component. The ratio of the effectiveness of historical path information to the effectiveness threshold is combined with the effectiveness correction factor to obtain the historical effective influence component. The semantic relevance of each candidate path is obtained by coupling the topic matching influence component, the entity overlap influence component, and the historical effective influence component.

3. The method for intelligent collection and semantic understanding of Internet data based on a large language model as described in claim 1, characterized in that: The step of dynamically adjusting the semantic relevance threshold of each task progress stage based on task progress constraints includes: Based on the task progress constraints, the entire task cycle is predefined into the exploration phase, the convergence phase, and the refinement phase, and a corresponding initial semantic relevance threshold and progress completion rate range are set for each progress phase. Real-time statistics of task progress status parameters, including task completion rate, remaining task time percentage, and coverage of collected valid data; Match the current task progress completion rate with the progress completion rate intervals corresponding to each progress stage to obtain the current progress stage, and obtain the initial semantic relevance threshold corresponding to the current progress stage. The ratio of the remaining task time percentage to the time percentage threshold is combined with the time percentage correction factor to obtain the time constraint impact component. The ratio of the effective data coverage rate to the coverage threshold is combined with the coverage correction factor to obtain the information coverage impact component. By weighted coupling of the duration constraint influence component and the information coverage influence component, the task progress convergence benchmark value is obtained. The task progress convergence benchmark value is input into the predefined progress convergence benchmark value-threshold dynamic correction coefficient mapping relationship to generate the threshold dynamic correction coefficient. The threshold dynamic correction coefficient is then multiplied with the initial semantic relevance threshold of the corresponding stage to obtain the adjusted semantic relevance threshold of each task progress stage.

4. The method for intelligent collection and semantic understanding of Internet data based on a large language model as described in claim 3, characterized in that: The step of dynamically selecting data acquisition paths based on the semantic relevance of each candidate path and the adjusted semantic relevance threshold of each task progress stage includes: The semantic relevance of each candidate path is compared with the adjusted semantic relevance threshold of the current progress stage. If the semantic relevance of a candidate path is greater than or equal to the threshold of the current stage, the path is included in the formal data collection path set; if the semantic relevance of a candidate path is less than the threshold of the current stage, the path is moved to the pool of paths to be activated. If the current number of officially collected paths is lower than the preset minimum path size, then paths will be added from the pool of paths to be activated in descending order of semantic relevance until the minimum path size requirement is met. If the current number of formal data collection paths is higher than or equal to the preset minimum path size, then the candidate paths in the formal data collection path set will be directly marked as collection paths.

5. The method for intelligent collection and semantic understanding of Internet data based on a large language model as described in claim 4, characterized in that: The steps for dynamically adjusting the data acquisition depth of each acquisition path based on the semantic relevance of each acquisition path and the task progress stage include: Input the semantic relevance of each acquisition path into the predefined semantic relevance-acquisition depth level mapping relationship to obtain the corresponding data acquisition depth level of each acquisition path. The data acquisition depth level includes deep acquisition level, balanced acquisition level and shallow acquisition level. Obtain the initial acquisition depth benchmark value corresponding to the current data acquisition depth level. Based on the information mining depth requirements corresponding to the current task progress stage, the initial acquisition depth benchmark value is multiplied by the preset stage depth adjustment factor to obtain the target acquisition depth benchmark value. The stage depth adjustment factor includes a shallow adjustment factor, a balanced adjustment factor, and a deep adjustment factor. The target acquisition depth benchmark value is input into the predefined acquisition depth benchmark value-data acquisition depth level mapping relationship to obtain the corresponding data acquisition depth level as the data acquisition depth of each acquisition path.

6. The method for intelligent collection and semantic understanding of Internet data based on a large language model as described in claim 5, characterized in that: The steps for dynamically acquiring Internet data based on the data acquisition depth include: According to the data collection depth level of each collection path, the data is input into the predefined data collection depth level-collection task configuration mapping relationship to obtain the collection task configuration of the corresponding collection path. The collection task configuration includes page crawling depth, parsed field range and duplicate data filtering rules. Based on the page crawling depth and parsed field range of each collection path, Internet data is dynamically collected, and after data preprocessing through duplicate data filtering rules, the data effectiveness rate of the returned data is statistically analyzed in real time. For acquisition paths with a data effectiveness rate lower than the preset effectiveness threshold, a downgrade process is triggered, lowering the data acquisition depth level of the acquisition path to the next lower level and adjusting its acquisition task configuration accordingly; for acquisition paths with a data effectiveness rate higher than or equal to the preset effectiveness threshold, the acquisition task configuration corresponding to the current data acquisition depth level is maintained.

7. The method for intelligent collection and semantic understanding of Internet data based on a large language model as described in claim 6, characterized in that: The steps for generating a semantic understanding graph by using a large language model based on collected internet data include: The collected internet data is cleaned, segmented, and reorganized into sentences and paragraphs to obtain standardized text; The normalized text is processed using a predefined information extraction prompt template to extract entities, entity attributes, and relationships between entities, and an initial graph structure is constructed. The large language model is used to semantically encode each extracted entity and its context in the normalized text, generating an entity context semantic vector; The comprehensive matching degree between entities is calculated based on the string similarity of entity names, the feature similarity of entity attributes, and the semantic similarity between entity context semantic vectors; and candidate aligned entity pairs are selected based on the comprehensive matching degree. The normalized text is subjected to referential relationship detection, and text segments containing referential relationships are extracted; the text segments and the entities extracted in the context of the segments are identified by using a preset coreference resolution prompt template to identify the antecedent entities pointed to by pronouns, indicator words and noun phrases in the text, establish the mapping relationship between the referents and the antecedent entities, and group all referents pointing to the same antecedent entity into the same referential chain to construct a coreference chain; The coreference chain is merged with the candidate alignment entity pair, and the entity in the candidate alignment entity pair is matched with a standard entity based on the external knowledge graph link to obtain the candidate standard entity. The preset entity merging rules are applied to eliminate conflicting alignment relationships, and all entity references pointing to the same real object are merged into a unique entity node. The merged entity nodes replace the corresponding entities in the initial graph structure to generate a semantic understanding graph.

8. The method for intelligent collection and semantic understanding of Internet data based on a large language model as described in claim 7, characterized in that: The step of comparing the semantic understanding graph with the preset task knowledge graph to generate a semantic deviation value includes: Extract the topological and semantic features of the semantic understanding graph to generate a first graph feature vector, and extract the topological and semantic features of the preset task knowledge graph to generate a second graph feature vector; compare the first graph feature vector and the second graph feature vector according to preset comparison dimensions to obtain the deviation value corresponding to each comparison dimension, wherein the comparison dimensions include entity layer, relation layer and attribute layer; Obtain predefined critical parameters for comparison dimensions, including critical values ​​for entity deviation, relationship deviation, and attribute deviation; obtain predefined correction parameters for comparison dimensions, including correction factors for entity deviation, relationship deviation, and attribute deviation. The ratio of the solid layer deviation value to the solid deviation critical value is combined with the solid deviation correction factor to obtain the solid deviation influence component. The ratio of the relationship layer deviation value to the relationship deviation critical value is combined with the relationship deviation correction factor to obtain the relationship deviation influence component. The ratio of the attribute layer deviation value to the attribute deviation critical value is combined with the attribute deviation correction factor to obtain the attribute deviation influence component. The entity bias influence component, relation bias influence component, and attribute bias influence component are coupled to obtain the final semantic bias value.

9. The method for intelligent collection and semantic understanding of Internet data based on a large language model as described in claim 8, characterized in that: The step of dynamically adjusting the preset noise ratio threshold based on the semantic deviation value includes: A predefined semantic deviation threshold is defined. The semantic deviation value is compared with the semantic deviation threshold. If the semantic deviation value is greater than or equal to the semantic deviation threshold, it is determined that the current acquisition direction deviates from the task target, and threshold feedback adjustment is performed. If the semantic deviation value is less than the semantic deviation threshold, it is determined that the current acquisition effect meets the requirements, and no additional adjustment is performed. The execution threshold feedback control includes: inputting the semantic deviation value into a predefined semantic deviation-noise ratio threshold adjustment coefficient mapping relationship to obtain the noise ratio threshold adjustment coefficient; The current preset noise ratio threshold is multiplied by the noise ratio threshold adjustment coefficient to obtain the adjusted preset noise ratio threshold. The adjusted preset noise ratio threshold is then used as the updated preset noise ratio threshold for noise filtering in subsequent paths.

10. A system for intelligent acquisition and semantic understanding of internet data based on a large language model, comprising the method for intelligent acquisition and semantic understanding of internet data based on a large language model as described in any one of claims 1-9, characterized in that, include: Task parsing and path preprocessing module: used to obtain task description instructions and extract and generate task objectives and task progress constraints; Obtain the noise ratio of each data acquisition path, compare it with the preset noise ratio threshold, filter out paths that exceed the threshold, and mark the remaining paths as candidate paths. Dynamic path filtering and threshold adjustment module: It is used to obtain the semantic relevance of each candidate path based on the task objective and the semantic representation parameters of each candidate path, dynamically adjust the semantic relevance threshold of each task progress stage based on the task progress constraint, and dynamically filter the data collection path based on the semantic relevance of each candidate path and the adjusted semantic relevance threshold of each task progress stage. Adaptive Depth Acquisition Module: Used to dynamically adjust the data acquisition depth of each acquisition path based on the semantic relevance of each acquisition path and the task progress stage, and dynamically acquire Internet data based on the data acquisition depth; Semantic understanding and closed-loop feedback module: used to perform semantic understanding based on the collected Internet data using a large language model, generate a semantic understanding graph; compare the semantic understanding graph with the preset task knowledge graph to generate a semantic deviation value, and dynamically adjust the preset noise ratio threshold based on the semantic deviation value.

Images