Internet big data extraction method and device, equipment and storage medium

Abstract

The invention relates to an internet big data extraction method and device, equipment and a storage medium, and the method comprises the following steps: carrying out distributed crawler collection on an internet data source, obtaining original network data, and converting the original network data into a structured data matrix; and performing multi-level semantic analysis on the matrix, constructing a semantic feature map, and performing topic segmentation and classification to form a topic domain knowledge tree. Association rules in the knowledge tree are further mined, and an implicit knowledge network is constructed. Performing semantic decomposition and expansion on query conditions based on the network to generate expanded query data, and performing similarity matching with the knowledge network to obtain a candidate data set; and finally, performing multi-factor sorting and extraction on the candidate data, and outputting target data, thereby solving the technical problem that in the second-hand car market, due to wide data sources and fuzzy semantics, an existing system has relatively large deviation when performing price prediction and maintenance cost analysis.

Description

Technical Field

[0001] The present invention relates to the field of data information technology, and in particular to an Internet big data extraction method and related equipment. Background Art

[0002] With the rapid development of internet technology, a vast amount of used car transaction information, vehicle maintenance records, and parts price data is constantly emerging. This data holds immense commercial value and decision-making potential. However, traditional data processing methods struggle to efficiently extract valuable information from such vast and heterogeneous network data. In the used car market, in particular, the diverse data sources, diverse formats, and ambiguous semantics lead to significant bias and uncertainty in existing systems for vehicle evaluation, price prediction, and repair cost analysis.

[0003] Current used car big data analysis methods mostly rely on structured databases or simple keyword matching techniques, lacking the ability to deeply understand and mine knowledge from multi-source, heterogeneous data. For example, when assessing the actual value of a used car, traditional methods often base their estimates solely on basic fields such as model, year, and mileage, ignoring implicit but critical factors such as accident records, repair history, regional differences, and frequency of part replacements. This information gap not only reduces the accuracy of the assessment results but also increases the risk for consumers and the cost of trust in the market.

[0004] Furthermore, while some research has attempted to enhance the depth of data analysis by introducing graph technology and association rule mining, practical applications still face numerous challenges, such as insufficient semantic parsing capabilities, ambiguous topic categorization, and inefficient knowledge network construction. Existing systems struggle to effectively integrate and reason about cross-domain knowledge, particularly in complex scenarios involving used parts price depreciation and repair strategy recommendations. This limits their effectiveness in refined services and intelligent decision-making. Therefore, a big data extraction method is urgently needed that can integrate multi-source data, mine deep semantic relationships, and support intelligent query expansion. This approach can address core issues currently existing in the used car and related fields, such as information fragmentation, crude evaluation, and non-standardized services. Summary of the Invention

[0005] The main purpose of the present invention is to provide an Internet big data extraction method to solve the technical problem that in the used car market, due to the wide range of data sources and semantic ambiguity, the existing system has large deviations when performing price prediction and maintenance cost analysis.

[0006] To achieve the above object, the present invention provides a method for extracting Internet big data, comprising the following steps: Performing distributed crawler collection on the Internet data source to obtain an original network data set, and performing heterogeneous data structured conversion on the original network data set to obtain a structured data matrix; Performing multi-level semantic analysis on the structured data matrix to obtain a semantic feature map; Performing topic segmentation and classification on the semantic feature graph to obtain a topic domain knowledge tree, and performing association rule mining on the topic domain knowledge tree to obtain an implicit knowledge network; Performing semantic decomposition and expansion on the preset query conditions based on the implicit knowledge network to obtain query expansion data; Performing similarity calculation on the query expansion data and the implicit knowledge network to obtain a candidate data set; The candidate data sets are sorted and extracted based on multiple factors to obtain target extracted data.

[0007] Furthermore, the heterogeneous data structured conversion is performed on the original network dataset to obtain a structured data matrix, including: Performing heterogeneous data analysis and classification annotation on the original network data set to obtain a type identification result, and performing data cleaning and format unification processing on the original network data set based on the type identification result to obtain a standardized data stream; Performing feature extraction and dimension mapping on the standardized data stream by a deep semantic vectorization method to obtain a high-dimensional feature tensor, and performing sparsity optimization and dimensionality reduction processing on the high-dimensional feature tensor to obtain a compact feature representation; A heterogeneous data association network is constructed based on the compact feature representation to obtain a data element relationship graph, and the data element relationship graph is structured transformed to obtain a structured data matrix.

[0008] Furthermore, the semantic feature map is segmented and classified into topics through unsupervised hierarchical clustering to obtain a topic domain knowledge tree, including: Performing local neighborhood topology construction on high-dimensional semantic nodes in the semantic feature graph to obtain an adjacency similarity matrix, and performing feature subspace decomposition based on spectral analysis on the adjacency similarity matrix to obtain a low-rank embedding vector field; Performing dynamic density-aware partitioning of the high-dimensional semantic nodes based on the low-rank embedding vector field to obtain initial clustering seed clusters, and performing adaptive diffusion fusion operations on the initial clustering seed clusters to obtain a multi-level semantic condensed structure; Performing a topic discriminant function calculation on the semantic regions in the multi-level semantic cohesion structure to obtain a topic attribution probability distribution, and performing hierarchical topic label propagation on the semantic regions based on the topic attribution probability distribution to obtain a preliminary topic domain hierarchical mapping; wherein the preliminary topic domain hierarchical mapping includes a top-level abstract topic, a middle-level classification topic, and a bottom-level fine-grained topic label; The subject structure of the semantic feature map is optimized based on the preliminary subject domain hierarchical mapping to obtain a subject domain knowledge tree.

[0009] Furthermore, the subject domain knowledge tree is subjected to association rule mining to obtain an implicit knowledge network, including: Extracting contextual co-occurrence patterns of branch topic nodes in the subject domain knowledge tree based on information entropy increase to obtain a high-frequency semantic co-occurrence sequence, and performing probabilistic dependency modeling on the high-frequency semantic co-occurrence sequence based on a conditional random field to obtain a semantic dependency probability graph; Based on the semantic dependency probability graph, multi-granularity path sampling is performed on leaf-level specific topic instances in the topic domain knowledge tree to obtain a topic evolution path set, and knowledge flow propagation modeling is performed on the topic evolution path set to obtain a causal transmission weight matrix between topics; Calculating the rule confidence of the causal transmission weight matrix based on the maximum likelihood estimation method to obtain a potential association rule set, and constructing a bidirectional reasoning chain for the root subject node in the subject domain knowledge tree based on the potential association rule set to obtain a cross-domain knowledge reasoning path; Based on the cross-domain knowledge reasoning path, the subject domain knowledge tree is expanded by performing knowledge entity mapping to obtain an implicit knowledge network.

[0010] Furthermore, the semantic decomposition and expansion of the preset query conditions based on the implicit knowledge network to obtain query expansion data includes: Performing deep syntactic dependency analysis on the preset query conditions to obtain a query semantic structure tree, and performing key entity extraction and relationship identification on the query semantic structure tree to obtain a query intent skeleton, wherein the query intent skeleton includes core query entities, query constraints, relationship links between entities, and implicit query intent; Based on the query intent skeleton, the implicit knowledge network is activated with semantic neighborhood to obtain a related knowledge subgraph, and the related knowledge subgraph is traversed and aggregated with semantic paths to obtain an extended semantic set, wherein the extended semantic set includes synonymous concept groups, hyponymous and hyponymous concept chains, cross-domain associated concepts, and temporal evolution variants; Performing vector representation conversion on the extended semantic set through distributed semantic coding to obtain a semantic embedding matrix, and performing attention-weighted fusion on the semantic embedding matrix to obtain a context-enhanced query representation, wherein the context-enhanced query representation includes an explicit query vector, an implicit intent vector, a relationship constraint vector, and a timeliness weight distribution; A knowledge reasoning chain is constructed on the implicit knowledge network based on the context-enhanced query representation to obtain a multi-path reasoning graph, and joint probability inference is performed on the multi-path reasoning graph to obtain query expansion data.

[0011] Furthermore, the similarity calculation between the query expansion data and the implicit knowledge network to obtain a candidate data set includes: Performing semantic tensor decomposition on the query expansion data to obtain a multi-dimensional query feature vector, and dynamically assigning weights to the multi-dimensional query feature vector to obtain an adaptive query representation; Performing multi-scale semantic matching on the implicit knowledge network based on the adaptive query representation to obtain a candidate knowledge subgraph, and performing graph neural network embedding on the candidate knowledge subgraph to obtain a knowledge graph embedding matrix; Performing high-order similarity calculation on the adaptive query representation and the knowledge graph embedding matrix through tensor decomposition technology to obtain a multimodal similarity tensor, and performing nonlinear dimensionality reduction and cluster analysis on the multimodal similarity tensor to obtain a similarity distribution spectrum; Adaptive threshold segmentation is performed on the implicit knowledge network based on the similarity distribution spectrum to obtain a candidate data set.

[0012] Furthermore, the candidate data set is sorted and extracted based on multiple factors to obtain target extracted data, including: Performing multi-dimensional feature extraction on the candidate data set to obtain a candidate data feature matrix, and performing nonlinear feature interaction modeling on the candidate data feature matrix to obtain an interaction feature tensor; Performing dynamic time series correlation analysis on the candidate data set based on the interactive feature tensor to obtain a time series evolution sequence, and performing multi-scale wavelet transform on the time series evolution sequence to obtain a multi-resolution time-frequency feature map; Importance weighting is performed on the multi-resolution time-frequency feature map through an adaptive attention mechanism to obtain a fused feature vector, and a nonlinear projection transformation is performed on the fused feature vector to obtain a high-dimensional sorting space; The candidate data sets are sorted by multi-objective Pareto optimization based on the high-dimensional sorting space to obtain a preliminary sorting result, and target features are extracted from the preliminary sorting result to obtain target extracted data.

[0013] The present invention also provides an Internet big data extraction device, comprising: A collection module, configured to perform distributed crawler collection on the Internet data source to obtain an original network data set, and perform heterogeneous data structured conversion on the original network data set to obtain a structured data matrix; A parsing module, configured to perform multi-level semantic parsing on the structured data matrix to obtain a semantic feature map; A classification module is used to perform topic segmentation and classification on the semantic feature map to obtain a subject domain knowledge tree, and to perform association rule mining on the subject domain knowledge tree to obtain an implicit knowledge network; An expansion module, configured to perform semantic decomposition and expansion on a preset query condition based on the implicit knowledge network to obtain query expansion data; A calculation module, configured to perform similarity calculation between the query expansion data and the implicit knowledge network to obtain a candidate data set; The sorting module is used to perform multi-factor comprehensive sorting and extraction on the candidate data set to obtain target extracted data.

[0014] The present invention also provides a computer device, comprising a memory and a processor, wherein the memory stores a computer program, and the processor implements the steps of any one of the above methods when executing the computer program.

[0015] The present invention also provides a computer-readable storage medium having a computer program stored thereon, and when the computer program is executed by a processor, the steps of any of the above methods are implemented.

[0016] The present invention provides a method for extracting big data from the Internet, comprising the following steps: performing distributed crawler collection on an Internet data source to obtain an original network data set, performing heterogeneous data structured conversion on the original network data set to obtain a structured data matrix; performing multi-level semantic parsing on the structured data matrix to obtain a semantic feature map; performing topic segmentation and classification on the semantic feature map to obtain a subject domain knowledge tree, and performing association rule mining on the subject domain knowledge tree to obtain an implicit knowledge network; performing semantic decomposition and expansion on a preset query condition based on the implicit knowledge network to obtain query expansion data; performing similarity calculation on the query expansion data and the implicit knowledge network to obtain a candidate data set; and performing multi-factor comprehensive sorting and extraction on the candidate data set to obtain target extraction data. The method solves the technical problem that, in the used car market, existing systems have large deviations in price prediction and repair cost analysis due to a wide range of data sources and semantic ambiguity. The method implements semantic decomposition and expansion of user queries based on the implicit knowledge network, which not only improves query accuracy but also automatically recommends expansion conditions based on context and related knowledge, significantly improving the semantic understanding and interactive capabilities of the system. BRIEF DESCRIPTION OF THE DRAWINGS

[0017] Figure 1 This is a schematic diagram of the steps of a method for extracting Internet big data in one embodiment of the present invention; Figure 2 This is a structural block diagram of an Internet big data extraction device in one embodiment of the present invention; Figure 3 It is a schematic block diagram of the structure of a computer device according to an embodiment of the present invention.

[0018] The purpose, features and advantages of the present invention will be further described with reference to the accompanying drawings and in conjunction with the embodiments. DETAILED DESCRIPTION

[0019] In order to make the purpose, technical solutions and advantages of the present invention more clearly understood, the present invention will be further described in detail below with reference to the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are only used to explain the present invention and are not intended to limit the present invention.

[0020] like Figure 1 As shown, Figure 1 In one embodiment of the present invention, a method for extracting Internet big data includes the following steps: Step S1: performing distributed crawler collection on Internet data sources to obtain an original network data set, and performing heterogeneous data structured conversion on the original network data set to obtain a structured data matrix.

[0021] Specifically, in this step, we first perform distributed crawler collection on Internet data sources to achieve efficient acquisition of massive, multi-source network information and form an original network data set; specifically, we use a distributed architecture to deploy multiple crawler nodes to capture data content from different websites, platforms or databases in parallel, including unstructured or semi-structured web page information such as used car trading platforms, auto repair forums, and parts sales websites, to ensure that the data coverage is wide and timely. Subsequently, for the collected original network dataset, since it usually contains multiple formats such as text, tables, and image descriptions, it presents significant heterogeneous characteristics. Therefore, it is necessary to perform heterogeneous data structured conversion on it, that is, through natural language processing, HTML parsing, pattern recognition and other technical means, various types of unstructured data are uniformly mapped into operable field forms, and organized according to a preset structured model (such as a relational database table structure or tensor matrix) to generate a structured data matrix. For example, an unstructured description of "a certain car model was registered in 2018, traveled 120,000 kilometers, and the gearbox was replaced" is converted into field values ​​such as "brand = A, registration year = 2018, mileage = 120,000 km, maintenance record = gearbox replacement", so that the subsequent analysis module can carry out in-depth semantic analysis and knowledge mining based on a unified format.

[0022] Step S2: performing multi-level semantic analysis on the structured data matrix to obtain a semantic feature map.

[0023] Specifically, in the process of performing multi-level semantic analysis on structured data matrices to obtain semantic feature graphs, the core lies in extracting semantically associated feature nodes and their mutual relationships from the structured field information, and modeling and expressing them in the form of a graph structure. The specific implementation method is to first parse the field content in the structured data matrix at the lexical, syntactic and contextual levels based on natural language processing and semantic analysis technology, identify key entities and attributes such as model year, mileage, and maintenance records, and further refine their semantic labels in combination with domain knowledge bases (such as automotive parts terminology libraries and used car evaluation rule libraries); then, by constructing a semantic dependency model, the potential logical connections, causal relationships, or attribute dependencies between various fields are mapped to edges in the graph, forming a semantic feature graph consisting of nodes and edges. For example, in the application scenario of used car price evaluation, if a record contains the structured fields of "registered in 2018, mileage 130,000 kilometers, engine replaced", the system will identify "registration year", "mileage", and "engine replacement" as semantic nodes, and judge the influence weights of the three on the residual value of the vehicle based on industry experience, thereby establishing a semantic association path between them, and finally forming a knowledge graph that can reflect the multi-dimensional characteristics of the vehicle, providing semantic support for subsequent topic classification and implicit knowledge mining.

[0024] Step S3, subject segmentation and classification are performed on the semantic feature graph to obtain a subject domain knowledge tree, and association rule mining is performed on the subject domain knowledge tree to obtain an implicit knowledge network.

[0025] Specifically, when segmenting and categorizing semantic feature graphs into topics to generate a topic domain knowledge tree, the key is to analyze the association strength, semantic similarity, and contextual dependencies between nodes in the semantic feature graph to identify subgraph structures with common semantic characteristics and cluster them into several topic domains, thereby constructing a hierarchical knowledge tree. This is achieved by using a graph clustering algorithm (such as the Louvain algorithm or spectral clustering) to discover communities among nodes in the semantic feature graph. Based on semantic density and connectivity, the entire graph is divided into multiple semantically independent topic areas. Each topic area corresponds to a specific business dimension, such as "vehicle year and mileage relationship," "parts replacement frequency," or "impact of accident records." A topic domain knowledge tree is then formed by assigning topic labels and hierarchical relationships. Subsequently, association rule mining techniques (such as FP-Growth or the Apriori algorithm) are applied to this topic domain knowledge tree to further reveal potential co-occurrence patterns and causal relationships between different topics and between nodes within topics, ultimately generating an implicit knowledge network. For example, in the application scenario of used car evaluation, if the subject domain knowledge tree contains multiple subject nodes such as "high mileage", "transmission replacement", and "water blister repair", the system can mine implicit knowledge relationships such as "high mileage vehicles are more prone to transmission failures" or "the residual value of the vehicle drops significantly after water blister repair", thereby supporting subsequent intelligent query expansion and data extraction tasks.

[0026] Step S4: semantically decompose and expand the preset query conditions based on the implicit knowledge network to obtain query expansion data.

[0027] Specifically, the implicit knowledge network is used to semantically decompose and expand the pre-set query conditions. This aims to automatically identify extended information related to the query intent but not explicitly mentioned by mining the core semantic elements in the user's query statement and combining them with the existing semantic association paths in the implicit knowledge network, thereby generating more comprehensive and semantically comprehensive query expansion data. The specific implementation method is as follows: First, the original query condition entered by the user (e.g., "Find a used car in good condition, low mileage, and few maintenance records") is converted into a structured semantic expression using natural language understanding technology, and key query nodes such as "good condition," "low mileage," and "few maintenance records" are identified. Subsequently, using these nodes as starting points, the implicit knowledge network is searched for other semantic nodes with strong correlations with them, such as "no major accidents," "regular maintenance," and "no structural damage." Using graph traversal algorithms and semantic similarity calculation methods, a set of logically supported additional query conditions is expanded, ultimately forming query expansion data that includes both the original and expanded semantics. For example, in the application scenario of used car evaluation, if the user enters "query for vehicles with replaced gearboxes", the system can identify the potential correlation between this query and "high mileage", "long service life" and "engine performance degradation" through the implicit knowledge network, and then incorporate these factors into the expanded query conditions, making the subsequent candidate data set screening more accurate and more business-relevant.

[0028] Step S5: performing similarity calculation between the query expansion data and the implicit knowledge network to obtain a candidate data set.

[0029] Specifically, the process of calculating the similarity between the query expansion data and the implicit knowledge network to obtain a candidate data set is essentially to screen out the most relevant set of data as the basis for subsequent sorting and extraction by measuring the degree of match between the user's query intention and the existing semantic structure in the knowledge network. The specific implementation method is: after the query expansion data is generated, the system will map each semantic node in the query (including the original query node and the expanded semantic node) to the corresponding position in the implicit knowledge network, and calculate its association strength with each data node in the network based on the path distance in the graph structure and the semantic similarity algorithm (such as cosine similarity, Jaccard coefficient or graph embedding vector matching). Subsequently, according to the set similarity threshold, the data entries that highly match the current query are screened out to form the candidate data set. For example, in the application scenario of used car evaluation, if the user's query expansion data contains semantic nodes such as "transmission replacement", "high mileage", and "engine performance degradation", the system will identify vehicle records with similar feature combinations in the implicit knowledge network. For example, although a vehicle is not clearly marked with transmission repair, its information such as "mileage exceeding 120,000 kilometers", "low maintenance frequency", and "abnormal engine noise" has a strong semantic correlation with the query conditions, and is therefore included in the candidate data set, thereby improving the breadth and depth of data retrieval and enhancing the relevance and practicality of the results.

[0030] Step S6: perform multi-factor comprehensive sorting and extraction on the candidate data set to obtain target extracted data.

[0031] Specifically, the candidate data sets are sorted and extracted based on multiple factors, aiming to screen out the target extraction data that best meets user needs or business goals from the candidate data sets by quantitatively analyzing and assigning weights to multiple key dimensions that affect the evaluation results. The specific implementation method is: after obtaining the candidate data sets, the system will calculate a comprehensive score for each data entry based on the preset evaluation model, combined with the core influencing factors in the used car transaction scenario, such as vehicle year, mileage, maintenance records, parts replacement, price fluctuations of similar models in the market, etc.; this process usually adopts a weighted scoring method, a machine learning ranking model (such as RankNet) or a multi-criteria decision-making method (such as TOPSIS), and weightedly integrates each factor according to its importance to the final evaluation result, and forms a unified ranking index through normalization. For example, in the application scenario of used car price evaluation, if the candidate data set contains several vehicles with similar "transmission replacement" and "high mileage" characteristics, the system will further consider factors such as "whether regular maintenance is performed", "whether there is structural damage", "market price differences in the region", etc., and score and rank each vehicle accordingly, and finally extract the vehicle information with the highest ranking and the most reference value as the target extraction data, providing users with more accurate evaluation basis and decision support.

[0032] In a specific embodiment, performing heterogeneous data structured conversion on the original network dataset to obtain a structured data matrix includes: Performing heterogeneous data analysis and classification annotation on the original network data set to obtain a type identification result, and performing data cleaning and format unification processing on the original network data set based on the type identification result to obtain a standardized data stream; Performing feature extraction and dimension mapping on the standardized data stream by a deep semantic vectorization method to obtain a high-dimensional feature tensor, and performing sparsity optimization and dimensionality reduction processing on the high-dimensional feature tensor to obtain a compact feature representation; A heterogeneous data association network is constructed based on the compact feature representation to obtain a data element relationship graph, and the data element relationship graph is structured transformed to obtain a structured data matrix.

[0033] Specifically, in the above technical solution, the step of "performing a structured conversion of heterogeneous data on the original network dataset to obtain a structured data matrix" is a key preprocessing link in the entire Internet big data extraction method. Its purpose is to convert the collected original network data from an unstructured, semi-structured, multi-source heterogeneous state into a unified, analyzable structured format. Specifically, the process includes three core sub-steps: heterogeneous data parsing and classification annotation, deep semantic vectorization and feature optimization, and heterogeneous data association network construction and structured conversion. First, in the process of performing heterogeneous data parsing and classification annotation on the original network dataset to obtain type recognition results, the system will use the corresponding parser to extract content based on the different formats of the data source (such as HTML web pages, JSON interfaces, text descriptions, table information, etc.), and automatically classify and semantically annotate the data fields through natural language processing and pattern recognition technology. For example, a piece of vehicle information captured on a used car trading platform may contain content such as "year of registration," "mileage," "accident record," and "maintenance items." However, due to different expressions on different platforms, there may be multiple expressions such as "registered in 2018," "registered in 2018," or "purchased in 2018." At this time, the system will use named entity recognition (NER) and syntactic analysis technology to identify the semantic categories of these fields and uniformly label them, thereby forming a type recognition result. Subsequently, based on the type recognition result, the system will clean the raw data, remove abnormal data such as duplication, missing data, and illegal characters, and standardize all fields into a unified format. For example, "120,000 kilometers" and "120,000 kilometers" will be unified into "120,000 km," thereby generating a standardized data stream. Secondly, during the process of extracting features and mapping dimensions from the standardized data stream using deep semantic vectorization to obtain a high-dimensional feature tensor, the system employs deep learning techniques such as word embedding models (such as Word2Vec and BERT), graph neural networks (GNNs), or convolutional neural networks (CNNs) to map different types of data, including text, numeric, and categorical fields, into a unified numerical vector space representation, i.e., a high-dimensional feature tensor. For example, for the maintenance record field "Engine Replaced," the system not only encodes it as a binary variable (0 / 1) but also combines it with contextual semantics to jointly model it with fields such as "mileage," "years of use," and "maintenance frequency," forming a semantically related multidimensional feature vector. Furthermore, to further improve computational efficiency and model generalization, the system performs sparsity optimization and dimensionality reduction on this high-dimensional feature tensor. Techniques such as principal component analysis (PCA), t-SNE, or autoencoders are typically used to remove redundant information while retaining the most representative feature dimensions, ultimately forming a compact feature representation.Finally, in the process of constructing a heterogeneous data association network based on the compact feature representation to obtain a data element relationship graph, and further performing a structured transformation to form a structured data matrix, the system models the semantic dependencies, causal relationships, and co-occurrence patterns between fields as nodes and edges in the graph structure, constructing a data element relationship graph that reflects the inherent logical relationships in the data. For example, in a used car appraisal scenario, if a record contains fields such as "high mileage," "transmission replacement," and "no major accidents," the system will identify potential semantic connections between these fields and model the correlation paths between them using a graph structure. The system then transforms this data element relationship graph according to a pre-defined structured schema (such as a two-dimensional table, triple form, or tensor structure), ultimately generating a structured data matrix. Each row of this matrix corresponds to a data record, and each column represents a field dimension after semantic enhancement and feature optimization. This enables subsequent multi-level semantic parsing, topic classification, and knowledge mining tasks to be efficiently carried out on a unified, high-quality data base. In summary, this step achieves intelligent conversion from massive heterogeneous data to a structured data matrix through deep analysis, semantic vectorization modeling, and structural reconstruction of the original network dataset. This not only improves data quality and usability, but also provides a solid foundation for subsequent knowledge graph construction and intelligent query expansion. This structured conversion capability is particularly critical in the actual application of used car appraisals. It can effectively integrate fragmented information from multiple platforms, accurately depicting the historical condition and market value of each vehicle, and thus supporting more intelligent and refined decision-making and analysis.

[0034] In a specific embodiment, the subject segmentation and classification of the semantic feature graph to obtain a subject domain knowledge tree includes: Performing local neighborhood topology construction on high-dimensional semantic nodes in the semantic feature graph to obtain an adjacency similarity matrix, and performing feature subspace decomposition based on spectral analysis on the adjacency similarity matrix to obtain a low-rank embedding vector field; Performing dynamic density-aware partitioning of the high-dimensional semantic nodes based on the low-rank embedding vector field to obtain initial clustering seed clusters, and performing adaptive diffusion fusion operations on the initial clustering seed clusters to obtain a multi-level semantic condensed structure; Performing a topic discriminant function calculation on the semantic regions in the multi-level semantic cohesion structure to obtain a topic attribution probability distribution, and performing hierarchical topic label propagation on the semantic regions based on the topic attribution probability distribution to obtain a preliminary topic domain hierarchical mapping; wherein the preliminary topic domain hierarchical mapping includes a top-level abstract topic, a middle-level classification topic, and a bottom-level fine-grained topic label; The subject structure of the semantic feature map is optimized based on the preliminary subject domain hierarchical mapping to obtain a subject domain knowledge tree.

[0035] Specifically, the core goal of the "topic segmentation and classification of the semantic feature graph to obtain a topic domain knowledge tree" step is to identify semantically cohesive topic regions through structured analysis and semantic clustering of the high-dimensional semantic nodes in the semantic feature graph. Based on these regions, a hierarchical and logically clear topic domain knowledge tree is constructed. This process involves several key technical steps: local neighborhood topology construction, low-rank embedding vector field generation, dynamic density-aware partitioning, adaptive diffusion fusion, calculation of topic attribution probability distribution, and hierarchical label propagation. First, when constructing the local neighborhood topology of the high-dimensional semantic nodes in the semantic feature graph to obtain an adjacency similarity matrix, the system extracts the graph adjacency relationships for each semantic node in the graph (e.g., "mileage," "gearbox replacement," or "number of repairs"). The adjacency similarity matrix is ​​then constructed by calculating the semantic similarity between nodes (e.g., using cosine similarity or graph path similarity). This matrix reflects the local connection strength and semantic relevance between semantic nodes, providing a foundation for subsequent subspace decomposition. Next, the adjacency similarity matrix is ​​subjected to a spectral analysis-based feature subspace decomposition, with the goal of mapping the complex relationships in the high-dimensional semantic space into a low-dimensional and more interpretable feature space, thereby forming a low-rank embedding vector field. Specifically, the system uses spectral clustering technology to perform eigenvalue decomposition on the adjacency matrix, extracting the first several largest eigenvectors to form a low-dimensional embedding space, so that each semantic node is represented by a low-dimensional vector in this space, retaining the main semantic structure information in the original graph while reducing noise interference and dimensionality disaster problems. Subsequently, based on this low-rank embedding vector field, the high-dimensional semantic nodes are dynamically density-aware divided, and the system can automatically identify semantically dense areas in the graph as initial clustering seed clusters. This step usually uses algorithms such as Density Peaks Clustering or DBSCAN to select appropriate cluster centers based on the distribution density and distance characteristics of the nodes in the low-dimensional space. For example, in a used car valuation application, certain nodes (such as "accident record," "engine replacement," and "maintenance frequency") may cluster together in the semantic space, reflecting their shared influence on vehicle residual value assessment and, therefore, being grouped into the same initial cluster. Furthermore, the system performs adaptive diffusion fusion on these initial seed clusters to discover a broader range of semantic associations, forming a multi-level semantic cohesive structure. This diffusion fusion process simulates the propagation of semantic information in a graph. By setting a threshold to control the propagation range, it gradually merges adjacent clusters with high semantic similarity, ultimately constructing semantic regions at different levels of abstraction.For example, an initial cluster might contain only "transmission replacement" and "high mileage." However, after diffusion, it might incorporate nodes such as "engine performance degradation" and "missed scheduled maintenance," thereby forming a more comprehensive topic domain for "powertrain health." Next, the system calculates a topic discriminant function for the semantic regions within the resulting multi-level semantic cohesion structure to determine the probability distribution of topic affiliation for each region. This topic discriminant function is typically based on Bayesian inference or a neural network classification model, combined with existing domain knowledge bases (such as automotive repair terminology or used car appraisal standards). Each semantic region is assigned a set of possible topic labels and their corresponding probabilities. For example, a semantic region might be assigned a 0.7 probability of belonging to the topic "factors affecting vehicle condition" and a 0.3 probability of belonging to the topic "parts depreciation analysis," thus achieving soft topic segmentation. Based on this, the system further performs hierarchical topic label propagation on the semantic regions, generating a preliminary topic domain hierarchy mapping, including top-level abstract topics (such as "vehicle value assessment"), mid-level categorical topics (such as "vehicle condition analysis" and "price fluctuation factors"), and bottom-level fine-grained topic labels (such as "whether an accident has occurred" and "whether there is structural damage"). This hierarchical structure not only facilitates understanding the semantic organization behind the data but also provides a clear semantic navigation system for subsequent knowledge mining and intelligent querying. Finally, based on this preliminary topic domain hierarchical mapping, the system optimizes the topic structure of the semantic feature graph. This system integrates the topic division results from each level, removes redundant, conflicting, or ambiguous topic nodes, and uses graph reconstruction techniques to organize the results into a clearly structured and semantically coherent topic domain knowledge tree. For example, in a used car transaction evaluation, this knowledge tree can clearly express the causal chain of "vehicle year → mileage → maintenance record → market price," helping users more accurately understand the inherent connection between vehicle condition and market value. In summary, this step achieves efficient conversion from the semantic feature graph to the topic domain knowledge tree through a series of graph-based semantic modeling and topic clustering techniques. Throughout this process, the system not only preserves the rich semantic relationships in the original semantic graph but also constructs a well-organized, easily understood, and applicable topic knowledge system, laying a solid foundation for subsequent implicit knowledge mining and intelligent query expansion. In the actual application scenario of used car price evaluation, the ability to construct this topic structure is particularly important. It can help the system accurately identify the key factors affecting the residual value of the vehicle, and improve the interpretability and practicality of data analysis through structured topic organization.

[0036] In a specific embodiment, the subject domain knowledge tree is subjected to association rule mining to obtain an implicit knowledge network, including: Extracting contextual co-occurrence patterns of branch topic nodes in the subject domain knowledge tree based on information entropy increase to obtain a high-frequency semantic co-occurrence sequence, and performing probabilistic dependency modeling on the high-frequency semantic co-occurrence sequence based on a conditional random field to obtain a semantic dependency probability graph; Based on the semantic dependency probability graph, multi-granularity path sampling is performed on leaf-level specific topic instances in the topic domain knowledge tree to obtain a topic evolution path set, and knowledge flow propagation modeling is performed on the topic evolution path set to obtain a causal transmission weight matrix between topics; Calculating the rule confidence of the causal transmission weight matrix based on the maximum likelihood estimation method to obtain a potential association rule set, and constructing a bidirectional reasoning chain for the root subject node in the subject domain knowledge tree based on the potential association rule set to obtain a cross-domain knowledge reasoning path; Based on the cross-domain knowledge reasoning path, the subject domain knowledge tree is expanded by performing knowledge entity mapping to obtain an implicit knowledge network.

[0037] Specifically, the core goal of the "mining association rules within the subject domain knowledge tree to obtain an implicit knowledge network" step is to deeply analyze the semantic dependencies and contextual co-occurrence patterns between nodes within the subject domain knowledge tree, uncovering latent knowledge associations hidden beneath the explicit structure and constructing an implicit knowledge network with causal reasoning capabilities. This process includes several key implementation steps: extracting high-frequency semantic co-occurrence sequences, modeling semantic dependency probability graphs, sampling and propagation modeling of topic evolution paths, calculating causal transmission weight matrices, discovering latent association rules, constructing cross-domain reasoning chains, and expanding knowledge entity mapping. First, the system extracts contextual co-occurrence patterns from branch topic nodes within the subject domain knowledge tree using information entropy increase to obtain high-frequency semantic co-occurrence sequences. Starting from each branch node of the knowledge tree, the system scans for frequently co-occurring semantic combinations within their contexts and identifies statistically significant co-occurrence patterns. For example, in a used car appraisal application, certain topic nodes such as "high mileage," "transmission replacement," and "engine noise" may frequently appear in the same vehicle records, forming a stable semantic co-occurrence sequence. This process typically employs a sliding window mechanism combined with incremental information entropy analysis to identify highly semantically relevant and frequently distributed co-occurrence sequences, providing data support for subsequent probabilistic modeling. Subsequently, based on these high-frequency semantic co-occurrence sequences, the system further utilizes sequence modeling techniques such as Conditional Random Fields (CRFs) to construct a semantic dependency probability graph. This graph structure not only depicts the co-occurrence frequency between different topic nodes but also represents the strength of the dependency relationship between them through probabilistic edges. For example, if a sequence shows that "high mileage" always appears before "gearbox replacement," the system will establish a directional semantic dependency edge between the two and assign corresponding transition probabilities, thereby constructing a semantic dependency probability graph reflecting the semantic evolution trend. Based on this semantic dependency probability graph, the system then performs multi-granular path sampling on leaf-level specific topic instances in the topic domain knowledge tree to obtain a set of topic evolution paths. "Leaf-level specific topic instances" refer to the lowest-level, most fine-grained topic expressions in the knowledge tree, such as "whether an accident has occurred" or "whether chassis components have been replaced." By simulating multiple possible paths from the root node to the leaf nodes and performing weighted sampling based on the transition probabilities set in the semantic dependency probability graph, the system can generate a large set of semantically meaningful topic evolution paths. These paths reflect the typical evolutionary order and logical relationships between different topics in real-world data. Next, the system performs knowledge flow propagation modeling on these collected topic evolution paths, aiming to quantify the causal transmission strength between topics and ultimately form a causal transmission weight matrix.This process typically employs Bayesian network propagation models or Markov chain methods to simulate how knowledge flows along paths between topics and calculate the indirect influence weights between each pair of topics. For example, in the used car market, "high mileage" may indirectly affect "increased repair costs" through "transmission wear." This transmission relationship is captured and reflected in a causal transmission weight matrix. The system then calculates rule confidences on this causal transmission weight matrix using maximum likelihood estimation, identifying sets of potential association rules with high confidence. These rules are not simply co-occurrence relationships but rather knowledge expressions with a degree of causal explanatory power, such as "If a vehicle has structural damage, its residual value will decrease by more than 15%" or "If the maintenance interval exceeds the recommended time by 30%, the risk of engine performance degradation increases by 40%." These potential association rules constitute the core reasoning unit of the knowledge network. Furthermore, based on these potential association rules, the system constructs a bidirectional reasoning chain for the root topic node in the subject domain knowledge tree, performing forward reasoning from the root node to the leaf nodes and reverse reasoning from the leaf nodes back to the root node for reverse verification, thus forming a cross-domain knowledge reasoning path. For example, when assessing the overall value of a vehicle, the system can start from the top-level topic "Vehicle Residual Value Prediction" and infer the impact of subtopics such as "High Mileage," "Transmission Replacement," and "Unscheduled Maintenance" along the path. It then verifies whether these subtopics reasonably support the residual value prediction, thus forming a closed-loop reasoning path. Finally, based on these cross-domain knowledge reasoning paths, the system expands the subject domain knowledge tree through knowledge entity mapping, mapping abstract semantic nodes that originally existed only in the subject hierarchy to specific knowledge entities in the real world (such as part name, repair item number, price index, etc.), thereby constructing a comprehensive and semantically rich implicit knowledge network. For example, a topic node "Powertrain Failure Rate" can be mapped to a specific maintenance record database, corresponding to entity indicators such as "Number of Transmission Replacements" and "Frequency of Clutch Replacements." This ensures that the entire knowledge network not only possesses logical reasoning capabilities but also has real-world data support. In summary, this step achieves the intelligent transformation from the subject domain knowledge tree to the implicit knowledge network through a series of techniques based on probabilistic graphs, path modeling, and knowledge reasoning. Throughout this process, the hierarchical and interpretable nature of the topic structure is preserved while causal reasoning and entity mapping mechanisms are introduced. This results in a resulting implicit knowledge network with strong semantic understanding capabilities and business application value. In the actual application scenario of used car appraisals, this implicit knowledge network can help the system more accurately identify the key factors affecting vehicle residual value. Through cross-topic reasoning, it can reveal the complex relationships between parts replacement, repair history, and market price, thereby providing more intelligent and precise support for user decision-making.

[0038] In a specific embodiment, the semantic decomposition and expansion of the preset query conditions based on the implicit knowledge network to obtain query expansion data includes: Performing deep syntactic dependency analysis on the preset query conditions to obtain a query semantic structure tree, and performing key entity extraction and relationship identification on the query semantic structure tree to obtain a query intent skeleton, wherein the query intent skeleton includes core query entities, query constraints, relationship links between entities, and implicit query intent; Based on the query intent skeleton, the implicit knowledge network is activated with semantic neighborhood to obtain a related knowledge subgraph, and the related knowledge subgraph is traversed and aggregated with semantic paths to obtain an extended semantic set, wherein the extended semantic set includes synonymous concept groups, hyponymous and hyponymous concept chains, cross-domain associated concepts, and temporal evolution variants; Performing vector representation conversion on the extended semantic set through distributed semantic coding to obtain a semantic embedding matrix, and performing attention-weighted fusion on the semantic embedding matrix to obtain a context-enhanced query representation, wherein the context-enhanced query representation includes an explicit query vector, an implicit intent vector, a relationship constraint vector, and a timeliness weight distribution; A knowledge reasoning chain is constructed on the implicit knowledge network based on the context-enhanced query representation to obtain a multi-path reasoning graph, and joint probability inference is performed on the multi-path reasoning graph to obtain query expansion data.

[0039] Specifically, the core goal of the step of "performing semantic decomposition and expansion of the preset query conditions based on the implicit knowledge network to obtain query expansion data" is to achieve multi-dimensional semantic enhancement and expansion of the query intent by deeply analyzing the original query semantics of the user input and combining it with the structured semantic relationships in the constructed implicit knowledge network, thereby generating query expansion data with higher coverage and business relevance. This process consists of four key stages: query semantic structure analysis, knowledge subgraph activation and semantic aggregation, context-enhanced representation learning, and multi-path reasoning and joint inference. First, when performing deep syntactic dependency analysis on the preset query conditions to obtain a query semantic structure tree, the system utilizes natural language processing techniques, particularly dependency-based syntactic analysis, to model the grammatical structure of the user's original query statement. For example, in a used car evaluation scenario, if a user enters "I want to find a car with a replaced transmission but in good condition," the system will identify "find" as the core verb, "vehicle" as the search object, "replaced transmission" as a constraint attribute, and "good condition" as an additional condition. Based on this, a query semantic structure tree reflecting semantic master-slave relationships is constructed. The system then further extracts key entities and identifies relationships within the structure tree, extracting core query entities such as "gearbox replacement," "mileage," and "maintenance record," as well as the logical relationships between them (such as parallelism, qualifiers, and causality). This ultimately forms a query intent skeleton, which encompasses not only the explicit query content but also implicit intent that the user may not have explicitly expressed, such as "reasonable price" and "no structural damage." The system then performs semantic neighborhood activation on the implicit knowledge network based on the extracted query intent skeleton, aiming to quickly locate knowledge subgraphs relevant to the query within the vast knowledge graph. Specifically, the system uses the core entity in the query as a starting point and performs a breadth-first or depth-first search within the implicit knowledge network, activating semantic neighboring nodes within a certain range around it to form a localized, relevant knowledge subgraph. For example, for the core entity "gearbox replacement," the system might activate related nodes such as "high mileage," "missing scheduled maintenance," and "degraded engine performance," thereby constructing a semantic neighborhood network surrounding this entity. On this basis, the system performs semantic path traversal and aggregation operations on the relevant knowledge subgraph, identifying a set of concepts that are directly or indirectly related to the query topic, ultimately forming an extended semantic set. This set typically includes synonymous concept groups (such as "transmission replacement"), hierarchical and lower-level concept chains (such as "power system failure" and "component aging"), cross-domain related concepts (such as "rising repair costs"), and temporal evolution variants (such as "operating time after replacement" and "subsequent failure rate changes"). The system then converts this extended semantic set into a vector representation through distributed semantic encoding, mapping it into a unified semantic embedding space to form a semantic embedding matrix.This process typically employs semantic encoding models such as BERT, TransE, and GraphSAGE, ensuring that each semantic concept has a corresponding low-dimensional dense representation in vector space. To further enhance the contextual adaptability of the semantic representation, the system performs attention-weighted fusion on the semantic embedding matrix. Taking into account factors such as the user's query history, regional preferences, and timeliness, the system dynamically adjusts the weights of each semantic component to generate a context-enhanced query representation. This representation not only incorporates the explicit query vector (e.g., "gearbox replacement") but also incorporates implicit intent vectors (e.g., "hope the car is in good condition"), relational constraint vectors (e.g., "no major accidents"), and timeliness weight distributions (e.g., "recent repairs are of particular concern"), resulting in a highly semantically complex query expression. Finally, based on this context-enhanced query representation, the system constructs a knowledge reasoning chain within the implicit knowledge network, aiming to discover multiple reasoning paths that are highly relevant to the current query intent, thereby forming a multi-path reasoning graph. For example, when evaluating a vehicle with a replaced transmission, the system might infer multiple causal chains, such as "high mileage → transmission wear → replacement history → increased repair costs → market price reduction," and simulate the semantic transmission mechanisms of these paths using graph neural networks or probabilistic graphical models. Based on this, the system performs joint probabilistic inference on the multi-path reasoning graph, comprehensively considering the confidence, support, and relevance of each path, calculating each path's contribution to the query result and generating the final query expansion data. This expansion data includes not only the content mentioned in the user's original query but also relevant information derived from the knowledge network, such as "recommend checking maintenance records" and "pay attention to whether there is structural damage," significantly improving the intelligence of the query and the completeness of the results. In summary, this step achieves the intelligent transformation from the user's original query statement to semantically rich and comprehensive query expansion data through a series of technical approaches based on semantic structure analysis, knowledge graph activation, vectorized representation learning, and multi-path reasoning. Throughout this process, the semantic integrity of the original query is preserved while leveraging the structured semantic relationships within the implicit knowledge network to uncover a wealth of potential business-related information, providing a solid foundation for subsequent data matching and sorting. This query expansion capability is particularly important in the practical application of used car valuation. It helps the system accurately capture the user's true intent and, through a knowledge-driven approach, recommend vehicles that better meet their specific needs, thereby enhancing the overall intelligence of the service and the quality of the user experience.

[0040] In a specific embodiment, the performing similarity calculation on the query expansion data and the implicit knowledge network to obtain a candidate data set includes: Performing semantic tensor decomposition on the query expansion data to obtain a multi-dimensional query feature vector, and dynamically assigning weights to the multi-dimensional query feature vector to obtain an adaptive query representation; Performing multi-scale semantic matching on the implicit knowledge network based on the adaptive query representation to obtain a candidate knowledge subgraph, and performing graph neural network embedding on the candidate knowledge subgraph to obtain a knowledge graph embedding matrix; Performing high-order similarity calculation on the adaptive query representation and the knowledge graph embedding matrix through tensor decomposition technology to obtain a multimodal similarity tensor, and performing nonlinear dimensionality reduction and cluster analysis on the multimodal similarity tensor to obtain a similarity distribution spectrum; Adaptive threshold segmentation is performed on the implicit knowledge network based on the similarity distribution spectrum to obtain a candidate data set.

[0041] Specifically, in the step of "calculating the similarity between the query expansion data and the implicit knowledge network to obtain a candidate data set", the core lies in achieving efficient matching between the user's query intention and the entities in the knowledge graph through a multi-level, multi-dimensional semantic modeling and matching mechanism, and screening out the most relevant set of data as the basis for subsequent sorting and extraction. This process consists of four key links: multi-dimensional query feature vector construction, multi-scale semantic matching and graph embedding, high-order similarity tensor calculation, and adaptive threshold segmentation to generate candidate data sets. First, in the process of performing semantic tensor decomposition on the query expansion data to obtain a multi-dimensional query feature vector, the system converts the query representation after context enhancement (such as the explicit query vector, implicit intent vector, etc. generated in the previous step) into a high-dimensional tensor structure, and then decomposes it into multiple low-dimensional feature vectors with clear semantic meanings through tensor decomposition techniques (such as CP decomposition or Tucker decomposition). For example, in the application scenario of used car evaluation, if the user's query contains multiple extended semantic nodes such as "transmission replacement", "low mileage", and "good vehicle condition", the system can map them into feature vectors corresponding to different dimensions such as the vehicle's mechanical condition, usage intensity, and appearance maintenance, thereby forming a multi-dimensional query feature vector set. Subsequently, the system further dynamically assigns weights to these feature vectors, and adjusts the importance of each dimension in the overall query based on factors such as the user's historical preferences, query timeliness, and regional differences. It finally generates an adaptive query representation so that the query expression can more accurately reflect the core requirements of the current task. Next, based on this adaptive query representation, the system performs multi-scale semantic matching on the implicit knowledge network, with the aim of identifying local subgraph structures in the knowledge graph that are highly relevant to the current query. Specifically, the system adopts a graph search strategy to select knowledge nodes from the knowledge network that are directly or indirectly associated with the query entity to construct candidate knowledge subgraphs. For example, if the query involves "transmission replacement", the system will not only match vehicle entries that clearly mark the maintenance record, but also identify those entries that do not directly mention it but have related semantics such as "power system failure" and "engine performance degradation", thereby expanding the matching range and improving the recall rate. Subsequently, the system uses a graph neural network (GNN) to embed the candidate knowledge subgraph, uniformly mapping the information of all its nodes and edges into a shared low-dimensional vector space to form a knowledge graph embedding matrix. Each vector in the matrix represents a knowledge entity or relationship and retains its semantic position information in the original graph. On this basis, the system uses tensor decomposition technology to perform high-order similarity calculations on the adaptive query representation and the knowledge graph embedding matrix to capture the complex semantic interaction relationship between the two.Specifically, the query feature vector and the knowledge graph embedding vector are combined into a three-dimensional tensor structure. The multimodal similarity score between each query-knowledge entity pair is calculated through tensor product, cosine similarity, or other neural network scoring functions. These scores are organized into a multimodal similarity tensor, where each layer corresponds to a semantic matching mode (e.g., lexical-level matching, relation-level matching, path-level matching). To further improve the interpretability and practicality of the results, the system performs nonlinear dimensionality reduction (e.g., t-SNE or Unified Mapping) and cluster analysis on the multimodal similarity tensor to identify highly similar data clusters, ultimately forming a similarity distribution spectrum. This distribution spectrum reflects the semantic closeness between the query and each entity in the knowledge graph, facilitating subsequent precise screening. Finally, based on this similarity distribution spectrum, the system performs adaptive threshold segmentation on the implicit knowledge network. Specifically, a reasonable similarity threshold is dynamically set based on the current query confidence level, the urgency of the user's need, and historical behavior data, and all knowledge entities above this threshold are included in the candidate dataset. For example, in the practical application of used car price evaluation, even if a vehicle doesn't explicitly record a "transmission replacement," if its "low powertrain health score" or "high maintenance frequency" metrics closely match the query criteria, it will still be included in the candidate dataset, thereby improving the comprehensiveness and intelligence of the recommendation results. This adaptive mechanism ensures that the candidate dataset covers the user's primary focus while avoiding excessive noise, achieving a balance between precision and recall. In summary, this step utilizes techniques such as semantic tensor modeling, graph neural network embedding, multimodal similarity calculation, and adaptive segmentation to achieve an efficient semantic matching process from user query to candidate dataset. Throughout this process, the multidimensional nature of the query semantics is preserved while fully leveraging the structured semantic information of the implicit knowledge network, resulting in a final candidate dataset with high semantic consistency and business relevance. In the practical application of used car evaluation, this mechanism helps the system accurately identify vehicles that closely match user needs, providing a high-quality data foundation for subsequent data sorting and target extraction, significantly improving the overall system's intelligent recommendation capabilities and user experience quality.

[0042] In a specific embodiment, the multi-factor comprehensive sorting and extraction of the candidate data set to obtain target extracted data includes: Performing multi-dimensional feature extraction on the candidate data set to obtain a candidate data feature matrix, and performing nonlinear feature interaction modeling on the candidate data feature matrix to obtain an interaction feature tensor; Performing dynamic time series correlation analysis on the candidate data set based on the interactive feature tensor to obtain a time series evolution sequence, and performing multi-scale wavelet transform on the time series evolution sequence to obtain a multi-resolution time-frequency feature map; Importance weighting is performed on the multi-resolution time-frequency feature map through an adaptive attention mechanism to obtain a fused feature vector, and a nonlinear projection transformation is performed on the fused feature vector to obtain a high-dimensional sorting space; The candidate data sets are sorted by multi-objective Pareto optimization based on the high-dimensional sorting space to obtain a preliminary sorting result, and target features are extracted from the preliminary sorting result to obtain target extracted data.

[0043] Specifically, the core objective of the step of "performing a comprehensive multi-factor ranking and extraction of candidate datasets to obtain target extracted data" is to select the target extracted data from the candidate dataset that best meets user needs or business objectives by constructing a complex ranking model that integrates semantic, temporal, and multidimensional features. This process includes several key technical steps: multi-dimensional feature extraction and interaction modeling, dynamic temporal correlation analysis and time-frequency feature extraction, feature fusion and spatial mapping using an attention mechanism, and multi-objective optimization sorting based on a high-dimensional ranking space. First, during the multi-dimensional feature extraction process to obtain a candidate data feature matrix, the system extracts various types of features from candidate vehicle records, including structured fields (such as mileage and registration year), semantic labels (such as whether there has been an accident or whether the engine has been replaced), price fluctuation trends, and regional market average prices. This heterogeneous information is then organized into a structured feature matrix. Each row corresponds to a candidate data record, and each column represents a standardized feature dimension. The system then further performs nonlinear feature interaction modeling on the candidate data feature matrix. Using cross layers, product layers, or multi-head interaction modules within deep neural networks, it automatically identifies and models complex combinations of different features. For example, "high mileage plus a transmission replacement" may have a greater impact on a vehicle's residual value than a single feature alone. This generates an interactive feature tensor, enhancing the model's understanding of the inherent patterns in the data. The system then performs dynamic temporal correlation analysis on the candidate dataset based on this interactive feature tensor, aiming to capture the evolutionary trends of the candidate vehicles over time. Specifically, the system chronologically arranges each vehicle's historical maintenance records, price changes, and maintenance intervals to construct a temporal evolution sequence reflecting its state evolution. For example, in a used car evaluation scenario, a candidate vehicle might have a transmission replacement in 2021, an engine failure in 2023, and a decrease in maintenance frequency in 2024. These events constitute an evolutionary path with temporal logic. To more precisely characterize this evolutionary behavior, the system further performs a multi-scale wavelet transform on the temporal evolution sequence, extracting its feature representations at different frequency levels and forming a multi-resolution time-frequency feature map. This operation helps to identify the difference between short-term fluctuations and long-term trends, and improves the ranking model's sensitivity to dynamic changes in data. Subsequently, the system uses an adaptive attention mechanism to perform importance weighting on the obtained multi-resolution time-frequency feature maps, with the aim of identifying the most influential feature fragments in the current query context. Specifically, the system dynamically adjusts the attention weight distribution based on the contextual preferences in the query expansion data (such as whether the user focuses on "low mileage" or "no major accidents"), so that certain key time nodes or specific frequency components occupy a higher proportion in the overall feature representation.For example, if a user is particularly interested in their vehicle's recent usage, the system will assign a higher attention weight to maintenance records from 2024, while relatively reducing the impact of earlier repair events. Ultimately, the system fuses the weighted time-frequency features with the interaction features to form a fused feature vector. This vector is then projected into a high-dimensional ranking space using a nonlinear projection transformation (such as a fully connected network or kernel mapping), giving each candidate data point a comparable ranking representation within this space. Based on this, the system performs a multi-objective Pareto optimization ranking of the candidate dataset within this high-dimensional ranking space, balancing multiple conflicting but equally important evaluation metrics. For example, in a used car transaction, a user may be concerned with price rationality, vehicle stability, repair costs, and location convenience. These objectives may conflict (e.g., a vehicle in good condition but with a high price), making it impossible to rank the data using a single scoring function. In this case, the system employs a multi-objective evolutionary algorithm (such as NSGA-II) or a gradient-based Pareto frontier search method to find a set of data points in the high-dimensional ranking space that optimally compromises each objective, generating a preliminary ranking result. These results not only excel in a single dimension but also achieve a balance across multiple objectives overall. Finally, the system performs target feature extraction on the preliminary ranking results. This involves extracting key feature fields from the top-ranked candidate data based on actual business needs or user preferences, which serve as the final output target extraction data. For example, for a user's query, the system might return several vehicles with "good condition, less than 80,000 kilometers, no structural damage, and no major repairs in the past year" as recommended results. Each result is accompanied by a detailed description of the features and scoring basis, providing the user with clear, reliable, and highly relevant decision support. In summary, this step achieves intelligent ranking and precise extraction from candidate datasets to target extraction data by integrating multiple techniques, including multidimensional feature modeling, temporal evolution analysis, attention mechanisms, and multi-objective optimization. Throughout this process, the semantic integrity of the original data is preserved while incorporating complex interaction modeling and dynamic reasoning mechanisms, resulting in the final generated target extraction data with highly personalized matching capabilities and business applicability. In the actual application scenarios of used car evaluation, this multi-factor comprehensive ranking mechanism is particularly important. It can help the system quickly locate high-quality vehicle resources that best meet the user's true intentions among massive candidate data, significantly improving the platform's intelligent service capabilities and user experience quality.

[0044] The above describes the Internet big data extraction method according to the embodiment of the present invention. The following describes the Internet big data extraction device according to the embodiment of the present invention. Figure 2 In one embodiment of the present invention, an Internet big data extraction device includes: The acquisition module 21 is used to perform distributed crawler acquisition on the Internet data source to obtain an original network data set, and perform heterogeneous data structured conversion on the original network data set to obtain a structured data matrix; The parsing module 22 is used to perform multi-level semantic parsing on the structured data matrix to obtain a semantic feature map; The classification module 23 is used to perform topic segmentation and classification on the semantic feature map to obtain a topic domain knowledge tree, and to perform association rule mining on the topic domain knowledge tree to obtain an implicit knowledge network; An expansion module 24 is configured to perform semantic decomposition and expansion on a preset query condition based on the implicit knowledge network to obtain query expansion data; A calculation module 25 is used to perform similarity calculation between the query expansion data and the implicit knowledge network to obtain a candidate data set; The sorting module 26 is used to perform multi-factor comprehensive sorting and extraction on the candidate data sets to obtain target extracted data.

[0045] In this embodiment, for the specific implementation of each unit in the above device embodiment, please refer to the above method embodiment, which will not be repeated here.

[0046] Reference Figure 3 In an embodiment of the present invention, a computer device is also provided, wherein the internal structure of the computer device can be as follows: Figure 3 As shown. The computer device includes a processor, memory, display screen, input device, network interface and database connected via a system bus. The processor of the computer design is used to provide computing and control capabilities. The memory of the computer device includes a non-volatile storage medium and an internal memory. The non-volatile storage medium stores an operating system, a computer program and a database. The internal memory provides an environment for the operation of the operating system and computer program in the non-volatile storage medium. The database of the computer device is used to store the corresponding data in this embodiment. The network interface of the computer device is used to communicate with an external terminal via a network connection. When the computer program is executed by the processor, the above method is implemented.

[0047] Those skilled in the art will understand that Figure 3 The structure shown in the figure is merely a block diagram of a portion of the structure related to the solution of the present invention and does not constitute a limitation on the computer device to which the solution of the present invention is applied.

[0048] An embodiment of the present invention further provides a computer-readable storage medium having a computer program stored thereon, which implements the above-described method when executed by a processor. It is understood that the computer-readable storage medium in this embodiment can be a volatile readable storage medium or a non-volatile readable storage medium.

[0049] Those skilled in the art will appreciate that all or part of the processes in the above-described method embodiments can be implemented by instructing the relevant hardware using a computer program. The computer program can be stored in a non-volatile computer-readable storage medium. When executed, the computer program can include the processes of the above-described method embodiments. Any reference to memory, storage, database, or other media provided herein and used in the embodiments may include non-volatile and / or volatile memory. Non-volatile memory may include read-only memory (ROM), programmable ROM (PROM), electrically programmable ROM (EPROM), electrically erasable programmable ROM (EEPROM), or flash memory. Volatile memory may include random access memory (RAM) or external cache memory. By way of illustration and not limitation, RAM is available in various forms, such as static RAM (SRAM), dynamic RAM (DRAM), synchronous DRAM (SDRAM), double-speed SDRAM (SSRSDRAM), enhanced SDRAM (ESDRAM), synchronous link DRAM (SLDRAM), RAMbus direct RAM (RDRAM), direct RAMbus dynamic RAM (DRDRAM), and RAMbus dynamic RAM.

[0050] It should be noted that, in this document, the terms "comprises," "includes," or any other variations thereof are intended to encompass non-exclusive inclusion, such that a process, apparatus, article, or method comprising a series of elements includes not only those elements but also other elements not explicitly listed, or elements inherent to such process, apparatus, article, or method. In the absence of further limitations, an element defined by the phrase "comprising a ..." does not exclude the presence of other identical elements in the process, apparatus, article, or method comprising the element.

[0051] The above description is only a preferred embodiment of the present invention and does not limit the patent scope of the present invention. Any equivalent structure or equivalent process transformation made by using the contents of the present invention description and drawings, or directly or indirectly applied in other related technical fields, are also included in the patent protection scope of the present invention.

Claims

1. A method for extracting Internet big data, characterized in that: The following steps are involved: Performing distributed crawler collection on Internet data sources to obtain original network data sets, and performing heterogeneous data structured conversion on the original network data sets to obtain a structured data matrix; Performing multi-level semantic analysis on the structured data matrix to obtain a semantic feature map; Performing topic segmentation and classification on the semantic feature graph to obtain a topic domain knowledge tree, and performing association rule mining on the topic domain knowledge tree to obtain an implicit knowledge network; Performing semantic decomposition and expansion on the preset query conditions based on the implicit knowledge network to obtain query expansion data; Performing similarity calculation on the query expansion data and the implicit knowledge network to obtain a candidate data set; The candidate data sets are sorted and extracted based on multiple factors to obtain target extracted data.

2. The Internet big data extraction method according to claim 1, characterized in that: The performing heterogeneous data structured conversion on the original network dataset to obtain a structured data matrix includes: Performing heterogeneous data analysis and classification annotation on the original network data set to obtain a type identification result, and performing data cleaning and format unification processing on the original network data set based on the type identification result to obtain a standardized data stream; Performing feature extraction and dimension mapping on the standardized data stream by a deep semantic vectorization method to obtain a high-dimensional feature tensor, and performing sparsity optimization and dimensionality reduction processing on the high-dimensional feature tensor to obtain a compact feature representation; A heterogeneous data association network is constructed based on the compact feature representation to obtain a data element relationship graph, and the data element relationship graph is structured transformed to obtain a structured data matrix.

3. The Internet big data extraction method according to claim 1, characterized in that: The subject segmentation and classification of the semantic feature graph to obtain a subject domain knowledge tree includes: Performing local neighborhood topology construction on high-dimensional semantic nodes in the semantic feature graph to obtain an adjacency similarity matrix, and performing feature subspace decomposition based on spectral analysis on the adjacency similarity matrix to obtain a low-rank embedding vector field; Performing dynamic density-aware partitioning of the high-dimensional semantic nodes based on the low-rank embedding vector field to obtain initial clustering seed clusters, and performing adaptive diffusion fusion operations on the initial clustering seed clusters to obtain a multi-level semantic condensed structure; Performing a topic discriminant function calculation on the semantic regions in the multi-level semantic cohesion structure to obtain a topic attribution probability distribution, and performing hierarchical topic label propagation on the semantic regions based on the topic attribution probability distribution to obtain a preliminary topic domain hierarchical mapping; wherein the preliminary topic domain hierarchical mapping includes a top-level abstract topic, a middle-level classification topic, and a bottom-level fine-grained topic label; The subject structure of the semantic feature map is optimized based on the preliminary subject domain hierarchical mapping to obtain a subject domain knowledge tree.

4. The Internet big data extraction method according to claim 1, characterized in that: The performing of association rule mining on the subject domain knowledge tree to obtain an implicit knowledge network includes: Extracting contextual co-occurrence patterns of branch topic nodes in the subject domain knowledge tree based on information entropy increase to obtain a high-frequency semantic co-occurrence sequence, and performing probabilistic dependency modeling on the high-frequency semantic co-occurrence sequence based on a conditional random field to obtain a semantic dependency probability graph; Based on the semantic dependency probability graph, multi-granularity path sampling is performed on leaf-level specific topic instances in the topic domain knowledge tree to obtain a topic evolution path set, and knowledge flow propagation modeling is performed on the topic evolution path set to obtain a causal transmission weight matrix between topics; Calculating the rule confidence of the causal transmission weight matrix based on the maximum likelihood estimation method to obtain a potential association rule set, and constructing a bidirectional reasoning chain for the root subject node in the subject domain knowledge tree based on the potential association rule set to obtain a cross-domain knowledge reasoning path; Based on the cross-domain knowledge reasoning path, the subject domain knowledge tree is expanded by performing knowledge entity mapping to obtain an implicit knowledge network.

5. The Internet big data extraction method according to claim 1, characterized in that: The semantic decomposition and expansion of the preset query conditions based on the implicit knowledge network to obtain query expansion data includes: Performing deep syntactic dependency analysis on the preset query conditions to obtain a query semantic structure tree, and performing key entity extraction and relationship identification on the query semantic structure tree to obtain a query intent skeleton, wherein the query intent skeleton includes core query entities, query constraints, relationship links between entities, and implicit query intent; Based on the query intent skeleton, the implicit knowledge network is activated with semantic neighborhood to obtain a related knowledge subgraph, and the related knowledge subgraph is traversed and aggregated with semantic paths to obtain an extended semantic set, wherein the extended semantic set includes synonymous concept groups, hyponymous and hyponymous concept chains, cross-domain associated concepts, and temporal evolution variants; Performing vector representation conversion on the extended semantic set through distributed semantic coding to obtain a semantic embedding matrix, and performing attention-weighted fusion on the semantic embedding matrix to obtain a context-enhanced query representation, wherein the context-enhanced query representation includes an explicit query vector, an implicit intent vector, a relationship constraint vector, and a timeliness weight distribution; A knowledge reasoning chain is constructed on the implicit knowledge network based on the context-enhanced query representation to obtain a multi-path reasoning graph, and joint probability inference is performed on the multi-path reasoning graph to obtain query expansion data.

6. The Internet big data extraction method according to claim 1, characterized in that: The step of performing similarity calculation on the query expansion data and the implicit knowledge network to obtain a candidate data set includes: Performing semantic tensor decomposition on the query expansion data to obtain a multi-dimensional query feature vector, and dynamically assigning weights to the multi-dimensional query feature vector to obtain an adaptive query representation; Performing multi-scale semantic matching on the implicit knowledge network based on the adaptive query representation to obtain a candidate knowledge subgraph, and performing graph neural network embedding on the candidate knowledge subgraph to obtain a knowledge graph embedding matrix; Performing high-order similarity calculation on the adaptive query representation and the knowledge graph embedding matrix through tensor decomposition technology to obtain a multimodal similarity tensor, and performing nonlinear dimensionality reduction and cluster analysis on the multimodal similarity tensor to obtain a similarity distribution spectrum; Adaptive threshold segmentation is performed on the implicit knowledge network based on the similarity distribution spectrum to obtain a candidate data set.

7. The Internet big data extraction method according to claim 1, characterized in that: The multi-factor comprehensive sorting and extraction of the candidate data set to obtain target extracted data includes: Performing multi-dimensional feature extraction on the candidate data set to obtain a candidate data feature matrix, and performing nonlinear feature interaction modeling on the candidate data feature matrix to obtain an interaction feature tensor; Performing dynamic time series correlation analysis on the candidate data set based on the interactive feature tensor to obtain a time series evolution sequence, and performing multi-scale wavelet transform on the time series evolution sequence to obtain a multi-resolution time-frequency feature map; Importance weighting is performed on the multi-resolution time-frequency feature map through an adaptive attention mechanism to obtain a fused feature vector, and a nonlinear projection transformation is performed on the fused feature vector to obtain a high-dimensional sorting space; The candidate data sets are sorted by multi-objective Pareto optimization based on the high-dimensional sorting space to obtain a preliminary sorting result, and target features are extracted from the preliminary sorting result to obtain target extracted data.

8. An Internet big data extraction device, characterized in that: include: A collection module, configured to perform distributed crawler collection on the Internet data source to obtain an original network data set, and perform heterogeneous data structured conversion on the original network data set to obtain a structured data matrix; A parsing module, configured to perform multi-level semantic parsing on the structured data matrix to obtain a semantic feature map; A classification module is used to perform topic segmentation and classification on the semantic feature map to obtain a subject domain knowledge tree, and to perform association rule mining on the subject domain knowledge tree to obtain an implicit knowledge network; An expansion module, configured to perform semantic decomposition and expansion on a preset query condition based on the implicit knowledge network to obtain query expansion data; A calculation module, configured to perform similarity calculation between the query expansion data and the implicit knowledge network to obtain a candidate data set; The sorting module is used to perform multi-factor comprehensive sorting and extraction on the candidate data set to obtain target extracted data.

9. A computer device comprising a memory and a processor, wherein a computer program is stored in the memory, wherein: When the processor executes the computer program, the steps of the method according to any one of claims 1 to 7 are implemented.

10. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, the steps of the method according to any one of claims 1 to 7 are implemented.

Images