A method for automatically generating massive, high-quality datasets based on feature recognition
By constructing a feature topological manifold structure and a logical feature completion matrix, the problem of limited feature coverage in existing datasets is solved, generating high-quality massive datasets, deeply exploring the inherent topological relationships in the feature space of the data, and improving the logical rigor and distribution coverage of the training data.
Patent Information
- Application Number
- CN202611276365.6
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2026-08-21
- Publication Date
- 2026-11-17
AI Technical Summary
Existing technologies rely on manual location of key points and edge delineation, resulting in limited feature coverage and a single distribution pattern in the generated datasets. This makes it difficult to deeply explore the inherent topological relationships within the data's feature space, leading to insufficient logical rigor and distribution coverage in the training data, and resulting in isolated logical points and data redundancy.
By measuring the Euclidean distance and cosine correlation of feature nodes, a topological structure graph is constructed, node coordinates and weight vectors are extracted, the activation interval of feature terms is determined, a logical feature completion matrix is established, the gradient flow divergence feature of coordinate points is measured, missing regions are marked and new sample points are constructed by interpolation, global manifold constraints and coordinate corrections are performed, and a massive high-quality dataset is generated.
It enables the automatic generation of massive, high-quality datasets, improves the logical rigor and distributed coverage of the datasets, eliminates isolated logical points and data redundancy, and deeply explores the inherent feature space topological relationships of the data.
Smart Images

Figure CN123115791A_ABST
Abstract
Description
Technical Field
[0001] This invention relates to the field of data processing technology, and in particular to a method for automatically generating massive, high-quality datasets based on feature recognition. Background Technology
[0002] The field of data processing technology involves the conversion, storage, retrieval, and logical computation of raw digital signals. It utilizes the instruction cycles of computing devices to clean, merge, and standardize multi-source heterogeneous data to form logical sequences suitable for analysis. Among these methods, the automatic generation of massive, high-quality datasets refers to the process of building underlying training resources for machine learning models. This typically involves annotators using graphical interface tools to locate key points and delineate target edges in original images or attach semantic tags to text paragraphs. Offline incremental processing is then performed on existing samples using geometric transformation functions such as random flipping, brightness adjustment, and translation / scaling. Simultaneously, outliers are removed based on manually verified consistency criteria, resulting in a training dataset with a specific distribution pattern.
[0003] In the current equipment data processing process, key point positioning and edge delineation mainly rely on manual labor. The operation process is limited by the subjective judgment criteria of the annotator, and there is a risk of data label logic being broken. Offline incremental processing is carried out by relying on geometric transformations such as random flipping, translation and scaling, which cannot deeply explore the internal feature space topological relationship of the data. The generated sample feature coverage is limited and the distribution pattern is simple. It lacks an effective expression of the feature dimension manifold structure, making it difficult to identify deep semantic logic, resulting in isolated logical points and data redundancy. The operation logic lacks the ability to cover the logical rigor and overall distribution of the training data. Summary of the Invention
[0004] The purpose of this invention is to address the shortcomings of existing technologies by proposing a method for automatically generating massive, high-quality datasets based on feature recognition.
[0005] To achieve the above objectives, the present invention adopts the following technical solution: a method for automatically generating massive high-quality datasets based on feature recognition, comprising the following steps:
[0006] S1: Collect the original feature data points of the acquisition device, determine the Euclidean distance and cosine correlation of the feature nodes, determine the node association strength, construct the topology graph, and obtain the feature topology manifold structure;
[0007] S2: Based on the feature topological manifold structure, match the preset semantic hierarchical logic matrix, extract the manifold node coordinates and logic branch weight vectors, perform vector inner product operation and determine that the feature terms are in the activation interval, and obtain the semantic logic feature completion matrix.
[0008] S3: Complete the semantic logic feature matrix, extract sample space coordinate points, measure the gradient flow divergence features of each coordinate point, compare the sample distribution density with the global expected distribution, mark the location of feature missing areas with a density level higher than the set threshold, statistically analyze the gradient deviation of each dimension of the feature missing area, and establish a gradient flow-guided feature distribution field.
[0009] S4: Based on the gradient flow-guided feature distribution field, interpolate to construct new sample points in the feature missing region, compare the gradient consistency deviation between the new sample points and adjacent sample points, determine that the deviation is within the preset gradient tolerance range, and generate logical associations to complete the sample data.
[0010] S5: Complete the sample data according to the logical association, perform global manifold constraints and sample point coordinate correction, adjust the spatial distribution of the dataset, eliminate logical island data, and output a massive high-quality device dataset.
[0011] As a further aspect of the present invention, the feature term being in the activation interval specifically means that the result of the inner product operation is less than a set logical activation threshold.
[0012] As a further aspect of the present invention, the feature topological manifold structure includes local curvature, adjacency association table, manifold dimension, and geometric centroid coordinates; the semantic logical feature completion matrix includes logical topological path, feature activation state, association probability mapping, and label weight index; the gradient flow-guided feature distribution field includes gradient divergence features, potential energy distribution, vector flow skew angle, and density deviation topological graph; the logical association completion sample data includes completion feature vector, logical cohesion coefficient, and spatial clustering bias; and the massive high-quality device dataset includes purified data samples, standardized feature matrix, and verification label metadata.
[0013] As a further aspect of the present invention, the specific steps for obtaining the characteristic topological manifold structure are as follows:
[0014] S111: Collect the original feature data points of the device, calculate the L2 norm of the spatial coordinate vector between the original feature data points of the device, and simultaneously determine the cosine correlation of the angle between the direction vectors between the original feature data points of the device. Based on the numerical mapping logic of the original feature data points of the device, the Euclidean distance and the cosine correlation, establish a node feature attribute association set and generate a feature node metric matrix.
[0015] S112: For the feature node metric matrix, retrieve the element values of each dimension in the feature node metric matrix, and perform a comparison operation between the element values and the preset association strength determination threshold. Combine the node association strength to establish logical topological connection edges between each feature node and perform node connectivity construction operation according to the logical topological connection edges to obtain the device association topology map.
[0016] S113: Based on the device association topology graph, extract the geometric manifold distribution features within the local neighborhood of each node in the device association topology graph, perform Laplace eigenmap processing and transform the device association topology graph into a nonlinear manifold space coordinate system, calculate the Euclidean coordinate values of each node in the nonlinear manifold space coordinate system, and generate a feature topology manifold structure.
[0017] As a further aspect of the present invention, the specific steps for obtaining the semantic logic feature completion matrix are as follows:
[0018] S211: Obtain a preset semantic hierarchy logic matrix, perform node matching based on the feature topology manifold structure, extract the manifold node coordinates within the feature topology manifold structure and the corresponding logical branch weight vector of the semantic hierarchy logic matrix, perform semantic association mapping between the manifold node coordinates and the logical branch weight vector, and generate a node logical feature distribution matrix.
[0019] S212: Based on the node logical feature distribution matrix, retrieve vector elements and perform vector inner product operation to obtain a projected activation scalar value. Compare the projected activation scalar value with a set logical activation threshold and determine that the projected activation scalar value is less than the set logical activation threshold activation condition to obtain an activation determination state value.
[0020] S213: For the activation determination state value, the feature item is determined to be in the activation interval. The feature item is retrieved and its neighborhood distribution is retrieved within the feature topology manifold structure. Based on the projected activation scalar value and the original attribute data of the feature node, missing item prediction is performed and node feature completion increment is determined to generate a feature increment mapping vector.
[0021] S214: Based on the feature increment mapping vector, perform dimension merging in conjunction with the node logical feature distribution matrix, and superimpose the feature increment mapping vector with the original attribute data of the feature node to establish a node association mapping relationship and obtain a semantic logical feature completion matrix.
[0022] As a further aspect of the present invention, the specific steps for obtaining the gradient flow-guided feature distribution field are as follows:
[0023] S311: Complete the semantic logic feature matrix, obtain the sample space coordinate points, retrieve the distribution vector of the sample space coordinate points in Euclidean space, determine the gradient flow divergence feature of each coordinate point and perform partial derivative operation, and perform feature vector and coordinate index association encapsulation based on the gradient flow divergence feature of each coordinate point to generate a coordinate gradient feature mapping table.
[0024] S312: Based on the coordinate gradient feature mapping table, extract the sample distribution attributes within the local window range, count the density value of the sample local region, compare the difference between the sample local region density value and the global expected distribution, calculate the density judgment deviation value, compare the density judgment deviation value with the set density level threshold execution value, mark the location of the feature missing area where the density judgment deviation value is higher than the set density level threshold, and obtain the feature missing block coordinate set.
[0025] S313: Based on the coordinate set of the missing feature blocks, retrieve the corresponding gradient components of the covered area using the coordinate gradient feature mapping table, calculate the gradient deviation of each dimension of the missing feature area, perform potential energy distribution mapping on the manifold space for the gradient deviation of each dimension of the missing feature area, and establish a gradient flow-guided feature distribution field.
[0026] As a further aspect of the present invention, the formula for calculating the density determination deviation value is as follows:
[0027] ;
[0028] in, This represents the deviation value for determining density. Represents the number of samples in a local area. Represents the volume of a local area. Represents the global expected distribution density. Represents the amplitude of the gradient flow divergence characteristic. Represents a unit spatial reference scale. This represents the quality constant of the benchmark sample.
[0029] As a further aspect of the present invention, the specific steps for obtaining the logical association completion sample data are as follows:
[0030] S411: Extract the extreme values of the gradient gradient and spatial evolution trend of the distribution potential energy of the gradient flow-guided feature distribution field, retrieve the location of the feature missing region and obtain the local manifold curvature parameters of the feature missing region, calculate the mean density of known sample points within the local window, perform nonlinear surface interpolation operation and interpolate to construct new sample points in the feature missing region, calculate the spatial dimension components of the new node and perform coordinate transformation, and establish the interpolated sample space coordinate matrix.
[0031] S412: Obtain the local gradient vector and tangent plane normal direction of each node in the interpolation sample space coordinate matrix, compare the gradient consistency deviation between the new sample point and the adjacent sample point, perform the second derivative continuity calculation of the gradient field, measure the gradient offset magnitude, and obtain the gradient direction deviation evaluation sequence.
[0032] S413: Retrieve the discrete scalar values of the gradient direction deviation evaluation sequence, compare the difference in deviation offset within the preset gradient tolerance range, determine that the deviation offset is within the logical discrimination interval of the preset gradient tolerance range, perform logical association mapping of feature attribute vectors based on the gradient direction deviation evaluation sequence, perform spatiotemporal index encapsulation of the completed data points and perform tensor aggregation and fusion with the original feature field to generate logical association completed sample data.
[0033] As a further aspect of the present invention, the specific steps for obtaining the massive high-quality device dataset are as follows:
[0034] S511: Extract the global geometric topological features of the logical association complete sample data, perform manifold tangent space alignment operation and determine the global manifold constraint deviation, calculate the ratio of Euclidean distance to geodesic distance in manifold space, determine the sample point coordinate correction increment, perform multidimensional coordinate drift superposition operation on the logical association complete sample data, perform spatial coordinate position correction, and generate the corrected coordinate feature matrix.
[0035] S512: Retrieve the local neighborhood sample density distribution value of each node in the corrected coordinate feature matrix, perform spatial clustering center mapping and analyze the logical path connectivity of the nodes, compare the local neighborhood sample density distribution value of each node with the preset island determination threshold value, determine the nodes with missing connectivity and exclude logical island data, and delete to obtain the filtered distribution feature set.
[0036] S513: Adjust the spatial distribution of the dataset according to the filtered distribution feature set, perform data feature dimension alignment and aggregation to generate standardized data record frames, compare the total amount of the dataset with the set scale threshold to perform sample expansion and balancing processing, establish a multi-dimensional topological association mapping data structure, and generate a massive high-quality device dataset.
[0037] Compared with the prior art, the advantages and positive effects of the present invention are as follows:
[0038] In this invention, by measuring the distance between data feature nodes and the cosine correlation, a topological manifold structure of the feature dimension is constructed. Node coordinates and weight vector operations are extracted, feature terms are determined to be in the activation interval, a logical feature completion matrix is established, the gradient flow divergence feature of coordinate points is measured, the difference between the sample distribution and the expected distribution is compared, the location of the feature missing region is marked and the gradient deviation is calculated, a gradient flow-guided feature distribution field is established, interpolation is performed to construct and verify gradient consistency, global manifold constraints and coordinate corrections are performed, the spatial distribution of the dataset is adjusted and logical islands are eliminated, and massive high-quality datasets are automatically generated. Attached Figure Description
[0039] Figure 1 This is a flowchart of the main steps of the present invention;
[0040] Figure 2 This is a flowchart illustrating the process of obtaining the characteristic topological manifold structure of the present invention.
[0041] Figure 3 This is a flowchart of the semantic logic feature completion matrix acquisition process of the present invention;
[0042] Figure 4 This is a flowchart of the gradient flow-guided feature distribution field acquisition process of the present invention;
[0043] Figure 5 This is a flowchart illustrating the logical association and completion process for obtaining sample data in this invention.
[0044] Figure 6 This is a flowchart illustrating the process of acquiring massive, high-quality device datasets for this invention. Detailed Implementation
[0045] To make the objectives, technical solutions, and advantages of this invention clearer, the invention will be further described in detail below with reference to the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are merely illustrative and not intended to limit the invention.
[0046] In the description of this invention, it should be understood that the terms "length," "width," "upper," "lower," "front," "rear," "left," "right," "vertical," "horizontal," "top," "bottom," "inner," and "outer," etc., indicating orientation or positional relationships, are based on the orientation or positional relationships shown in the accompanying drawings and are only for the convenience of describing the invention and simplifying the description, and do not indicate or imply that the device or element referred to must have a specific orientation, or be constructed and operated in a specific orientation, and therefore should not be construed as a limitation of the invention. Furthermore, in the description of this invention, "a plurality of" means two or more, unless otherwise explicitly specified.
[0047] Please see Figure 1 A method for automatically generating massive, high-quality datasets based on feature recognition includes the following steps:
[0048] S1: Collect the original feature data points of the acquisition device, determine the Euclidean distance and cosine correlation of the feature nodes, determine the node association strength, construct the topology graph, and obtain the feature topology manifold structure;
[0049] S2: Based on the feature topological manifold structure, match the preset semantic hierarchical logic matrix, extract the manifold node coordinates and logic branch weight vectors, perform vector inner product operation, compare whether the inner product operation result is less than the set logic activation threshold, determine whether the feature term is in the activation interval, and obtain the semantic logic feature completion matrix.
[0050] S3: Complete the matrix for semantic logic features, extract sample space coordinate points, measure the gradient flow divergence features of each coordinate point, compare the sample distribution density with the global expected distribution, mark the location of feature missing areas with a density level higher than the set threshold, statistically analyze the gradient deviation of each dimension of the feature missing area, and establish a gradient flow-guided feature distribution field.
[0051] S4: Based on gradient flow-guided feature distribution field, interpolate to construct new sample points in the feature missing region, compare the gradient consistency deviation between the new sample points and adjacent sample points, determine that the deviation is within the preset gradient tolerance range, and generate logical associations to complete the sample data.
[0052] S5: Complete the sample data based on logical associations, perform global manifold constraints and sample point coordinate correction, adjust the spatial distribution of the dataset, eliminate logical island data, and output a massive high-quality device dataset.
[0053] The feature topological manifold structure includes local curvature, adjacency association table, manifold dimension, and geometric centroid coordinates; the semantic logical feature completion matrix includes logical topological paths, feature activation states, association probability mappings, and label weight indices; the gradient flow-guided feature distribution field includes gradient divergence features, potential energy distribution, vector flow skew angle, and density deviation topological graph; the logical association completion sample data includes completion feature vectors, logical cohesion coefficients, and spatial clustering biases; and the massive high-quality device dataset includes cleaned data samples, standardized feature matrices, and verification label metadata.
[0054] Please see Figure 2 The specific steps of S1 are as follows:
[0055] S111: Collect the original feature data points of the equipment, calculate the L2 norm of the spatial coordinate vector between the original feature data points of the equipment, and simultaneously determine the cosine correlation of the angle between the direction vectors between the original feature data points of the equipment. Based on the numerical mapping logic of the original feature data points of the equipment, Euclidean distance and cosine correlation, establish the node feature attribute association set and generate the feature node metric matrix.
[0056] Multi-source sensing arrays deployed in large industrial sites, such as triaxial accelerometers mounted on reciprocating compressor bearing housings, thin-film pressure sensors embedded in the main shaft, and high-precision infrared thermistor temperature measurement components, are used to collect raw characteristic data points of the equipment in real time, with frequencies ranging from 0.1Hz to 10kHz. The analog voltage signals generated by the sensors are converted into digital sequences by a data acquisition card with a sampling rate set to 25.6kHz. A data access operator with multi-partition fault tolerance is established in a distributed stream computing platform. A globally unified clock synchronization protocol (such as IEEE 1588) is used to perform nanosecond-level time-scale alignment on petabyte-level raw data streams from different physical locations, ensuring the consistency of spatial characteristics in a massive concurrent environment. For the massive data streams, denoising processing based on a sliding window mechanism is performed. A Chebyshev II digital filter is used to filter out system harmonics above 1500Hz, and a standardized operator is applied to uniformly map the raw physical quantities to a standard normal distribution range, forming a high-dimensional feature vector. The calculation involves determining the L2 norm of the spatial coordinate vector between the original feature data points of the calculation device, specifically by extracting the vector. and The Euclidean distance is determined by summing the squares of the components and taking the arithmetic square root of the numerical differences in various dimensions (such as vibration amplitude, instantaneous pressure, and operating temperature). For example, when a vector In the three-dimensional attribute space, it is represented as (0.85, 0.42, 0.15), a vector. When the expression is (0.88, 0.40, 0.18), its L2 norm value is calculated to be 0.047. The cosine correlation between the angles of the direction vectors between the original feature data points of the synchronous measurement equipment is obtained by performing a vector dot product operation and dividing the result by the product of the magnitudes of the two vectors. In the distributed computing framework Spark, the MLLib matrix operation library is called to perform mapping on hundreds of millions of node pairs using parallelized RDD operators, based on numerical mapping logic using Euclidean distance and cosine correlation. (in (Set to 2.0) Establish a node feature attribute association set and generate a feature node metric matrix that reflects the coupling strength of the device's state features throughout its entire life cycle.
[0057] S112: For the feature node metric matrix, retrieve the values of each dimension element in the feature node metric matrix, and perform a comparison operation between the element values and the preset association strength judgment threshold. Combine the node association strength to establish logical topological connection edges between each feature node and perform node connectivity construction operation according to the logical topological connection edges to obtain the device association topology map.
[0058] For the feature node metric matrix, a high-performance distributed memory file system is used to perform fast data retrieval and slice traversal. A Bloom filter mechanism is employed to quickly filter out a large number of sparse and redundant elements with weights below 0.1 in the matrix, thereby reducing the memory overhead of subsequent graph computation. The numerical values of elements in each dimension of the feature node metric matrix are retrieved. The value is then compared with a preset correlation strength threshold. The preset correlation strength threshold is set to 0.76, a value determined by statistically analyzing the correlation distribution of 100,000 historical samples of equipment failure precursors, selecting a lower limit with a confidence level of 98%. If the correlation value is greater than or equal to 0.76, a strong physical constraint or logical causal relationship is determined between the two nodes. Logical topological connections are established between each feature node based on the node association strength. The edge information and corresponding attribute weights are synchronously written into the distributed graph database, and a graph topology storage structure based on multi-level indexes is constructed. Node connectivity is built based on the logical topological connections. When the association value between the gearbox vibration feature node and the generator speed feature node in the metric matrix reaches 0.82, a bidirectional weighted edge is automatically established between them. In the context of petabyte-scale big data, a divide-and-conquer parallel computing strategy is adopted, dividing the entire node set into multiple logical computing subsets and distributing them to different nodes in the computing cluster. Finally, these subsets are integrated to generate an equipment association topology graph reflecting the complex operating mechanism and multivariate coupling structure of the equipment.
[0059] S113: Based on the device association topology graph, extract the geometric manifold distribution features of each node in the local neighborhood of the device association topology graph, perform Laplace eigenmap processing and transform the device association topology graph into a nonlinear manifold space coordinate system, calculate the Euclidean coordinate values of each node in the nonlinear manifold space coordinate system, and generate the feature topology manifold structure.
[0060] Based on the device association topology, a parallel extraction process of local neighborhood features is initiated using Message Passing Interface (MPI) technology in a distributed computing environment. For each node in the topology, its geometric manifold distribution features within its second-order physical neighborhood and third-order logical neighborhood are retrieved. The local curvature of the surface is characterized by calculating the second-order tensor of the local tangent space. Laplace eigenmap processing is then performed to construct the Laplace matrix. , where the diagonal matrix The elements are The summation of the row weights is calculated. For the massive sparse Laplacian matrix composed of a huge number of nodes, the Lanczos iterative algorithm in the distributed linear computation framework is invoked to perform eigenvalue decomposition in parallel within the computing cluster. The eigenvectors corresponding to the three smallest non-zero eigenvalues are selected, transforming the device association topology graph into a nonlinear manifold spatial coordinate system. The Euclidean coordinates of each node in the nonlinear manifold spatial coordinate system are then calculated. This allows the original high-dimensional nonlinear features to be projected onto a low-dimensional manifold surface. Based on the mapped spatial density distribution, the big data platform automatically builds a spatial index based on quadtrees or octrees to optimize the query efficiency of massive coordinate points and generate a feature topological manifold structure that can reveal the essential evolutionary laws of equipment operation.
[0061] Please see Figure 3 The specific steps of S2 are as follows:
[0062] S211: Obtain the preset semantic level logic matrix, perform node matching based on the feature topology manifold structure, extract the manifold node coordinates and the corresponding logical branch weight vectors of the semantic level logic matrix, perform semantic association mapping between manifold node coordinates and logical branch weight vectors, and generate a node logical feature distribution matrix.
[0063] A preset semantic hierarchy logic matrix is obtained, which is acquired by integrating an industrial IoT knowledge graph. This matrix stores five levels of semantic nesting logic, from top-level equipment assemblies to bottom-level sensor measurement points, containing 1024 logical classification items. Node matching is performed based on the feature topology manifold structure, aligning the three-dimensional coordinates in the manifold space with the logical IDs in the semantic hierarchy through hash mapping. The coordinates of manifold nodes within the feature topology manifold structure and the corresponding logical branch weight vectors of the semantic hierarchy logic matrix are extracted. The component values of the weight vectors are set according to expert system scores and historical data contributions; the weight vectors of the transmission chain branches are set to [0.65, 0.20, 0.15]. Semantic association mapping between manifold node coordinates and logical branch weight vectors is performed. Tensor product operations are executed in a distributed computing cluster using a tensor computing library (such as TensorFlow's cluster mode) to nonlinearly fuse spatial location information with semantic logic depth, generating a node logical feature distribution matrix, thus assigning a clear device function semantic label to each numerical feature point.
[0064] S212: Based on the node logical feature distribution matrix, retrieve vector elements and perform vector inner product operation to obtain the projected activation scalar value. Compare the projected activation scalar value with the set logical activation threshold. Determine if the projected activation scalar value is less than the set logical activation threshold to obtain the activation determination state value.
[0065] Based on the node logical feature distribution matrix, a high-performance inner product operator in the GPU array is invoked to perform pipelined vector inner product operations on massive vector elements. The node feature vector represented by each row is multiplied by a set global baseline activation vector to obtain a projected activation scalar value. This projected activation scalar value is compared with a set logical activation threshold, which was obtained through activation response experiments on 500 sets of missing sample data, and a value of 0.48 was selected. Parallel logical judgment branches are executed in the distributed processing unit to determine activation conditions where the projected activation scalar value is less than 0.48, thereby identifying regions with data breakpoints or feature gaps in the current semantic logical path. If the determination is true, an activation determination state value of 1 is output and stored in a distributed cache, enabling subsequent steps to achieve microsecond-level fast logical responses when processing petabyte-scale data, obtaining activation determination state values reflecting the distribution of missing features.
[0066] S213: For the activation determination state value, determine that the feature term is in the activation interval, retrieve the neighborhood distribution of the feature term in the feature topology manifold structure, perform missing term prediction based on the projected activation scalar value and the original attribute data of the feature node, determine the node feature completion increment, and generate the feature increment mapping vector.
[0067] For activation determination states, if a feature term is in the activation interval, its distribution within a spherical neighborhood of 5 units within the feature topological manifold is retrieved, locking in 24 surrounding known sample points. Missing term prediction is performed based on the projected activation scalar value and the original attribute data of the feature nodes. A distributed Gaussian process regression model is employed, utilizing a computing cluster to perform parallel inversion operations on the local covariance matrix to predict the expected value of the attribute at the missing location. Based on the deviation between the prediction result and the original mean, the node feature completion increment is determined, with the value strictly limited to within three standard deviations of historical normal fluctuations (e.g., between 0.05 and 0.35). The big data platform monitors the prediction residuals in real time, optimizing completion accuracy by dynamically adjusting the hyperparameters of the regression model (e.g., setting the length scale parameter to 1.2), generating a feature increment mapping vector to ensure logical consistency of the completed information.
[0068] S214: Based on the feature increment mapping vector, perform dimension merging in conjunction with the node logical feature distribution matrix, perform superposition of the feature increment mapping vector and the original attribute data of the feature nodes, establish node association mapping relationship, and obtain the semantic logical feature completion matrix;
[0069] Based on the feature increment mapping vector, dimension merging is performed in conjunction with the node logical feature distribution matrix. During Spark's operator execution phase, the Data Frame Join operation is used to associate the completed vector with the original feature tensor, and the vector superposition operator is called to perform spatial fusion of the feature increment mapping vector and the original attribute data of the feature nodes. Node association mapping relationships are established, and the completed data points are inserted into the corresponding manifold coordinate indices. For petabyte-scale large-scale datasets, Parquet, a schema-based columnar storage format, is used to persist the completed matrix, and Snappy compression is enabled to optimize storage utilization, resulting in a semantic logical feature completion matrix.
[0070] Please see Figure 4 The specific steps of S3 are as follows:
[0071] S311: Complete the matrix for semantic logic features, obtain the coordinate points of the sample space, retrieve the distribution vector of the coordinate points of the sample space in Euclidean space, determine the gradient flow divergence features of each coordinate point and perform partial derivative operations, and perform feature vector association and coordinate index encapsulation based on the gradient flow divergence features of each coordinate point to generate a coordinate gradient feature mapping table.
[0072] To complete the semantic logical feature matrix, sample space coordinates are obtained, and the 3D distribution vector of these coordinates in Euclidean space is quickly retrieved from the index nodes. For each coordinate point, its partial derivatives with respect to spatial location are calculated for its three-dimensional components. The gradient flow divergence characteristics of each coordinate point are determined, and partial derivative calculations are performed. Utilizing the cluster's parallel floating-point computing power, a gradient field is constructed on a massive number of discrete points using the finite difference method, and the divergence value of each point is calculated to characterize the diffusion degree of the feature flow in that region. Based on the gradient flow divergence characteristics of each coordinate point, feature vectors are associated and encapsulated with coordinate indices. A coordinate gradient feature mapping table based on spatial hashing is constructed at the big data index layer, supporting multi-dimensional range queries and gradient-related retrieval of petabyte-scale feature data.
[0073] S312: Based on the coordinate gradient feature mapping table, extract the sample distribution attributes within the local window, statistically analyze the local density values of the samples, and compare the differences between the local density values and the global expected distribution using the following formula:
[0074] ;
[0075] The density judgment deviation value is obtained through calculation. This deviation value is then compared with a set density level threshold value. The locations of feature-missing regions where the density judgment deviation value is lower than the set density level threshold are marked, resulting in a set of coordinates for the feature-missing blocks. This represents the deviation value for determining density. Represents the number of samples in a local area. Represents the volume of a local area. Represents the global expected distribution density. Represents the amplitude of the gradient flow divergence characteristic. Represents a unit spatial reference scale. Represents the mass constant of the benchmark sample;
[0076] Based on the coordinate gradient feature mapping table, the sample distribution attributes within the local window are extracted, and the local probe sphere radius is set to 12. The density values of the local sample regions are statistically analyzed, and the differences between the local sample density values and the global expected distribution are compared. The formula used is:
[0077]
[0078] The density judgment deviation value is obtained through calculation. In the distributed aggregation operator, the parameter values are set as follows: local region spatial volume. Global expected distribution density Unit spatial reference scale benchmark sample mass constant Suppose the number of samples in a certain missing region is... Its gradient flow divergence characteristic amplitude .
[0079] Calculation steps: First, calculate the square root term. Since the expected value is 75 and the absolute value of the deviation is 15, the result of the square root term is 0.447; then multiply by the gradient correction term. ,get .
[0080] The density judgment deviation value was compared with the set density level threshold of 0.1. Since 0.107 is higher than 0.1, this location was marked as a feature missing region. The judgment results generated by performing full batch processing on petabyte-level data are shown in Table 1.
[0081] Table 1. Distribution of Feature Region Density Determination
[0082] Regional Index Sample size Expected sample density divergence characteristic amplitude Determine the deviation value status flags REG-001 60 0.50 2.4 0.107 Feature missing REG-002 92 0.50 1.1 0.052 Normal area REG-003 125 0.50 0.6 0.048 Normal area REG-004 45 0.50 3.2 0.202 Feature missing
[0083] As shown in Table 1, this formula effectively identifies "pseudo-holes" caused by acquisition failure by introducing gradient flow divergence features as density correction terms, and obtains the coordinate set of feature missing blocks.
[0084] S313: Based on the coordinate set of the feature missing block, retrieve the corresponding gradient component of the covered area using the coordinate gradient feature mapping table, calculate the gradient deviation of each dimension of the feature missing area, perform potential energy distribution mapping in the manifold space for the gradient deviation of each dimension of the feature missing area, and establish a gradient flow-guided feature distribution field.
[0085] Potential energy distribution mapping is performed on the manifold space for gradient deviations in various dimensions of the feature-missing region. Using a distributed finite element analysis operator, the gradient deviations in the feature-missing region are transformed into boundary conditions for the Poisson equation, and the potential energy distribution field on the manifold surface is solved in parallel. A gradient flow guides the feature distribution field, generating directional vector field indices in the computation cluster. This provides physically meaningful guiding paths for subsequent interpolation of new sample points, ensuring that the data completion process conforms to the geometric constraints of the manifold.
[0086] Please see Figure 5 The specific steps of S4 are as follows:
[0087] S411: Extract the extreme values of the gradient gradient and spatial evolution trend of the distribution potential energy of the gradient flow-guided feature distribution field, retrieve the location of the feature missing region and obtain the local manifold curvature parameters of the feature missing region, calculate the mean density of known sample points within the local window, perform nonlinear surface interpolation operation and interpolate to construct new sample points in the feature missing region, calculate the spatial dimension components of the new node and perform coordinate transformation, and establish the spatial coordinate matrix of the interpolated sample.
[0088] The algorithm extracts the extreme values and spatial evolution trends of the potential energy gradient in the feature distribution field guided by gradient flow, and quickly locates the equilibrium position with the lowest potential energy gradient using a multi-level parallel search operator. It retrieves the locations of feature-missing regions and obtains the local manifold curvature parameters at these locations. The algorithm statistically analyzes the mean density of known sample points within a local window and performs nonlinear surface interpolation. A distributed radial basis function interpolation algorithm is employed, constructing a system of linear equations within the Spark operator and solving for millions of interpolation coefficients using the distributed conjugate gradient (PCG) method. New sample points are constructed through interpolation in the feature-missing regions, and the spatial dimensionality components of the new nodes are calculated and coordinate transformation is performed. The big data platform performs dynamic load balancing based on the coordinate distribution of the interpolation points, ensuring that the interpolation tasks are evenly distributed across the computing nodes, and establishing the spatial coordinate matrix of the interpolated samples.
[0089] S412: Obtain the local gradient vectors and tangent plane normal directions of each node in the interpolation sample space coordinate matrix, compare the gradient consistency deviation between the new sample point and the adjacent sample point, perform the second derivative continuity calculation of the gradient field, measure the gradient offset magnitude, and obtain the gradient direction deviation evaluation sequence.
[0090] The local gradient vectors and tangent plane normal directions of each node in the interpolation sample space coordinate matrix are obtained. Parallel gradient computation is optimized using GPU texture memory. The difference in the second-order mixed partial derivatives between each new interpolation point and the original sample set in the tangent space is determined using a CUDA kernel function. The gradient offset magnitude is measured, and a continuity tolerance of 0.12 is set. A gradient direction deviation evaluation sequence is obtained through computation. The big data system performs real-time sliding window mean drift detection on this sequence. Once the deviation exceeds 0.12, it is identified as an interpolation artifact and marked for removal, resulting in a gradient direction deviation evaluation sequence reflecting the smoothness of the data.
[0091] S413: Retrieve discrete scalar values of gradient direction deviation evaluation sequence, compare the difference in deviation offset within the preset gradient tolerance range, determine that the deviation offset is within the preset gradient tolerance range logical discrimination interval, perform logical association mapping of feature attribute vectors based on gradient direction deviation evaluation sequence, perform spatiotemporal index encapsulation of data points to complete data points and perform tensor aggregation and fusion with the original feature field to generate logical association complete sample data.
[0092] The discrete scalar values of the gradient direction deviation evaluation sequence are retrieved, and the difference in deviation offset within the preset gradient tolerance range is compared. The deviation offset is determined to be within the logical discrimination interval [-0.05, 0.05]. Logical association mapping of feature attribute vectors is performed based on the gradient direction deviation evaluation sequence. Temporal, spatial, and semantic three-dimensional indexing and encapsulation of the completed data points are performed, and the completed results are fused with the original feature field in a distributed tensor computing engine using tensor aggregation. For the generated completed data, a full consistency audit is performed in the big data lake to ensure that the massive samples after completion highly fit the original physical features in the global topological distribution, generating logically associated completed sample data.
[0093] Please see Figure 6 The specific steps of S5 are as follows:
[0094] S511: Extract global geometric topological features of logical association completion sample data, perform manifold tangent space alignment operation and determine global manifold constraint deviation, calculate the ratio of Euclidean distance to geodesic distance in manifold space, determine the sample point coordinate correction increment, perform multidimensional coordinate drift superposition operation on logical association completion sample data, perform spatial coordinate position correction, and generate the corrected coordinate feature matrix.
[0095] Logical associations are extracted to complete the global geometric topology features of the sample data. In the parallel processing of large datasets, a distributed genetic algorithm is used to optimize the coordinate correction coefficients, ensuring that the rate of change of the corrected manifold curvature is minimized. Multidimensional coordinate drift superposition operations are performed on the logical association-completed sample data to correct the spatial coordinate positions. The cluster scheduler performs parallel coordinate transformation operations on hundreds of millions of records, controlling the correction amount to between 0.1% and 2% of the original magnitude, generating a corrected coordinate feature matrix.
[0096] S512: Retrieve the local neighborhood sample density distribution value of each node in the corrected coordinate feature matrix, perform spatial clustering center mapping and analyze the logical path connectivity of the nodes, compare the local neighborhood sample density distribution value of each node with the preset island determination threshold value, determine the nodes with missing connectivity and exclude logical island data, and delete to obtain the filtered distribution feature set.
[0097] The algorithm retrieves the local neighborhood sample density distribution values of each node within the corrected coordinate feature matrix, performs spatial clustering center mapping, and analyzes the logical path connectivity of the nodes. It then uses a distributed connected component algorithm to detect isolated subgraphs in the dataset, comparing the density values of each node with a preset isolated subgraph threshold of 0.04. Nodes with missing connectivity are identified, and logically isolated data is excluded. Physical cleanup based on reference counting is performed in the distributed storage layer to permanently remove noisy and outlier points from the large dataset, resulting in a filtered distribution feature set.
[0098] S513: Adjust the spatial distribution of the dataset based on the filtered distribution feature set, perform data feature dimension alignment and aggregation to generate standardized data record frames, compare the total amount of the dataset with the set scale threshold to perform sample expansion and balancing processing, establish a multi-dimensional topological association mapping data structure, and generate a massive high-quality device dataset.
[0099] The spatial distribution of the dataset is adjusted based on the filtered distribution feature set, and sample augmentation and balancing are performed by comparing the total dataset size with a set size threshold (500 million records). If the data volume is insufficient, the Wasserstein Generative Adversarial Network (WGAN) deployed on the cluster is invoked to generate synthetic samples conforming to the original probability distribution in parallel within the manifold space. This establishes a multidimensional topological association mapping data structure. Experimental data comparisons are shown in Table 2.
[0100] Table 2 Comparison of Equipment Dataset Quality Optimization Results
[0101] Data evaluation metrics Before processing (raw data collection) After processing (this solution) Optimization range Sample space coverage 62.4% 98.2% 57.3% Data distribution information entropy 4.82 7.95 64.9% Sparsity of the correlation matrix 0.85 0.12 -85.8% Fault warning response delay 1250ms 450ms 64.0%
[0102] As shown in Table 2, big data experiments demonstrate that in TB / PB-level industrial data environments, this solution, through multi-stage manifold completion and distributed optimization, greatly supports the predictive maintenance decisions of industrial internet platforms and generates massive, high-quality equipment datasets that meet the requirements of high-precision modeling.
[0103] The above are merely preferred embodiments of the present invention and are not intended to limit the present invention in any other way. Any person skilled in the art may make changes or modifications to the above-disclosed technical content to create equivalent embodiments that can be applied to other fields. However, any simple modifications, equivalent changes, and modifications made to the above embodiments based on the technical essence of the present invention without departing from the scope of the present invention shall still fall within the protection scope of the present invention.
Claims
1. A method for automatically generating massive, high-quality datasets based on feature recognition, characterized in that, Includes the following steps: S1: Collect the original feature data points of the acquisition device, determine the Euclidean distance and cosine correlation of the feature nodes, determine the node association strength, construct the topology graph, and obtain the feature topology manifold structure; S2: Based on the feature topological manifold structure, match the preset semantic hierarchical logic matrix, extract the manifold node coordinates and logic branch weight vectors, perform vector inner product operation and determine that the feature terms are in the activation interval, and obtain the semantic logic feature completion matrix. S3: Complete the semantic logic feature matrix, extract sample space coordinate points, measure the gradient flow divergence features of each coordinate point, compare the sample distribution density with the global expected distribution, mark the location of feature missing areas with a density level higher than the set threshold, statistically analyze the gradient deviation of each dimension of the feature missing area, and establish a gradient flow-guided feature distribution field. S4: Based on the gradient flow-guided feature distribution field, interpolate to construct new sample points in the feature missing region, compare the gradient consistency deviation between the new sample points and adjacent sample points, determine that the deviation is within the preset gradient tolerance range, and generate logical associations to complete the sample data. S5: Complete the sample data according to the logical association, perform global manifold constraints and sample point coordinate correction, adjust the spatial distribution of the dataset, eliminate logical island data, and output a massive high-quality device dataset.
2. The method for automatically generating massive high-quality datasets based on feature recognition according to claim 1, characterized in that: The term "feature in activation interval" specifically means that the result of the inner product operation is less than a set logical activation threshold.
3. The method for automatically generating massive, high-quality datasets based on feature recognition according to claim 1, characterized in that: The feature topological manifold structure includes local curvature, adjacency association table, manifold dimension, and geometric centroid coordinates; the semantic logical feature completion matrix includes logical topological path, feature activation state, association probability mapping, and label weight index; the gradient flow-guided feature distribution field includes gradient divergence features, potential energy distribution, vector flow skew angle, and density deviation topological graph; the logical association completion sample data includes completion feature vector, logical cohesion coefficient, and spatial clustering bias; the massive high-quality device dataset includes cleaned data samples, standardized feature matrix, and verification label metadata.
4. The method for automatically generating massive, high-quality datasets based on feature recognition according to claim 1, characterized in that, The specific steps for obtaining the characteristic topological manifold structure are as follows: S111: Collect the original feature data points of the device, calculate the L2 norm of the spatial coordinate vector between the original feature data points of the device, and simultaneously determine the cosine correlation of the angle between the direction vectors between the original feature data points of the device. Based on the numerical mapping logic of the original feature data points of the device, the Euclidean distance and the cosine correlation, establish a node feature attribute association set and generate a feature node metric matrix. S112: Retrieve the element values of each dimension in the feature node metric matrix, and perform a comparison operation between the element values and the preset association strength determination threshold. Combine the node association strength to establish logical topological connection edges between each feature node and perform node connectivity construction operation according to the logical topological connection edges to obtain the device association topology map. S113: Extract the geometric manifold distribution features within the local neighborhood of each node in the device association topology graph, perform Laplace eigenmap processing, and transform the device association topology graph into a nonlinear manifold space coordinate system. Calculate the Euclidean coordinate values of each node in the nonlinear manifold space coordinate system. Generate characteristic topological manifold structures.
5. The method for automatically generating massive, high-quality datasets based on feature recognition according to claim 1, characterized in that, The specific steps for obtaining the semantic logical feature completion matrix are as follows: S211: Obtain a preset semantic hierarchy logic matrix, perform node matching based on the feature topology manifold structure, extract the manifold node coordinates within the feature topology manifold structure and the corresponding logical branch weight vector of the semantic hierarchy logic matrix, perform semantic association mapping between the manifold node coordinates and the logical branch weight vector, and generate a node logical feature distribution matrix. S212: Based on the node logical feature distribution matrix, retrieve vector elements and perform vector inner product operation to obtain a projected activation scalar value. Compare the projected activation scalar value with a set logical activation threshold and determine that the projected activation scalar value is less than the set logical activation threshold activation condition to obtain an activation determination state value. S213: For the activation determination state value, the feature item is determined to be in the activation interval. The feature item is retrieved and its neighborhood distribution is retrieved within the feature topology manifold structure. Based on the projected activation scalar value and the original attribute data of the feature node, missing item prediction is performed and node feature completion increment is determined to generate a feature increment mapping vector. S214: Based on the feature increment mapping vector, perform dimension merging in conjunction with the node logical feature distribution matrix, and superimpose the feature increment mapping vector with the original attribute data of the feature node to establish a node association mapping relationship and obtain a semantic logical feature completion matrix.
6. The method for automatically generating massive, high-quality datasets based on feature recognition according to claim 1, characterized in that, The specific steps for obtaining the gradient flow-guided feature distribution field are as follows: S311: Complete the semantic logic feature matrix, obtain the sample space coordinate points, retrieve the distribution vector of the sample space coordinate points in Euclidean space, determine the gradient flow divergence feature of each coordinate point and perform partial derivative operation, and perform feature vector and coordinate index association encapsulation based on the gradient flow divergence feature of each coordinate point to generate a coordinate gradient feature mapping table. S312: Based on the coordinate gradient feature mapping table, extract the sample distribution attributes within the local window range, count the density value of the sample local region, compare the difference between the sample local region density value and the global expected distribution, calculate the density judgment deviation value, compare the density judgment deviation value with the set density level threshold execution value, mark the location of the feature missing area where the density judgment deviation value is higher than the set density level threshold, and obtain the feature missing block coordinate set. S313: Based on the coordinate set of the missing feature blocks, retrieve the corresponding gradient components of the covered area using the coordinate gradient feature mapping table, calculate the gradient deviation of each dimension of the missing feature area, perform potential energy distribution mapping on the manifold space for the gradient deviation of each dimension of the missing feature area, and establish a gradient flow-guided feature distribution field.
7. The method for automatically generating massive, high-quality datasets based on feature recognition according to claim 6, characterized in that, The formula for calculating the density determination deviation value is as follows: ; in, This represents the deviation value for determining density. Represents the number of samples in a local area. Represents the volume of a local area. Represents the global expected distribution density. Represents the amplitude of the gradient flow divergence characteristic. Represents a unit spatial reference scale. This represents the quality constant of the benchmark sample.
8. The method for automatically generating massive high-quality datasets based on feature recognition according to claim 1, characterized in that, The specific steps for obtaining the logical association-completed sample data are as follows: S411: Extract the extreme values of the gradient gradient and spatial evolution trend of the distribution potential energy of the gradient flow-guided feature distribution field, retrieve the location of the feature missing region and obtain the local manifold curvature parameters of the feature missing region, calculate the mean density of known sample points within the local window, perform nonlinear surface interpolation operation and interpolate to construct new sample points in the feature missing region, calculate the spatial dimension components of the new node and perform coordinate transformation, and establish the interpolated sample space coordinate matrix. S412: Obtain the local gradient vector and tangent plane normal direction of each node in the interpolation sample space coordinate matrix, compare the gradient consistency deviation between the new sample point and the adjacent sample point, perform the second derivative continuity calculation of the gradient field, measure the gradient offset magnitude, and obtain the gradient direction deviation evaluation sequence. S413: Retrieve the discrete scalar values of the gradient direction deviation evaluation sequence, compare the difference in deviation offset within the preset gradient tolerance range, determine that the deviation offset is within the logical discrimination interval of the preset gradient tolerance range, perform logical association mapping of feature attribute vectors based on the gradient direction deviation evaluation sequence, perform spatiotemporal index encapsulation of the completed data points and perform tensor aggregation and fusion with the original feature field to generate logical association completed sample data.
9. The method for automatically generating massive high-quality datasets based on feature recognition according to claim 1, characterized in that, The specific steps for obtaining the massive, high-quality device dataset are as follows: S511: Extract the global geometric topological features of the logical association complete sample data, perform manifold tangent space alignment operation and determine the global manifold constraint deviation, calculate the ratio of Euclidean distance to geodesic distance in manifold space, determine the sample point coordinate correction increment, perform multidimensional coordinate drift superposition operation on the logical association complete sample data, perform spatial coordinate position correction, and generate the corrected coordinate feature matrix. S512: Retrieve the local neighborhood sample density distribution value of each node in the corrected coordinate feature matrix, perform spatial clustering center mapping and analyze the logical path connectivity of the nodes, compare the local neighborhood sample density distribution value of each node with the preset island determination threshold value, determine the nodes with missing connectivity and exclude logical island data, and delete to obtain the filtered distribution feature set. S513: Adjust the spatial distribution of the dataset according to the filtered distribution feature set, perform data feature dimension alignment and aggregation to generate standardized data record frames, compare the total amount of the dataset with the set scale threshold to perform sample expansion and balancing processing, establish a multi-dimensional topological association mapping data structure, and generate a massive high-quality device dataset.
