Industry model training data optimization method based on sample distribution analysis

By constructing a sample distribution distance matrix to divide regions, extracting boundary and center sample vectors for sample adjustment, and generating an optimized training dataset, the problem of unbalanced distribution of training data for industry models is solved, thereby improving the model's recognition and generalization capabilities.

CN123115835APending Publication Date: 2026-11-17SHANGHAI QINGFENG INTELLIGENT DIGITAL TECHNOLOGY CO LTD +1
View PDF 0 Cites 0 Cited by

Patent Information

Application Number
CN202611302169.1
Authority / Receiving Office
CN · China
Patent Type
Applications(China)
Current Assignee / Owner
Filing Date
2026-08-26
Publication Date
2026-11-17

AI Technical Summary

Technical Problem

In existing technologies, the uneven distribution of training data for industry models leads to the easy deletion of boundary samples in dense areas and insufficient representativeness of central samples in sparse areas, affecting the model's recognition ability and stability.

Method used

By constructing a sample distribution distance matrix, dividing dense and sparse sub-regions, extracting boundary sample vectors and center sample vectors, performing edge shrinkage sampling and center expansion sampling, generating an optimized training dataset, and reducing the impact of sample distribution bias on model training.

Benefits of technology

It achieves the goal of alleviating sample clustering and scarcity problems without disrupting the original sample distribution structure, improving the model's ability to identify boundary regions and cover the central distribution features of sparse regions, and enhancing the model's fitting ability and generalization effect.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure CN123115835A_ABST
    Figure CN123115835A_ABST
Patent Text Reader

Abstract

This invention discloses a method for optimizing industry model training data based on sample distribution analysis, belonging to the field of machine learning data processing technology. The method includes: acquiring the original industry training dataset and extracting multivariate features of each sample to generate sample feature vectors; calculating the spatial distribution distance between any two samples based on the positional relationship of the sample feature vectors in the feature space, and constructing a sample distribution distance matrix; dividing the feature space into regions based on the sample distribution distance matrix to obtain multiple candidate sample sub-regions and calculating sample density values; selecting candidate sample sub-regions with sample density values ​​greater than a preset density threshold as dense sub-regions and candidate sample sub-regions with sample density values ​​less than the preset density threshold as sparse sub-regions; extracting boundary sample vectors from dense sub-regions and extracting center sample vectors from sparse sub-regions, and combining them to generate sample adjustment vectors.
Need to check novelty before this filing date? Find Prior Art

Description

Technical Field

[0001] This invention relates to the field of machine learning data processing technology, specifically to a method for optimizing training data for industry models based on sample distribution analysis. Background Technology

[0002] Industry model training data typically comes from various business systems, including numerical and categorical fields. After transformation, the sample features exhibit an uneven distribution in a high-dimensional feature space, with some areas showing highly clustered samples and others having sparse samples. The quality of the industry model training data directly affects the model's ability to fit industry business features and its generalization effect. When constructing an industry model, it is necessary to collect samples from the original business data to form a training dataset. Uneven distribution leads to insufficient learning of boundary features in dense areas and inadequate coverage of central distribution features in sparse areas. Therefore, it is necessary to optimize the distribution of the training data before model training.

[0003] Existing technologies for handling imbalanced training data mainly include random undersampling, random oversampling, synthetic minority class oversampling, and cluster-based data sampling. Random undersampling balances the sample count by deleting some samples from dense regions, but it doesn't consider the sample distribution in the feature space, easily discarding samples at the edges of dense regions, causing loss of class boundary information and reducing the trained model's ability to recognize boundary regions. Random oversampling and synthetic minority class oversampling increase the number of samples in sparse regions through duplication or interpolation, but duplicating samples introduces a large amount of repetitive information, and interpolated samples may deviate from the true center distribution of sparse regions, generating noisy samples and affecting the stability of model training. While cluster-based data sampling methods can select samples according to cluster structure, the clustering results are greatly affected by parameter settings, and they usually only use a single distance to measure sample similarity, making it difficult to simultaneously reflect the spatial and directional differences between mixed numerical and categorical features.

[0004] In the process of optimizing industry model training data, how to identify dense and sparse sub-regions based on the distribution density of samples in the feature space, and how to selectively adjust samples at the boundaries of dense sub-regions and the centers of sparse sub-regions to avoid the loss of boundary information and the introduction of noise caused by random sampling, are problems that current technologies have not yet effectively addressed. Furthermore, how to utilize the gradient of dense region boundaries and the expansion direction of sparse region centers to perform targeted compression and supplementation of the original training data during sample distribution adjustment also requires further improvement. Summary of the Invention

[0005] To address the problems of uneven distribution of industry training data, accidental deletion of boundary samples in dense regions, and insufficient representativeness of center samples in sparse regions in existing technologies, the present invention aims to provide an industry model training data optimization method based on sample distribution analysis. This method constructs a sample distribution distance matrix by using the spatial distribution distance between sample feature vectors, divides dense and sparse sub-regions based on local density, extracts boundary sample vectors of dense sub-regions and center sample vectors of sparse sub-regions, and performs edge shrinkage sampling and center expansion sampling on the original industry training dataset according to the sample adjustment vectors to generate an optimized training dataset. This makes the distribution of training data in the feature space more consistent with the training needs of industry models.

[0006] To achieve the above objectives, the present invention provides the following technical solution: The present invention provides a method for optimizing industry model training data based on sample distribution analysis, comprising: acquiring an original industry training dataset; extracting multivariate features of each sample in the original industry training dataset to generate a sample feature vector; calculating the spatial distribution distance between any two samples based on the positional relationship of all the sample feature vectors in the feature space to construct a sample distribution distance matrix; dividing the feature space into regions based on the sample distribution distance matrix to obtain multiple candidate sample sub-regions, and calculating the sample density value in each candidate sample sub-region; selecting candidate sample sub-regions with sample density values ​​greater than a preset density threshold as dense sub-regions, and selecting candidate sample sub-regions with sample density values ​​less than a preset density threshold as sparse sub-regions; extracting boundary sample vectors from each dense sub-region and extracting center sample vectors from each sparse sub-region, and combining the boundary sample vectors and the center sample vectors to generate a sample adjustment vector; adjusting the sample distribution of the original industry training dataset according to the sample adjustment vector to generate an optimized training dataset. This method uses the distribution distance of samples in the feature space to identify regional density, and guides sample adjustment by using boundary samples and center samples to balance the training data of industry models between dense and sparse regions, thereby reducing the impact of sample distribution bias on model training.

[0007] Furthermore, when extracting multivariate features for each sample, field parsing is performed on each sample to identify numerical and categorical fields. Numerical fields are normalized, and categorical fields are encoded and mapped to generate an initial feature array with a unified format. Principal component analysis is then performed on the initial feature array to extract principal component features with a cumulative variance contribution rate greater than a preset contribution rate threshold. These principal component features are then combined to form the sample feature vector. This process unifies the representation of heterogeneous fields, reduces feature redundancy, and improves the stability and computational efficiency of subsequent distance calculations.

[0008] As a preferred technical solution, when calculating the spatial distribution distance between any two samples, the difference in each dimension between the corresponding feature vectors of the two samples is obtained. The squares of each difference are summed and the square root is taken to obtain a first distance value. The cosine similarity between the corresponding feature vectors of the two samples is obtained, and the difference between the first and cosine similarity is determined as a second distance value. The first and second distance values ​​are then weighted and summed according to preset first and second weighting coefficients to obtain the spatial distribution distance. The spatial distribution distance matrix is ​​constructed from the spatial distribution distances between all samples. This distance metric considers both feature amplitude and directional differences, and can more accurately reflect the distribution relationship of industry samples in the feature space.

[0009] Furthermore, when dividing the region based on the sample distribution distance matrix, the minimum spatial distribution distance corresponding to each sample is extracted from the sample distribution distance matrix, and the initial neighborhood radius is determined based on the minimum spatial distribution distance. An initial neighborhood sample set is determined with each sample as the center point and the initial neighborhood radius as the search radius. The local density corresponding to each sample is determined based on the number of samples in the initial neighborhood sample set, and all samples are arranged in descending order of local density to obtain a density sequence. A segmentation point is determined based on the local density difference between adjacent samples in the density sequence, and the feature space is divided into regions using the segmentation point as the boundary, resulting in multiple candidate sample sub-regions. Adaptively identifying the distribution structure in the feature space based on local density changes can avoid the distortion of region boundaries caused by a fixed division method.

[0010] Furthermore, when selecting dense and sparse sub-regions, the average and standard deviation of the sample density values ​​of all candidate sample sub-regions are calculated, and the sum of the average and standard deviation is determined as a preset density threshold. Candidate sample sub-regions with sample density values ​​greater than the preset density threshold are marked as dense sub-regions, those with sample density values ​​less than the preset density threshold are marked as sparse sub-regions, and those with sample density values ​​equal to the preset density threshold are marked as transitional sub-regions. Dense sub-regions containing fewer than a preset lower limit of sample number are removed from all dense sub-regions, and sparse sub-regions containing more than a preset upper limit of sample number are removed from all sparse sub-regions. Dynamically determining the density threshold and removing noisy dense regions and pseudo-sparse regions helps improve the accuracy of subsequent sample adjustment.

[0011] Furthermore, when generating the sample adjustment vector, for each dense sub-region, the first predetermined number of samples closest to the boundary of the dense sub-region are obtained as boundary samples, and the sample feature vector corresponding to each boundary sample is extracted as the boundary sample vector; for each sparse sub-region, the mean vector of the sample feature vectors of all samples in the sparse sub-region is obtained as the center sample vector; the boundary sample vector and the center sample vector are weighted and concatenated according to the type of their respective sub-regions to generate the sample adjustment vector. Using boundary samples and center samples as representatives, the boundary shape of dense regions and the central trend of sparse regions can be described more accurately.

[0012] Furthermore, when adjusting the sample distribution of the original industry training dataset based on the sample adjustment vector, the boundary gradient direction of each dense sub-region is determined according to the boundary sample vector in the sample adjustment vector. Edge shrinkage sampling is performed on the samples at the boundary of the dense sub-region along the boundary gradient direction to obtain a compressed sample set. The center expansion direction of each sparse sub-region is determined according to the center sample vector in the sample adjustment vector. Interpolation filling sampling is performed in the sparse sub-region along the center expansion direction to obtain a supplementary sample set. The compressed sample set and the supplementary sample set are merged with the unadjusted samples in the original industry training dataset to generate an optimized training dataset. By combining dense region shrinkage and sparse region supplementation, the problems of sample clustering and sample scarcity can be alleviated without destroying the overall structure of the original sample distribution.

[0013] In a preferred embodiment, edge shrinkage sampling includes: obtaining the average value of all boundary sample vectors in the dense sub-region to obtain the boundary center vector; obtaining the region center vector of the dense sub-region; determining the direction from the boundary center vector to the region center vector as the boundary gradient direction; shrinking inward from the boundary of the dense sub-region along the boundary gradient direction by a predetermined step; selecting a second predetermined number of sampling points on the shrunken boundary; determining the feature vectors corresponding to the sampling points as compressed samples; marking samples located outside the shrunken boundary within the dense sub-region as samples to be removed; removing the samples to be removed from the original industry training dataset; and adding the compressed samples to the original industry training dataset. This process can stably compress the boundary range of the dense region and reduce overfitting caused by redundant samples near the boundary during model training.

[0014] In another preferred embodiment, the interpolation filling sampling includes: obtaining the covariance matrix of the sample feature vectors of all samples in the sparse sub-region; performing eigenvalue decomposition on the covariance matrix to obtain multiple eigenvalues ​​and corresponding eigenvectors; determining the direction of the eigenvector corresponding to the largest eigenvalue as the main expansion direction; selecting a third predetermined number of interpolation positions along the main expansion direction, starting from the center sample vector; generating a candidate interpolation sample at each interpolation position; and determining the candidate interpolation samples whose minimum spatial distribution distance from the existing samples in the sparse sub-region is greater than a predetermined distance threshold as supplementary samples and adding them to the supplementary sample set. Interpolation based on the main expansion direction ensures that the newly added samples conform to the main distribution trend of the sparse region, avoiding the generation of noisy samples that deviate from the actual sample structure.

[0015] As a preferred technical solution, the boundary gradient direction is corrected based on the principal component directions of the local covariance matrix of samples within the dense sub-region. The corrected edge contraction direction is obtained by weighting the boundary gradient direction with the principal component directions. The corrected contraction direction better adapts to the sample distribution pattern within the dense sub-region, reducing the damage to the regional structure during the contraction process.

[0016] The technical effects and advantages provided by the present invention in the above technical solution are as follows: Positional differences are obtained by calculating the Euclidean distance between any two sample feature vectors, and directional differences are obtained by calculating the difference between 1 and cosine similarity. A weighted sum of positional and directional differences is then performed. This sample distribution distance simultaneously reflects the differences in distance and feature direction in the numerical feature space. During the weighted summation, preset first and second weighting coefficients are used to adjust the influence of positional and directional differences on the spatial distribution distance, ensuring the distance metric is adapted to the industry data feature structure. An initial neighborhood radius is determined based on the minimum spatial distribution distance of each sample. The number of neighboring samples is counted to obtain the local density. Candidate sample sub-regions are then divided according to local density sorting and the difference between adjacent local densities. The sum of the average and standard deviation of the density values ​​of all candidate sample sub-regions is used as a preset density threshold to identify dense and sparse sub-regions. This region division method uses the sample's own neighborhood density and distribution dispersion to form the threshold, adapting to the actual distribution patterns of training data in different industries. This reduces the possibility of incorrectly classifying dense and sparse regions under a fixed threshold, allowing subsequent boundary and center sample extraction to be performed within more accurate regions.

[0017] For a dense sub-region, the first predetermined number of samples closest to the region boundary are extracted as boundary sample vectors. The boundary sample vectors reflect the distribution information at the boundary between the dense sub-region and the adjacent region. The direction between the average value of the boundary sample vector and the region center vector is calculated to obtain the boundary gradient direction. The boundary sub-region boundary is indented inward by a predetermined step along the boundary gradient direction. Sampling points are selected on the indented boundary as compressed samples, and samples outside the indented boundary are marked as samples to be removed. This concentrates the sample compression at the edge of the dense region rather than randomly deleting them, thus preserving the central distribution of the dense region and the structural information inside the boundary. For sparse sub-regions, the mean of all sample feature vectors is extracted as the center sample vector, which reflects the concentrated location of existing samples in the sparse sub-region. The covariance matrix of the sample feature vectors in the sparse sub-region is calculated and eigenvalue decomposition is performed. The direction of the feature vector corresponding to the largest eigenvalue is taken as the main expansion direction. Interpolation positions are generated along the main expansion direction starting from the center sample vector. Candidate interpolation samples with a minimum spatial distribution distance greater than a predetermined distance threshold from existing samples are selected as supplementary samples. Interpolation filling is performed along the main distribution direction of the sparse region to reduce the deviation and noise caused by generating samples in arbitrary directions in the sparse region. This ensures that the adjusted training data retains representative boundary samples in dense regions and supplements new samples that conform to the local distribution direction in sparse regions. Attached Figure Description

[0018] To more clearly illustrate the technical solutions in the embodiments of this application or the prior art, the drawings used in the embodiments will be briefly introduced below. Obviously, the drawings described below are only some embodiments recorded in this invention. For those skilled in the art, other drawings can be obtained based on these drawings.

[0019] Figure 1 This is a flowchart of an industry model training data optimization method based on sample distribution analysis; Figure 2 This is a flowchart of the sample feature extraction and sample distribution distance matrix construction process; Figure 3 This is a flowchart of a training dataset optimization method based on sample adjustment vectors; Figure 4 This is a schematic diagram of the distribution of dense sub-region boundary samples and sparse sub-region center samples based on sample feature vectors. Detailed Implementation

[0020] To make the objectives, technical solutions, and advantages of the embodiments of the present invention clearer, the technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0021] See Figure 1 This invention provides a method for optimizing industry model training data based on sample distribution analysis, comprising: acquiring an original industry training dataset; extracting multivariate features of each sample in the original industry training dataset to generate a sample feature vector; calculating the spatial distribution distance between any two samples based on the positional relationship of all the sample feature vectors in the feature space to construct a sample distribution distance matrix; dividing the feature space into regions based on the sample distribution distance matrix to obtain multiple candidate sample sub-regions, and calculating the sample density value within each candidate sample sub-region; selecting candidate sample sub-regions with sample density values ​​greater than a preset density threshold as dense sub-regions, and selecting candidate sample sub-regions with sample density values ​​less than a preset density threshold as sparse sub-regions; extracting boundary sample vectors from each dense sub-region and extracting center sample vectors from each sparse sub-region, and combining the boundary sample vectors and the center sample vectors to generate a sample adjustment vector; and adjusting the sample distribution of the original industry training dataset according to the sample adjustment vector to generate an optimized training dataset.

[0022] In specific implementation, please refer to Figure 2 The process involves acquiring the original industry training dataset, extracting multivariate features from each sample in the dataset, and generating a sample feature vector. The original industry training dataset is obtained by reading historical business records from the industry business system database. Each sample in the original industry training dataset contains multiple fields, including numeric and categorical fields. Field parsing is performed on each sample in the original industry training dataset to identify the numeric and categorical fields contained in each sample. Numeric fields represent attributes recorded in continuous numerical form, while categorical fields represent attributes recorded in discrete categorical form. Numeric fields are normalized using a min-max normalization method, mapping them to the 0-1 range. Categorical fields are encoded using a unique encoding method, converting them into multidimensional 0-1 vectors. After normalization and encoding / mapping, all fields of each sample are converted into an initial feature array in a unified format. The length of the initial feature array is determined by the number of numeric fields and the sum of the dimensions of the categorical fields after one-hot encoding.

[0023] In practice, principal component analysis (PCA) is performed on the initial feature array to extract principal component features with a cumulative variance contribution rate greater than a preset contribution rate threshold. These principal component features are then combined into sample feature vectors. PCA calculates the covariance matrix of the initial feature array, performs eigenvalue decomposition on the covariance matrix to obtain multiple eigenvalues ​​and corresponding eigenvectors, arranges the eigenvalues ​​in descending order, and calculates the cumulative variance contribution rate. The preset contribution rate threshold is set to 0.90. This threshold is chosen because retaining principal components with a cumulative variance contribution rate of 90% in the original features can cover most sample information while reducing feature dimensionality and lowering the complexity of subsequent sample distribution distance calculations. When the cumulative variance contribution rate first exceeds the preset contribution rate threshold, the first few corresponding principal component eigenvectors are used as retained principal component directions. The initial feature array of each sample is projected onto these retained principal component directions to obtain dimensionality-reduced principal component features, which are then combined into sample feature vectors.

[0024] In practice, based on the positional relationship of all sample feature vectors in the feature space, the spatial distribution distance between any two samples is calculated, and a sample distribution distance matrix is ​​constructed. The difference in each dimension between the corresponding sample feature vectors of any two samples is obtained, and the square root of the sum of the squares of each difference is taken to obtain the first distance value. The first distance value reflects the Euclidean distance between the two samples in the feature space. The cosine similarity between the corresponding sample feature vectors of any two samples is obtained, and the difference between the first and cosine similarity is determined as the second distance value. The second distance value reflects the directional difference between the two samples in the feature space. The first and second distance values ​​are weighted and summed according to preset first and second weight coefficients to obtain the spatial distribution distance between any two samples, and the spatial distribution distances between all samples constitute the sample distribution distance matrix.

[0025] The formula for calculating spatial distribution distance is: in, Indicates the first The sample and the first Spatial distribution distance between samples; This represents the preset first weighting coefficient; This represents the preset second weighting coefficient; Indicates the first The sample and the first The first distance value between samples calculated based on Euclidean distance; Indicates the first The sample and the first The second distance value is calculated between samples based on the difference in cosine similarity.

[0026] Preset first weighting coefficient The value range is 0 to 1, and the preset second weighting coefficient is... The value range is from 0 to 1, and First weighting coefficient Second weighting coefficient The specific value is determined based on the relative importance of numerical and categorical features in the sample space distribution of the original industry training dataset. When numerical features dominate in sample differentiation... Set to greater than The value; when categorical features dominate in distinguishing samples. Set to less than The value of the first distance. The calculation method is as follows: obtain the first... The sample feature vector of the nth sample and the nth sample Given the feature vectors of two samples, calculate the difference between the feature vectors of two samples in each dimension, and then take the square root of the sum of the squares of the differences in all dimensions. Second distance value. The calculation method is as follows: obtain the first... The sample feature vector of the nth sample and the nth sample Given the feature vectors of two samples, calculate the cosine similarity between the feature vectors of the two samples, and use the difference between the numerical value 1 and the cosine similarity as the second distance value.

[0027] The sample distribution distance matrix is ​​formed by the spatial distribution distances between all samples. This sample distribution distance matrix is ​​a symmetric matrix. The nth sample in the sample distribution distance matrix... Line 1 The elements of the column are The diagonal elements of the sample distribution distance matrix are 0.

[0028] In practice, based on the sample distribution distance matrix, the feature space is divided into regions to obtain multiple candidate sample sub-regions, and the sample density value within each candidate sample sub-region is calculated. The minimum spatial distribution distance corresponding to each sample is extracted from the sample distribution distance matrix, and the initial neighborhood radius is determined based on this minimum spatial distribution distance. When extracting the minimum spatial distribution distance for each sample, each row of the sample distribution distance matrix is ​​traversed, and after removing the diagonal elements in each row, the minimum value among the remaining elements is taken as the minimum spatial distribution distance for the sample corresponding to that row. After extracting the minimum spatial distribution distances for all samples, the average of all minimum spatial distribution distances is calculated, and this average value is used as the initial neighborhood radius. The reason for using the average of all minimum spatial distribution distances as the initial neighborhood radius is that the minimum spatial distribution distance reflects the spatial distance between a sample and its nearest neighbor, and the average value can characterize the overall nearest neighbor distance level of a sample in the feature space.

[0029] In practice, each sample is used as the center point, and the initial neighborhood radius is used as the search radius to determine the initial neighborhood sample set corresponding to each sample. For any sample, the spatial distribution distance between that sample and every other sample is calculated, and samples whose spatial distribution distance is less than the initial neighborhood radius are added to the initial neighborhood sample set of that sample. Based on the number of samples in the initial neighborhood sample set, the local density corresponding to each sample is determined. The local density is directly represented by the number of samples in the initial neighborhood sample set; a larger local density indicates that there are more neighboring samples around the sample, and a smaller local density indicates that there are fewer neighboring samples around the sample. All samples are sorted in descending order of local density to obtain a density sequence. Each element in the density sequence contains a sample identifier and a local density value.

[0030] In practice, segmentation points are determined based on the local density differences between adjacent samples in the density sequence. The local density difference between every two adjacent samples in the density sequence is calculated, resulting in a set of local density differences. The average of all local density differences in the set is calculated and used as the segmentation threshold. Adjacent positions where the local density difference is greater than the segmentation threshold are identified as segmentation points. These segmentation points are located between two adjacent samples in the density sequence. The feature space is divided into regions using the segmentation points as boundaries, resulting in multiple candidate sample sub-regions. Specifically, samples between two adjacent segmentation points in the density sequence are grouped into the same candidate sample sub-region; samples before the first segmentation point are grouped into the first candidate sample sub-region; and samples after the last segmentation point are grouped into the last candidate sample sub-region. Samples within a candidate sample sub-region exhibit continuity in local density, and adjacent samples with a local density difference less than the segmentation threshold belong to the same candidate sample sub-region.

[0031] In practice, the sample density value within each candidate sample sub-region is calculated. The sample density value within a candidate sample sub-region is represented by the average local density of all samples within that sub-region. The sum of the local densities of all samples within each candidate sample sub-region is calculated, and this sum is divided by the number of samples within the candidate sample sub-region to obtain the sample density value of the candidate sample sub-region.

[0032] In practice, candidate sample sub-regions with sample density values ​​greater than a preset density threshold are selected as dense sub-regions, and candidate sample sub-regions with sample density values ​​less than the preset density threshold are selected as sparse sub-regions. The mean and standard deviation of the sample density values ​​of all candidate sample sub-regions are calculated, and the sum of the mean and standard deviation is determined as the preset density threshold. The formula for calculating the preset density threshold is: in, Indicates the preset density threshold; This represents the average sample density value of all candidate sample sub-regions; This represents the standard deviation of the sample density values ​​for all candidate sample sub-regions. The calculation method is as follows: sum the sample density values ​​of all candidate sample sub-regions and divide by the number of candidate sample sub-regions. The calculation method is as follows: for each candidate sample sub-region, the sample density value and the average value are... The squared differences between the regions are summed, and the sum of all squared values ​​is divided by the number of candidate sample sub-regions. The square root of the quotient is then taken. The basis for determining the sum of the mean and standard deviation as the preset density threshold is that the mean reflects the general level of sample density, while the standard deviation reflects the degree of fluctuation in sample density. The sum of the mean and standard deviation can distinguish between regions with significantly higher than the general level of density and regions with significantly lower than the general level of density.

[0033] In practice, candidate sample sub-regions with sample density values ​​greater than a preset density threshold are marked as dense sub-regions, those with sample density values ​​less than the preset density threshold are marked as sparse sub-regions, and those with sample density values ​​equal to the preset density threshold are marked as transitional sub-regions. Transitional sub-regions do not participate in the subsequent extraction of boundary sample vectors and center sample vectors. Dense sub-regions containing fewer than a preset lower limit of sample number are removed from all dense sub-regions, and sparse sub-regions containing more than a preset upper limit of sample number are removed from all sparse sub-regions. The preset lower limit of sample number is set to 3. The reason for setting the preset lower limit of sample number to 3 is that when the number of samples in a dense sub-region is less than 3, the selection of the first predetermined number of boundary samples during the boundary sample vector extraction process lacks sufficient candidate samples, and the local density statistical significance is insufficient. The preset upper limit for the number of samples is set to twice the average number of samples in all candidate sample sub-regions. The basis for setting the preset upper limit for the number of samples to twice the average number of samples in all candidate sample sub-regions is that when the number of samples in a sparse sub-region exceeds twice the average level, the sample distribution in that region is not sparse and should not be used as a sparse sub-region for subsequent extraction of the center sample vector.

[0034] In specific implementation, please refer to Figure 3Boundary sample vectors are extracted from each dense sub-region. For each dense sub-region, a first predetermined number of samples closest to the boundary of the dense sub-region are selected as boundary samples, and the corresponding sample feature vector of each boundary sample is extracted as the boundary sample vector. The boundary of the dense sub-region is determined by the convex hull of the sample feature vectors in the feature space of the dense sub-region. The convex hull of all sample feature vectors in the dense sub-region is constructed, and the sample feature vectors located on the surface of the convex hull are determined as candidate boundary samples. When the number of candidate boundary samples is greater than the first predetermined number, they are sorted in ascending order of distance from each candidate boundary sample to the convex hull surface, and the first predetermined number of candidate boundary samples with the smallest distance are selected as boundary samples. When the number of candidate boundary samples is less than or equal to the first predetermined number, all candidate boundary samples are selected as boundary samples. The first predetermined number is set to 10% of the number of samples in the dense sub-region. The reason for setting the first predetermined number to 10% of the number of samples in the dense sub-region is that the proportion of boundary samples in the dense sub-region is usually small, and 10% can cover the main boundary samples at the boundary of the dense sub-region, while avoiding redundancy of boundary information in the sample adjustment vector due to an excessive number of boundary sample vectors. Extract the sample feature vector corresponding to each boundary sample as the boundary sample vector. The boundary sample vector retains the multi-dimensional feature representation of the original sample.

[0035] In practice, a center sample vector is extracted from each sparse sub-region. For each sparse sub-region, the mean vector of the feature vectors of all samples in the sparse sub-region is taken as the center sample vector. The mean vector is calculated as follows: the feature vectors of all samples in the sparse sub-region are summed one by one along each dimension to obtain the cumulative value in each dimension. The cumulative value in each dimension is then divided by the number of samples in the sparse sub-region to obtain the mean in each dimension. The vector formed by the means in all dimensions is the center sample vector. The center sample vector represents the geometric center position of the sample in the feature space of the sparse sub-region.

[0036] In practice, the boundary sample vector and the center sample vector are weighted and concatenated according to the type of their respective sub-regions to generate a sample adjustment vector. During the weighted concatenation process, type weights are assigned based on the type of the sub-region to which the boundary and center sample vectors belong. Each source sample vector is multiplied by its type weight and then concatenated in order to form the sample adjustment vector. The formula for generating the sample adjustment vector is: in, Represents the first element in the sample adjustment vector. 1 vector element; Indicates the first Each vector element corresponds to a source sample vector, which is either a boundary sample vector or a center sample vector. Indicates the first The type weights corresponding to each vector element; This represents the index of the vector element in the sample adjustment vector, with a value ranging from 1 to the total number of vector elements in the sample adjustment vector.

[0037] Type weight According to the The type of the sub-region to which the source sample vector corresponding to the nth vector element belongs is determined. When the source sample vector corresponding to each vector element is a boundary sample vector, the type weights are used. The value is a boundary sample vector type weight. When the first When the source sample vector corresponding to each vector element is the center sample vector, the type weights The value is the center sample vector type weight. Boundary sample vector type weights The value range is from 0 to 1, and the center sample vector type weight is... The value range is from 0 to 1, and Boundary sample vector type weights The specific value is the total number of boundary sample vectors divided by the sum of the total number of boundary sample vectors and the total number of center sample vectors, with the center sample vector type weight. The specific value is 1 minus the boundary sample vector type weight. When the sum of the total number of boundary sample vectors and the total number of center sample vectors is 0, the sample adjustment vector is empty.

[0038] In practice, the boundary gradient direction of each dense sub-region is determined based on the boundary sample vectors in the sample adjustment vector. The boundary gradient direction is determined according to the distribution extension trend of the boundary sample vectors in the feature space within the dense sub-region. All boundary sample vectors in the dense sub-region are obtained, and the mean of all boundary sample vectors in each dimension is calculated to obtain the mean position of the boundary sample vectors. The mean of the sample feature vectors of all samples in the dense sub-region in each dimension is also calculated to obtain the center position of the dense sub-region. The direction pointing from the mean position of the boundary sample vectors to the center position of the dense sub-region is determined as the boundary gradient direction. Edge shrinkage sampling is performed on the samples at the boundary of the dense sub-region along the boundary gradient direction to obtain a compressed sample set. During the edge shrinkage sampling process, a predetermined step size is moved from the boundary of the dense sub-region into the interior of the dense sub-region along the boundary gradient direction. The predetermined step size is set to 0.5 times the average spatial distribution distance between the sample feature vectors of all samples in the dense sub-region. After the movement, a shrinkage boundary is obtained. A second predetermined number of sampling points are selected on the shrinkage boundary, and the second predetermined number is set to half the number of boundary samples in the dense sub-region. The feature vectors corresponding to the sampling points are determined as compressed samples and added to the compressed sample set. Samples located outside the shrinkage boundary within a dense sub-region are marked as samples to be removed, and these samples are removed from the original industry training dataset.

[0039] In the specific implementation, the central expansion direction of each sparse sub-region is determined based on the central sample vector in the sample adjustment vector. The difference between the sample feature vectors of all samples in the sparse sub-region and the central sample vector is calculated to obtain the deviation vector for each sample. All deviation vectors are summed along their dimensions and divided by the number of samples to obtain the average deviation vector. The direction from the central sample vector to the end of the average deviation vector is determined as the central expansion direction. Interpolation filling sampling is performed along the central expansion direction within the sparse sub-region to obtain a supplementary sample set. During the interpolation filling sampling process, starting from the central sample vector, a third predetermined number of interpolation positions are selected along the central expansion direction. This third predetermined number is set to half the number of samples in the sparse sub-region. A candidate interpolation sample is generated at each interpolation position, and the sample feature vector of the candidate interpolation sample is the feature vector at the corresponding interpolation position. The minimum spatial distribution distance between each candidate interpolation sample and the existing samples in the sparse sub-region is calculated. Candidate interpolation samples with a minimum spatial distribution distance greater than a predetermined distance threshold are determined as supplementary samples and added to the supplementary sample set. The predetermined distance threshold is set to 0.8 times the average spatial distribution distance between the sample feature vectors of all samples in the sparse sub-region.

[0040] In practice, the compressed sample set and the supplementary sample set are merged with the unadjusted samples in the original industry training dataset to generate an optimized training dataset. During the merging process, samples marked as to be removed from all dense sub-regions in the original industry training dataset are first removed. Then, the compressed samples from the compressed sample set are added to the original industry training dataset. Finally, the supplementary samples from the supplementary sample set are added to the original industry training dataset. The remaining samples in the original industry training dataset, the compressed samples from the compressed sample set, and the supplementary samples from the supplementary sample set together constitute the optimized training dataset. The feature vectors of the samples in the optimized training dataset are stored in the same format as the original industry training dataset and are used for subsequent industry model training.

[0041] See Figure 4 In the figure, the horizontal axis represents the first dimension of the sample feature vector, and the vertical axis represents the second dimension. Blue dots represent dense sub-region samples, and orange triangles represent sparse sub-region samples. Dense sub-region samples exhibit a relatively clustered distribution in the feature space, mainly concentrated in the range of approximately -2 to 3 in the first dimension and approximately -2 to 3 in the second dimension; sparse sub-region samples are more dispersed, with the first dimension ranging from approximately 0 to 10 and the second dimension ranging from approximately -2 to 9.

[0042] In the figure, the red pentagrams mark the boundary sample vectors extracted from the dense sub-regions. These boundary sample vectors are located at the edges of the dense sub-regions, which conforms to the step of determining boundary samples by constructing the convex hull of the dense sub-region samples in the above embodiment. The number of boundary sample vectors accounts for approximately 10% of the total number of samples in the dense sub-regions, and they are distributed along the periphery of the dense sub-regions, reflecting the spatial distribution characteristics of the boundary samples.

[0043] The green diamond represents the central sample vector extracted from the sparse sub-region. This central sample vector is the mean vector of all sample feature vectors in the sparse sub-region and is located at the geometric center of the samples in the sparse sub-region. The central sample vector is located at approximately 4 in the first dimension and approximately 3 in the second dimension, which reflects the central characteristics of the sparse sub-region well.

[0044] The diagram clearly distinguishes between dense sub-region samples, boundary sample vectors, and sparse sub-region samples, and center sample vectors, vividly illustrating the technical solution in the above embodiment for extracting boundary sample vectors and center sample vectors for dense and sparse sub-regions respectively. The boundary sample vectors and center sample vectors provide the basic data for subsequently generating sample adjustment vectors, used for adjusting the sample distribution and optimizing the construction of the training dataset.

[0045] In specific implementation, the boundary gradient direction of each dense sub-region is determined based on the boundary sample vector in the sample adjustment vector. For each dense sub-region, the average value of all boundary sample vectors in the dense sub-region is obtained to obtain the boundary center vector. The boundary sample vector is the sample feature vector corresponding to the boundary sample extracted from the dense sub-region in the above embodiment. The boundary center vector is calculated as follows: sum all boundary sample vectors in the dense sub-region one by one according to each dimension to obtain the cumulative value in each dimension; divide the cumulative value in each dimension by the total number of boundary sample vectors to obtain the mean in each dimension; the vector formed by the means in all dimensions is the boundary center vector. The region center vector of the dense sub-region is obtained. The region center vector is calculated as follows: sum the sample feature vectors of all samples in the dense sub-region one by one according to each dimension to obtain the cumulative value in each dimension; divide the cumulative value in each dimension by the total number of samples in the dense sub-region to obtain the mean in each dimension; the vector formed by the means in all dimensions is the region center vector. The direction from the boundary center vector to the region center vector is determined as the boundary gradient direction.

[0046] In practice, the boundary gradient direction is represented by a unit direction vector. The boundary gradient vector is obtained by subtracting the boundary center vector from the region center vector. Each component of the boundary gradient vector is then divided by its magnitude to obtain the unit direction vector of the boundary gradient direction. The magnitude of the boundary gradient vector is calculated by squaring each component, summing all the squares, and taking the square root. The boundary gradient direction points from the boundary of the dense sub-region to the center of the dense sub-region, indicating the contraction direction of edge contraction sampling.

[0047] In practice, along the boundary gradient direction, the region is indented inward by a predetermined step size from the boundary of the dense sub-region. A second predetermined number of sampling points are selected on the indented boundary, and the feature vectors corresponding to these sampling points are determined as compressed samples. The predetermined step size is set to 0.5 times the average spatial distribution distance between the feature vectors of all samples in the dense sub-region. The average spatial distribution distance between the feature vectors of all samples in the dense sub-region is calculated by extracting the spatial distribution distance between every two samples in the dense sub-region from the sample distribution distance matrix, summing all spatial distribution distances, and then dividing by the number of distance values. The reason for setting the predetermined step size to 0.5 times the average spatial distribution distance is that when the indentation distance is half the average nearest neighbor distance, the indented boundary is located near the boundary of the dense sub-region and does not exceed the center of the region, avoiding excessive contraction that could alter the sample distribution within the dense sub-region.

[0048] In practice, the shrunken boundary is determined based on the boundary gradient direction and a predetermined step size. Each boundary position on the boundary of the dense sub-region is moved along the boundary gradient direction by a predetermined step size, and all moved boundary positions constitute the shrunken boundary. A second predetermined number of sampling points are selected on the shrunken boundary, set to half the number of boundary samples in the dense sub-region. The reason for setting the second predetermined number to half the number of boundary samples in the dense sub-region is that the number of sampling points required for the shrunken boundary is less than the number of original boundary samples, thus reducing redundancy of samples near the boundary of the dense sub-region while preserving the main shape features of the boundary. The sampling points are selected as follows: a second predetermined number of positions are selected at uniform intervals on the shrunken boundary, with each position corresponding to one sampling point. The feature vector of the sampling point is the feature vector at the corresponding position on the shrunken boundary. The feature vector corresponding to the sampling point is determined as a compressed sample, and the compressed sample is added to a compressed sample set.

[0049] In practice, samples located outside the indented boundary within a dense sub-region are marked as samples to be removed. To determine whether each sample within the dense sub-region is outside the indented boundary, the spatial distribution distance from the sample's feature vector to the indented boundary is calculated. If the sample's feature vector is located on the opposite side of the gradient direction along the indented boundary, the sample is marked as a sample to be removed. Samples to be removed are those between the boundary of the dense sub-region and the indented boundary. Samples to be removed are removed from the original industry training dataset, and compressed samples are added to the original industry training dataset. When removing samples to be removed, the original sample record corresponding to the sample to be removed is deleted from the original industry training dataset.

[0050] In practical implementation, the central expansion direction of each sparse sub-region is determined based on the central sample vector in the sample adjustment vector. The covariance matrix of the sample eigenvectors of all samples in the sparse sub-region is obtained. Eigenvalue decomposition is performed on the covariance matrix to obtain multiple eigenvalues ​​and corresponding eigenvectors. The direction of the eigenvector corresponding to the largest eigenvalue is determined as the main expansion direction. The formula for calculating the covariance matrix of the sample eigenvectors of all samples in the sparse sub-region is as follows: in, The covariance matrix represents the sample feature vectors of all samples in the sparse subregion; This represents the total number of samples in the sparse subregion; Represents the sparse subregion of the first The sample feature vector of each sample, which is in column vector form; The mean vector of the sample feature vectors of all samples in the sparse subregion is the center sample vector, which is in column vector form. Indicates the index of the sample in the sparse sub-region. The value range is 1 to ; Indicates the transpose operation; Represents the sparse subregion of the first The difference vector between the feature vector of each sample and the center sample vector.

[0051] In practical implementation, the covariance matrix The calculation method for eigenvalue decomposition is as follows: solve for the condition that... eigenvalues and eigenvectors All the solved eigenvalues ​​are arranged in descending order, and the eigenvector corresponding to the largest eigenvalue is determined as the main expansion direction vector. The main expansion direction vector is the unit direction vector obtained by normalizing the eigenvector corresponding to the largest eigenvalue. The main expansion direction represents the direction in which the sample feature distribution extends the most in the sparse sub-region. Interpolation filling along the main expansion direction can expand the sample coverage of the sparse sub-region.

[0052] In practice, starting from the central sample vector, a third predetermined number of interpolation positions are selected along the main expansion direction, and a candidate interpolation sample is generated at each interpolation position. The third predetermined number is set to half the number of samples in the sparse sub-region. This third predetermined number is set to half the number of samples in the sparse sub-region to ensure that the number of interpolated supplementary samples does not exceed the original number of samples, thus avoiding an imbalance between the supplementary samples and the original samples in the sparse sub-region. The spacing between interpolation positions is set to the average spatial distribution distance between the feature vectors of all samples in the sparse sub-region. The interpolation positions are determined as follows: starting from the central sample vector, the position is moved sequentially along the main expansion direction by one spacing, two spacings, up to the third predetermined number of spacings; each moved position is considered an interpolation position. A candidate interpolation sample is generated at each interpolation position, and the feature vector of the candidate interpolation sample is the feature vector corresponding to the interpolation position. The feature vector corresponding to the interpolation position is obtained by adding an integer multiple of the spacing to the central sample vector and multiplying it by the unit vector in the main expansion direction.

[0053] In specific implementation, candidate interpolation samples whose minimum spatial distribution distance from existing samples within the sparse sub-region is greater than a predetermined distance threshold are identified as supplementary samples and added to the supplementary sample set. The spatial distribution distance between the feature vector of each candidate interpolation sample and the feature vector of each existing sample within the sparse sub-region is calculated, and the minimum of all spatial distribution distances is determined as the minimum spatial distribution distance of that candidate interpolation sample. The spatial distribution distance is calculated using the same method as in the previous embodiment when constructing the sample distribution distance matrix. The predetermined distance threshold is set to 0.8 times the average spatial distribution distance between the feature vectors of all samples within the sparse sub-region. The basis for setting the predetermined distance threshold to 0.8 times the average spatial distribution distance is that supplementary samples need to maintain a certain degree of separation from existing samples within the sparse sub-region to avoid overlap and to prevent supplementary samples from being too far from existing samples and exceeding the range of the sparse sub-region. Candidate interpolation samples whose minimum spatial distribution distance is greater than the predetermined distance threshold are identified as supplementary samples and added to the supplementary sample set.

[0054] In practice, the boundary gradient direction is corrected based on the principal component directions of the local covariance matrix of samples within the dense sub-region. The corrected edge contraction direction is obtained by weighting the boundary gradient direction with the principal component directions. The boundary gradient direction is determined in the above embodiment by pointing the boundary center vector to the region center vector. The principal component directions are obtained by performing eigenvalue decomposition on the local covariance matrix of samples within the dense sub-region.

[0055] In practice, the local covariance matrix of samples within a dense sub-region is obtained. To obtain the local covariance matrix, the feature vectors of all samples within the dense sub-region are first summed along each dimension. The accumulated value in each dimension is then divided by the total number of samples within the dense sub-region to obtain the mean vector of the samples in the dense sub-region. For each sample within the dense sub-region, the mean vector of the samples in the dense sub-region is subtracted from the feature vector of the sample to obtain the difference vector of that sample. The difference vector is then multiplied by its transpose to obtain the difference matrix of that sample. Finally, the difference matrices of all samples within the dense sub-region are summed, and the resulting matrix is ​​divided by the total number of samples within the dense sub-region to obtain the local covariance matrix. The local covariance matrix is ​​a symmetric matrix, and the number of rows and columns of the local covariance matrix are equal to the number of dimensions of the sample feature vectors.

[0056] In practice, the local covariance matrix is ​​decomposed into eigenvalues ​​and corresponding eigenvectors. During eigenvalue decomposition, the equation satisfying that the product of the local covariance matrix and its eigenvectors equals the product of the eigenvalues ​​and their eigenvectors is solved, yielding all eigenvalues ​​and their corresponding eigenvectors. All eigenvalues ​​are arranged in descending order, and the eigenvector corresponding to the largest eigenvalue is determined as the principal component direction. The principal component direction is represented by a unit direction vector; the eigenvector corresponding to the largest eigenvalue is divided by its own magnitude to obtain the unit direction vector of the principal component direction. The principal component direction represents the direction in which the feature distribution of samples within a dense sub-region extends the most.

[0057] In practice, the boundary gradient direction is weighted and combined with the principal component directions to obtain the corrected edge contraction direction. The formula for calculating the corrected edge contraction direction is: in, This represents the corrected edge contraction direction, in vector form. The boundary gradient direction is represented in vector form. The direction of the principal component is represented in vector form. This represents the deflection weight, with a value ranging from 0 to 1.

[0058] In practice, the deflection weight The deflection weight is determined by the proportion of the eigenvalue corresponding to the principal component direction in the sum of all eigenvalues. The eigenvalue corresponding to the principal component direction is the largest eigenvalue obtained from the eigenvalue decomposition of the local covariance matrix. The sum of all eigenvalues ​​is the cumulative value of all eigenvalues ​​obtained from the eigenvalue decomposition of the local covariance matrix. The ratio obtained by dividing the largest eigenvalue by the sum of all eigenvalues ​​is used as the deflection weight. Deflection weights The larger the value, the greater the direction of the corrected edge contraction. The closer to the principal component direction ; Deflection weight The smaller the value, the more the corrected edge contraction direction... The closer to the boundary gradient direction .

[0059] In practical implementation, when calculating the corrected edge contraction direction, the boundary gradient direction is considered. Multiply Multiply The two product results are added together using vector addition to obtain the corrected edge contraction direction vector. The corrected edge contraction direction vector is then divided by its own magnitude to obtain the corrected edge contraction direction in unit form.

[0060] In practice, the corrected edge contraction direction is used to replace the boundary gradient direction for edge contraction sampling. Along the corrected edge contraction direction, the boundary of the dense sub-region is indented inward by a predetermined step size. This predetermined step size is set to 0.5 times the average spatial distribution distance between the feature vectors of all samples in the dense sub-region. A second predetermined number of sampling points are selected on the indented boundary, which is set to half the number of boundary samples in the dense sub-region. The feature vectors corresponding to these sampling points are identified as compressed samples, and these compressed samples are added to a compressed sample set. Samples located outside the indented boundary within the dense sub-region are marked as samples to be removed, and these samples are removed from the original industry training dataset.

[0061] In practice, the selection of compressed samples and the marking of samples to be removed are both performed along the corrected edge contraction direction. To determine whether each sample within the dense sub-region is located outside the contracted boundary, the spatial distribution distance from the sample's feature vector to the contracted boundary is calculated. If the sample's feature vector is located on the opposite side of the contracted boundary along the corrected edge contraction direction, the sample is marked as a sample to be removed. The samples to be removed are those between the original boundary of the dense sub-region and the contracted boundary. The contracted boundary is determined based on the corrected edge contraction direction and a predetermined step size. Each boundary position on the dense sub-region boundary is moved by a predetermined step size along the corrected edge contraction direction, and all moved boundary positions constitute the contracted boundary.

[0062] The above description is merely a specific embodiment of this application, but the scope of protection of this application is not limited thereto. Any changes or substitutions that can be easily conceived by those skilled in the art within the scope of the technology disclosed in this application should be included within the scope of protection of this application.

Claims

1. A method for optimizing industry model training data based on sample distribution analysis, characterized in that, include: Obtain the original industry training dataset, extract the multivariate features of each sample in the original industry training dataset, and generate a sample feature vector; Based on the positional relationship of all the sample feature vectors in the feature space, calculate the spatial distribution distance between any two samples and construct the sample distribution distance matrix; Based on the sample distribution distance matrix, the feature space is divided into regions to obtain multiple candidate sample sub-regions, and the sample density value in each candidate sample sub-region is calculated. Candidate sample sub-regions with sample density values ​​greater than a preset density threshold are selected as dense sub-regions, and candidate sample sub-regions with sample density values ​​less than a preset density threshold are selected as sparse sub-regions. Boundary sample vectors are extracted from each of the dense sub-regions, and center sample vectors are extracted from each of the sparse sub-regions. The boundary sample vectors and the center sample vectors are combined to generate a sample adjustment vector. The original industry training dataset is adjusted in terms of sample distribution based on the sample adjustment vector to generate an optimized training dataset.

2. The method for optimizing industry model training data based on sample distribution analysis according to claim 1, characterized in that, The process of obtaining the original industry training dataset, extracting multivariate features from each sample in the original industry training dataset, and generating a sample feature vector includes: For each sample in the original training dataset of the industry, perform field parsing to identify the numerical and categorical fields contained in each sample; The numerical fields are normalized, and the categorical fields are encoded and mapped to generate an initial feature array in a unified format. Principal component analysis is performed on the initial feature array to extract principal component features whose cumulative variance contribution rate is greater than a preset contribution rate threshold, and the principal component features are combined into the sample feature vector.

3. The method for optimizing industry model training data based on sample distribution analysis according to claim 2, characterized in that, The step of calculating the spatial distribution distance between any two samples and constructing a sample distribution distance matrix based on the positional relationship of all the sample feature vectors in the feature space includes: Obtain the difference in each dimension between the feature vectors corresponding to any two samples, and take the square root of the sum of the squares of each dimension difference to obtain the first distance value. Obtain the cosine similarity between the sample feature vectors corresponding to any two samples, and determine the difference between the first and the cosine similarity as the second distance value; The first distance value and the second distance value are weighted and summed according to the preset first weight coefficient and the preset second weight coefficient to obtain the spatial distribution distance between any two samples, and the spatial distribution distance between all samples constitutes the sample distribution distance matrix.

4. The method for optimizing industry model training data based on sample distribution analysis according to claim 3, characterized in that, The step involves dividing the feature space into regions based on the sample distribution distance matrix to obtain multiple candidate sample sub-regions, and calculating the sample density value within each candidate sample sub-region, including: Extract the minimum spatial distribution distance for each sample from the sample distribution distance matrix, and determine the initial neighborhood radius based on the minimum spatial distribution distance; Using each sample as the center point and the initial neighborhood radius as the search radius, determine the initial neighborhood sample set corresponding to each sample; Based on the number of samples in the initial neighborhood sample set, determine the local density corresponding to each sample, and arrange all samples in descending order of the local density to obtain a density sequence; The segmentation point is determined based on the local density difference between adjacent samples in the density sequence. The feature space is then divided into regions using the segmentation point as the boundary to obtain multiple candidate sample sub-regions.

5. The method for optimizing industry model training data based on sample distribution analysis according to claim 4, characterized in that, The step of selecting candidate sample sub-regions with sample density values ​​greater than a preset density threshold as dense sub-regions and selecting candidate sample sub-regions with sample density values ​​less than a preset density threshold as sparse sub-regions includes: Calculate the mean and standard deviation of the sample density values ​​of all candidate sample sub-regions, and determine the sum of the mean and the standard deviation as the preset density threshold; Candidate sample sub-regions with sample density values ​​greater than the preset density threshold are marked as dense sub-regions, candidate sample sub-regions with sample density values ​​less than the preset density threshold are marked as sparse sub-regions, and candidate sample sub-regions with sample density values ​​equal to the preset density threshold are marked as transitional sub-regions. Remove dense sub-regions containing fewer than a preset lower limit of sample number from all the dense sub-regions, and remove sparse sub-regions containing more than a preset upper limit of sample number from all the sparse sub-regions.

6. The method for optimizing industry model training data based on sample distribution analysis according to claim 5, characterized in that, The step of extracting boundary sample vectors from each of the dense sub-regions and extracting center sample vectors from each of the sparse sub-regions, and combining the boundary sample vectors and the center sample vectors to generate a sample adjustment vector includes: For each dense sub-region, the first predetermined number of samples closest to the boundary of the dense sub-region are obtained as boundary samples, and the sample feature vector corresponding to each boundary sample is extracted as the boundary sample vector. For each sparse sub-region, the mean vector of the sample feature vectors of all samples in the sparse sub-region is obtained as the center sample vector. The boundary sample vector and the center sample vector are weighted and concatenated according to the type of their respective sub-regions to generate the sample adjustment vector.

7. The method for optimizing industry model training data based on sample distribution analysis according to claim 6, characterized in that, The step of adjusting the sample distribution of the original industry training dataset according to the sample adjustment vector to generate an optimized training dataset includes: Based on the boundary sample vector in the sample adjustment vector, determine the boundary gradient direction of each dense sub-region, and perform edge shrinkage sampling on the samples at the boundary of the dense sub-region along the boundary gradient direction to obtain a compressed sample set; Based on the central sample vector in the sample adjustment vector, the central expansion direction of each sparse sub-region is determined, and interpolation filling sampling is performed in the sparse sub-region along the central expansion direction to obtain a supplementary sample set. The compressed sample set and the supplementary sample set are merged with the unadjusted samples in the original industry training dataset to generate an optimized training dataset.

8. The method for optimizing industry model training data based on sample distribution analysis according to claim 7, characterized in that, The step involves determining the boundary gradient direction of each dense sub-region based on the boundary sample vector in the sample adjustment vector, and performing edge shrinkage sampling on the samples at the boundary of the dense sub-region along the boundary gradient direction to obtain a compressed sample set, including: The average value of all boundary sample vectors in the dense sub-region is obtained to obtain the boundary center vector. The region center vector of the dense sub-region is obtained, and the direction from the boundary center vector to the region center vector is determined as the boundary gradient direction. Along the boundary gradient direction, the boundary of the dense sub-region is indented inward by a predetermined step size, and a second predetermined number of sampling points are selected on the indented boundary. The feature vectors corresponding to the sampling points are determined as compressed samples. The samples located outside the indented boundary within the dense sub-region are marked as samples to be removed. The samples to be removed are removed from the original industry training dataset, and the compressed samples are added to the original industry training dataset.

9. The method for optimizing industry model training data based on sample distribution analysis according to claim 8, characterized in that, The step involves determining the center expansion direction of each sparse sub-region based on the center sample vector in the sample adjustment vector, and performing interpolation filling sampling along the center expansion direction within the sparse sub-region to obtain a supplementary sample set, including: Obtain the covariance matrix of the sample feature vectors of all samples in the sparse sub-region, perform eigenvalue decomposition on the covariance matrix to obtain multiple eigenvalues ​​and corresponding eigenvectors, and determine the direction of the eigenvector corresponding to the largest eigenvalue as the main expansion direction. Starting from the central sample vector, a third predetermined number of interpolation positions are selected along the main expansion direction, and a candidate interpolation sample is generated at each interpolation position. Candidate interpolation samples whose minimum spatial distribution distance from existing samples in the sparse sub-region is greater than a predetermined distance threshold are identified as supplementary samples and added to the supplementary sample set.

10. The method for optimizing industry model training data based on sample distribution analysis according to claim 7, characterized in that, The boundary gradient direction is corrected based on the principal component direction of the local covariance matrix of the samples in the dense sub-region. The corrected edge contraction direction is obtained by weighting the boundary gradient direction with the principal component direction.