Service data processing method, system, equipment and medium

By preprocessing cross-domain economic data and detecting semantic drift in real time, and combining dynamic alignment signals for incremental training of the mapping model, the problem of semantic drift in cross-domain economic data processing is solved, enabling real-time detection and automatic compensation of data, and improving the accuracy and timeliness of data processing.

CN121579859APending Publication Date: 2026-02-27CHINA UNIV OF PETROLEUM (EAST CHINA)
View PDF 0 Cites 2 Cited by

Patent Information

Application Number
CN202511723896.0
Authority / Receiving Office
CN · China
Patent Type
Applications(China)
Current Assignee / Owner
Filing Date
2025-11-22
Publication Date
2026-02-27

AI Technical Summary

Technical Problem

Existing technologies cannot detect and automatically compensate for semantic shifts in cross-domain economic data in real time, resulting in insufficient accuracy and timeliness of data processing, as well as high maintenance costs.

Method used

By preprocessing cross-domain economic data, detecting semantic drift in real time, generating drift detection results, and incrementally training the mapping model based on dynamic alignment signals, the real-time mapping and transformation of data is finally achieved.

Benefits of technology

It enables real-time detection and automatic compensation of cross-regional economic data, improving the accuracy, timeliness, and adaptability of data processing.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure CN121579859A_ABST
    Figure CN121579859A_ABST
Patent Text Reader

Abstract

The invention relates to a business data processing method and system, equipment and a medium. The method comprises the following steps: preprocessing multi-source heterogeneous cross-domain economic data to generate a standardized data stream; semantic drift in the standardized data flow is detected in real time, and a detection result is generated; semantic alignment judgment is carried out based on the detection result, and a dynamic alignment signal is generated; performing incremental training of a mapping model by using the dynamic alignment signal to generate an updated cross-domain mapping model; and finally, performing mapping conversion on the data stream based on the updating model, and outputting service data with unified semantics. By adopting the method, the semantic change in the economic data can be responded in real time, the problems of semantic drift detection lag, long model updating period and high maintenance cost in the traditional technology are effectively solved, and the accuracy and timeliness of cross-domain economic data processing are remarkably improved.
Need to check novelty before this filing date? Find Prior Art

Description

TECHNICAL FIELD

[0001] The application belongs to the technical field of data processing, and particularly relates to a business data processing method, system, device and medium. BACKGROUND

[0002] With the acceleration of economic globalization, the field of economic data analysis generally needs to integrate multi-source heterogeneous data from different countries, regions or statistical standards. These data have significant differences in index definition, statistical caliber and calculation method, forming complex cross-domain data integration needs.

[0003] In the traditional technology, there are mainly two ways to process cross-domain economic data: one is a data mapping method based on predefined rules, which realizes data standardization by establishing fixed field conversion rules and unit conversion coefficients; the second is a semantic mapping model based on batch training, which trains the conversion model offline using historical data and uses it fixedly within a certain period. These methods try to solve the semantic inconsistency problem between different source economic data through preset mapping relationship or periodic model updating.

[0004] However, the existing technology has obvious deficiencies: first, static rule mapping cannot adapt to the frequent semantic changes in economic data. When the statistical standard or index definition is updated, manual intervention is needed to modify the mapping rules, which has a lag response and high maintenance cost; second, the batch training model has a long update period and cannot respond to semantic drift in real-time data flow in time, resulting in a significant time delay between the occurrence of drift and the correction of the model; third, the existing solution lacks an online detection and automatic triggering mechanism for semantic drift, making it difficult to discover and compensate for changes in data semantics in time. These problems seriously affect the accuracy and timeliness of cross-domain economic data processing. SUMMARY

[0005] Therefore, it is necessary to provide a business data processing method, system, device and medium capable of real-time detection of semantic drift and rapid triggering of compensation mechanism in view of the above technical problems.

[0006] In a first aspect, the application provides a business data processing method, comprising:

[0007] S1, preprocessing cross-domain economic data from a plurality of heterogeneous data sources to generate a standardized economic data stream;

[0008] S2, detecting semantic drift of the standardized economic data stream to generate a drift detection result;

[0009] S3, judging semantic alignment according to the drift detection result to generate a dynamic alignment signal;

[0010] S4, based on the dynamic alignment signal, incremental training of the mapping model is performed to generate an updated cross-domain mapping model;

[0011] S5, based on the updated cross-domain mapping model, the standardized economic data stream is mapped and converted to generate semantic unified business data.

[0012] In one of the embodiments, S2 includes:

[0013] S21, the standardized economic data stream is divided into time windows to generate a current window data set;

[0014] S22, the statistical characteristics of the current window data set are extracted to generate current data distribution characteristics;

[0015] S23, the reference distribution of the historical economic data is modeled to generate a baseline data distribution;

[0016] S24, the difference between the current data distribution characteristics and the baseline data distribution is calculated to generate a distribution difference value;

[0017] S25, according to the distribution difference value and the preset drift determination condition, threshold judgment processing is performed to generate a drift detection result.

[0018] In one of the embodiments, S24 includes:

[0019] S241, the probability density of the current data distribution characteristics and the baseline data distribution is estimated to generate a first probability distribution and a second probability distribution;

[0020] S242, the JS divergence of the first probability distribution and the second probability distribution is calculated to generate an initial difference value; the initial difference value calculation formula is:

[0021]

[0022] wherein, is the initial difference value, P is the first probability distribution, Q is the second probability distribution, M is the mean distribution of P and Q, is the KL divergence calculation function;

[0023] S243, the first probability distribution and the second probability distribution are converted into cumulative distribution functions to generate a first cumulative distribution function and a second cumulative distribution function;

[0024] S244, the first cumulative distribution function and the second cumulative distribution function are calculated by KS test to generate a distribution deviation value;

[0025] S245, the distribution difference value and the distribution deviation value are weighted and fused to generate a distribution difference value.

[0026] In one of the embodiments, S3 comprises:

[0027] S31, according to the economic indicators in the drift detection result, generating a concept association graph through knowledge graph query processing;

[0028] S32, performing graph embedding processing on the concept association graph to generate a concept vector representation;

[0029] S33, performing similarity calculation on the concept vector representation to generate a semantic similarity matrix;

[0030] S34, comparing each element value in the semantic similarity matrix with a preset similarity threshold to generate a preliminary alignment decision;

[0031] S35, performing signal generation condition judgment processing on the preliminary alignment decision through predefined business rules to obtain a dynamic alignment signal.

[0032] In one of the embodiments, S33 comprises:

[0033] S331, performing vector normalization processing on the concept vector representation to generate a standardized vector;

[0034] S332, performing improved similarity calculation on the standardized vector to generate a concept similarity value; the calculation formula of the concept similarity value is:

[0035]

[0036] wherein, is the concept similarity value, is the i-th concept vector, is the j-th concept vector, is a distance weight adjustment parameter, is an interaction weight adjustment parameter, denotes a Hadamard product operator, is the L2 norm of the vector;

[0037] S333, performing matrix construction on the concept similarity value to generate an initial similarity matrix;

[0038] S334, performing domain knowledge correction on the initial similarity matrix to generate a corrected similarity matrix;

[0039] S335, performing sparsification processing on the corrected similarity matrix to generate a semantic similarity matrix.

[0040] In one of the embodiments, S4 comprises:

[0041] S41, data acquisition is performed from the training sample library according to the dynamic alignment signal, and an incremental training set is generated;

[0042] S42, feature enhancement processing is performed on the incremental training set, and an enhanced feature representation is generated;

[0043] S43, model parameter update processing is performed on the enhanced feature representation, and an intermediate model is generated;

[0044] S44, performance evaluation is performed on the intermediate model, and a model performance indicator is generated;

[0045] S45, early stopping judgment processing is performed according to the model performance indicator, and an updated cross-domain mapping model is generated.

[0046] In one of the embodiments, S43 includes:

[0047] S431, the enhanced feature representation is divided into small batches to generate training batch data;

[0048] S432, forward propagation calculation is performed on the training batch data to generate a prediction result;

[0049] S433, loss calculation is performed on the prediction result to generate a loss value;

[0050] S434, parameter optimization processing is performed according to the loss value to generate updated model parameters, and the calculation formula of the updated model parameters is:

[0051]

[0052] wherein, is the updated model parameter, is the model parameter of the t-th iteration, is the adaptive learning rate, is the first moment estimate, is the second moment estimate, is a numerical stability parameter;

[0053] S435, smoothing processing is performed on the updated model parameters to generate an intermediate model.

[0054] In a second aspect, the present application also provides a business data processing system, comprising:

[0055] A data preprocessing module for preprocessing cross-domain economic data from a plurality of heterogeneous data sources to generate standardized economic data streams;

[0056] A semantic drift detection module for detecting semantic drift of the standardized economic data stream to generate a drift detection result;

[0057] The semantic alignment judgment module is configured to perform semantic alignment judgment according to the drift detection result, and generate a dynamic alignment signal.

[0058] The mapping model incremental training module is configured to perform incremental training of the mapping model based on the dynamic alignment signal, and generate an updated cross-domain mapping model.

[0059] The mapping conversion processing module is configured to perform mapping conversion processing on the standardized economic data stream based on the updated cross-domain mapping model, and generate business data with unified semantics.

[0060] In a third aspect, the present application also provides a computer device, comprising a memory and a processor, wherein the memory stores a computer program, and the processor implements the processing method of business data according to the first aspect when executing the computer program.

[0061] In a fourth aspect, the present application also provides a computer readable storage medium, which stores a computer program, and the computer program is executed by a processor to implement the processing method of business data according to the first aspect.

[0062] The processing method, system, device and medium of business data described above, by first preprocessing the cross-domain economic data from multiple heterogeneous data sources to generate a standardized economic data stream, then performing semantic drift detection on the data stream to identify data distribution anomalies and generate a drift detection result, then performing semantic alignment judgment according to the result combined with a knowledge graph to generate a dynamic alignment signal, then updating the cross-domain mapping model based on the signal through incremental learning to adapt to semantic changes, and finally applying the updated model to the standardized economic data stream for real-time mapping conversion, the real-time detection and automatic compensation of semantic drift in cross-domain economic data are realized, and the accuracy, timeliness and self-adaptation ability of cross-domain economic data processing are effectively improved. BRIEF DESCRIPTION OF DRAWINGS

[0063] In order to more clearly illustrate the technical solutions in the embodiments of the present application or the related art, the drawings needed to be used in the embodiment or related art description will be briefly introduced. Obviously, the drawings in the following description are only some embodiments of the present application, and other drawings can be obtained by those skilled in the art without creative labor.

[0064] Figure 1 A flowchart of a processing method of business data provided by the present application;

[0065] Figure 2 A flowchart of a method for generating an updated cross-domain mapping model in an optional embodiment provided by the present application;

[0066] Figure 3A structural schematic diagram of a service data processing system provided by the present application. DETAILED DESCRIPTION

[0067] For the purpose, technical solutions and advantages of the present application to be more clearly and obviously understood, the present application is further described in detail below in combination with the drawings and embodiments. It should be understood that the specific embodiments described herein are only used to explain the present application and do not limit the present application.

[0068] In one embodiment, as shown in Figure 1 A service data processing method is provided, and the present embodiment is exemplified by the method applied to a data processing terminal. It should be understood that the method can also be applied to a server, and can also be applied to a system including the data processing terminal and the server, and is realized through the interaction of the data processing terminal and the server. In the present embodiment, the method includes the following steps:

[0069] S1, preprocessing cross-domain economic data from multiple heterogeneous data sources to generate standardized economic data stream.

[0070] Optionally, the data source access link adopts modular design and supports dynamic adaptation of multiple core protocols. For government open API data sources, data acquisition is realized through an interface calling mechanism, and identity authentication and retry strategies are configured; for enterprise private databases, connection pool technology is used to ensure access efficiency and security; for batch historical file data, transmission protocols and integrity checking mechanisms are combined to ensure data reliability; for high-concurrency real-time data streams, asynchronous access is realized through a message queue to balance data consumption rate; for unstructured data, OCR technology and regular expressions are used to extract key indicator information.

[0071] The data cleaning link focuses on missing value, abnormal value and repeated value processing. Missing value processing uses adaptive methods for different data types: cubic spline interpolation method is used for time series data, and the calculation formula is:

[0072] wherein, is the target value after interpolation, is the time point to be interpolated, is the time point corresponding to the known data, , , , and coefficients of the piecewise interpolation function are solved by the boundary conditions of "function continuity, first derivative continuity, second derivative continuity in the interpolation interval"; the mode attribute data is filled by the mode filling method, and the reasonable mode is selected according to the semantic correlation of the data in the field; the numerical non-time series data adopts the median filling method to avoid the influence of extreme values on the filling result. The abnormal value processing adopts a double detection-classification processing mechanism: first, the suspected abnormal values are screened by the box plot method, and the four quantiles (lower quartile), (upper quartile) are calculated, and the abnormal value determination interval is determined as , wherein represents the interquartile range; then the real abnormal values are verified by the local anomaly factor algorithm, the ratio of the local reachable density of the data point to the average local reachable density of the neighborhood data points is calculated, the abnormal degree is determined according to the ratio, and the correction-elimination classification processing is performed. The repeated value processing takes the index code + statistical period + statistical area as the unique identification combination, stores the processed identification through the hash table, and realizes the screening of redundant data.

[0073] The field, unit, code and format are unified in the format standardization link. The field standardization is based on the construction of the standard field library of the national economic accounting system, and the heterogeneous field name is mapped to the standard field by using the bidirectional longest matching segmentation algorithm, and the field library supports dynamic expansion; the unit standardization establishes a "basic unit-conversion coefficient" mapping library to unify different types of units into international standard units, and the conversion process is calculated by formula according to the real-time obtained reference data (such as exchange rate), without involving specific coefficient values; the code standardization adopts the international general coding system to unify the coding of statistical area, index type, statistical period, etc.; the format standardization clearly defines the unified format requirements of data storage, including date and time format, numerical type, text coding, missing value identification, etc.

[0074] S2, performing semantic drift detection on the standardized economic data stream to generate a drift detection result.

[0075] Optionally, the semantic feature modeling constructs a three-dimensional system of "text semantic features + structured attribute features + calculation logic features". The text semantic feature extraction adopts an economic field optimized multilingual model, and the preprocessed input text generates a fixed dimension semantic vector, and the semantic correlation degree is measured by the cosine similarity formula, and the calculation formula is: ,

[0076] wherein is the similarity value of the two semantic vectors, , are the two semantic vectors to be compared, represents the dot product of the two vectors, , Respectively represent the length of two vectors. The structured attribute feature extraction converts the structured fields such as statistical period, region code, index type into numerical features, and generates fixed-dimension feature vectors through one-hot encoding, embedding layer encoding, etc. The calculation logic feature extraction describes the calculation method of data in the text, and generates keyword feature vectors through TF-IDF algorithm. The TF (term frequency) calculation formula is: TF = number of occurrences of keywords in the text / total number of words in the text; the IDF (inverse document frequency) calculation formula is: IDF = log (total number of documents / number of documents containing the keyword + 1); the TF-IDF value is the product of TF and IDF, and finally the high-weight keywords are selected to form the feature vector. The three-dimensional feature vectors are spliced to form the comprehensive semantic feature vector.

[0077] The benchmark semantic feature distribution construction adopts the initial sample + rolling update strategy. The standardized data verified by artificial verification without semantic drift is selected as the initial benchmark sample, and the kernel density estimation is performed on the comprehensive semantic feature vector thereof. A Gaussian kernel function is adopted, and the calculation formula thereof is: ,

[0078] Among them, is the kernel function value, is the feature vector to be estimated, is the feature vector in the benchmark sample, is the bandwidth parameter of the kernel function, is the constant pi (the value is about 3.1416), is the exponential function, is the Euclidean distance of two feature vectors. After generating the benchmark probability density function, it is stored in the distributed cache, and a rolling update mechanism is set. Every time a certain amount of data without drift detection result is accumulated, the kernel density estimation is recalculated and the benchmark distribution is updated. The difference is determined to ensure the timeliness of the benchmark distribution.

[0079] The real-time semantic drift detection adopts a sliding window mechanism, sets the window size and sliding step, only specifies the mechanism, and does not involve specific numerical values. The comprehensive semantic feature vector of the data in the current window is re-performed kernel density estimation to obtain the current distribution. The difference between the current distribution and the benchmark distribution is quantified through KL divergence, and the calculation formula thereof is: ,

[0080] Among them, is the KL divergence value, is the current data distribution, is the benchmark data distribution, is the number of intervals after discretization of the feature space, is the feature vector of the th interval, The logarithmic function is set. The three-level threshold is calibrated by offline experiment, the drift degree is judged according to the KL divergence value, and the trend verification mechanism is introduced to continuously monitor the KL divergence trend of multiple sliding windows, avoid the misjudgment of single window detection, and finally generate the drift detection result.

[0081] S3, according to the drift detection result, the semantic alignment judgment is carried out, and the dynamic alignment signal is generated.

[0082] Optionally, the difference analysis link is carried out for different drift detection results: no drift result needs to verify the semantic consistency of the data, extract the comprehensive semantic feature vector of the current data and the benchmark feature distribution for dimension-by-dimension similarity calculation; the type of suspicious feature dimension needs to be analyzed for slight drift result, the economic semantic connotation of the dimension is queried combined with the semantic ontology knowledge base, and it is judged whether the drift is reasonable fluctuation; the core drift feature dimension needs to be located for serious drift result, the semantic feature change trajectory of the historical and current data is compared, the drift reason is clarified combined with the data source update announcement, and the semantic mapping relationship before and after the drift is extracted.

[0083] The semantic alignment matching degree calculation constructs a multi-dimensional evaluation system, including semantic similarity, caliber consistency and calculation compatibility. The semantic similarity is calculated based on the cosine similarity of the text semantic feature vector, the formula is consistent with the cosine similarity formula in the semantic feature modeling link, and the meanings of the parameters are the same; the caliber consistency is evaluated by the matching degree of the structured attribute features, and the weighted sum formula is adopted, which is expressed as: ,

[0084] wherein, is the caliber consistency value, , , statistical period, regional code and index type matching degree are respectively, , , The calculation compatibility is calculated based on the Jaccard similarity of the calculation logic feature vector, and the calculation formula is: ,

[0085] wherein, is the calculation compatibility value, is the current calculation logic keyword set, is the benchmark calculation logic keyword set, is the number of intersection elements of the two sets, is the number of union elements of the two sets. The comprehensive matching degree is calculated by using the weighted average formula, and the weighted results of the three indexes are used as the core basis for semantic alignment judgment.

[0086] The alignment strategy decision is based on the comprehensive matching degree and the drift detection result to formulate a differentiated scheme: no drift and high comprehensive matching degree, determine that no semantic alignment is needed, generate a dynamic alignment signal of "maintaining the existing alignment strategy"; slight drift and medium comprehensive matching degree, determine that local semantic alignment is needed, generate a "local alignment adjustment" signal, and specify the fine-tuning dimension and execution priority; slight drift and low comprehensive matching degree, determine that moderate semantic alignment is needed, generate a "moderate alignment adjustment" signal, including the core alignment field and the training data screening condition; severe drift and medium comprehensive matching degree, determine that deep semantic alignment is needed, generate a "deep alignment adjustment" signal, and specify the drift type and the incremental training data size requirement; severe drift and low comprehensive matching degree, determine that complete semantic alignment is needed, generate a "complete alignment adjustment" signal, including the new semantic mapping relationship table and the manual review trigger condition. The dynamic alignment signal is output in a standardized format, including signal ID, generation time, associated information, execution parameters, confidence, and validity period.

[0087] S4, based on the dynamic alignment signal, incrementally train the mapping model to generate an updated cross-domain mapping model.

[0088] Optionally, the training strategy adaptation link selects the corresponding training scheme according to the alignment strategy type of the dynamic alignment signal: the "maintain existing alignment strategy" signal only performs model performance verification and does not update parameters; the "local alignment adjustment" signal uses a combination of rule fine-tuning and model parameter micro-update to adjust only the parameters related to the drift dimension; the "moderate alignment adjustment" signal starts the model local update process, and the training set is composed of associated historical data and current data; the "deep alignment adjustment" signal performs complete incremental training, and the training set includes historical support vector data, current drift data, and supplementary labeled data; the "complete alignment adjustment" signal uses a "rule reconstruction + incremental training" hybrid scheme, which first reconstructs the mapping rule library and then performs large-scale incremental training.

[0089] The training data preprocessing and labeling link follows strict quality standards: the selected training data is cleaned again to remove duplicate values and outliers, and the outlier processing method in the semantic drift detection link is used; automatic labeling is performed according to the semantic mapping relationship in the dynamic alignment signal, and the labeling rule is: input feature vector → target standard vector; samples with low automatic labeling confidence are pushed to the manual labeling platform, and the labeling personnel calibrate them combined with the semantic ontology knowledge base and the alignment signal, and after calibration, they are re-added to the training set. The training data is divided into training set, validation set, and test set using stratified sampling method to ensure that the semantic distribution and drift type proportion of each data set are consistent.

[0090] The cross-domain mapping model adopts a hybrid architecture of deep learning and rule engine. The core model is a combination of incremental support vector machine (SVM) and Transformer encoder: the Transformer encoder is responsible for extracting high-level semantic features, with a fixed number of network layers and attention heads; the incremental SVM is responsible for mapping features to standard semantics, using a support vector preservation-new data training strategy. The objective function is expressed as: ,

[0091] in, This is the weight vector of the model. Let the magnitude of the weight vector be . The penalty coefficient is... The number of training samples. For the first The goal is to minimize the combination of the magnitude of the weight vector and the weighted sum of the slack variables for each sample. Stochastic gradient descent is used to optimize the objective function, and cosine annealing is employed as the learning rate. Model performance is monitored using a validation set during iterations, and an early stopping mechanism is implemented to prevent overfitting.

[0092] The model evaluation process employs a multi-indicator comprehensive system, including mapping accuracy, precision, recall, F1 score, and semantic consistency score. The mapping accuracy formula is: A = Number of mapped samples / Total number of samples; the precision formula is: P = Number of samples / (Number of true positive samples + Number of false positive samples); the recall formula is: R = Number of samples / (Number of true positive samples + Number of false negative samples); the F1 score formula is: The semantic consistency score is calculated based on the cosine similarity between the output vector and the standard vector. Evaluation criteria are set: models that pass the evaluation are automatically deployed; models that fail require supplementary data for retraining or manual intervention. Model version management uses semantic version numbers, storing training parameters, performance metrics, data snapshots, and associated signal information for each version, and supports version rollback.

[0093] S5. Based on the updated cross-domain mapping model, the standardized economic data stream is mapped and transformed to generate semantically unified business data.

[0094] Optionally, a four-step processing logic is adopted: in the first step, based on the three-dimensional feature system of the semantic feature modeling link, the comprehensive semantic feature vector of the current data is extracted to ensure consistency with the model input dimension; in the second step, the feature vector is input into the updated cross-domain mapping model to output the preliminary mapping result, including standard index coding, mapping confidence and core mapping logic; in the third step, the rule engine calls the adaptive rules in the mapping rule library to correct the preliminary mapping result, including unit conversion, region merging, statistical period unification and other operations. The conversion process is calculated according to the coefficient in the dynamic alignment signal: standard value = original mapping value x adjustment coefficient, where the standard value is the corrected target value, the original mapping value is the preliminary value output by the model, and the adjustment coefficient is the parameter generated according to the historical data fitting; in the fourth step, the results involving statistical caliber adjustment are executed for value conversion to ensure data semantic consistency.

[0095] The data consistency verification link adopts a combination of "real-time verification + batch verification": real-time verification is for single data conversion result, checking standard index code validity, value rationality (judging range based on index historical data distribution), semantic consistency (judging similarity between converted semantic vector and standard vector), and classifying and marking abnormal data; batch verification is for each batch of conversion results, checking index distribution consistency (judging deviation of index number proportion between current batch and historical batch), value statistical consistency (judging deviation of mean and standard deviation between current batch and historical batch), and cross-data source consistency (judging conversion result difference of the same index in different data sources). When batch verification is abnormal, the conversion process is suspended, and after checking the matching of the model and the alignment signal, it is restarted.

[0096] The semantic unified business data generation link follows the principle of "standard field-unified format-metadata complete": standard field setting covers core fields of data identification, semantic information, value information and metadata information, including data unique ID, standard index code, standard index name, statistical period, statistical region information, standard value, currency unit, value precision, data source, model version, rule version, mapping confidence, data state, conversion time, note, etc., and clearly defines the meaning and function of each field; the data format is unified as column storage format, using compression algorithm and unified coding, and clearly defining the format requirements and compression logic; metadata information ensures that each data contains complete traceability information, supports reverse query of original data, conversion process, model version and alignment signal, and meets the audit and traceability requirements.

[0097] The business data output link supports multi-channel distribution: the database writing supports mainstream database types, adopts a batch insertion strategy and opens transaction management to ensure data atomicity; the file output supports multiple common file formats, and the user can configure the output path and file name rules to clearly define the block storage mechanism; the API interface output adopts the RESTful specification, provides query and export two request methods, and clearly defines the response time requirements and asynchronous processing mechanism; the data stream output supports message queues, which push data to the downstream analysis platform in real time, set the queue topic, partition number and replica number to ensure high availability of transmission. The output data performs integrity verification to verify data integrity through a checksum algorithm, and finally generates semantic unified business data that meets business needs.

[0098] In the above business data processing method, the multi-source heterogeneous data is standardized and preprocessed, the data semantic drift is detected, the semantic alignment is judged based on the drift result, the cross-domain mapping model is incrementally trained, and finally the data is converted through the model. The economic data from different sources and in disordered formats are converted into standard and unified business data, making data processing more standardized and solving the problem of slow fixed rule changes and long time model updates in traditional methods.

[0099] In one of the embodiments, S2 includes:

[0100] S21, performing time window division processing on the standardized economic data stream to generate a current window data set.

[0101] Optionally, time window division is the basis for realizing real-time semantic drift detection, and the core is to balance the detection timeliness and computing resource consumption through fixed time dimension data set slicing. The division adopts a sliding window mode, and the window size and sliding step are set according to the update frequency (such as minute level, hour level) of the standardized economic data stream and the real-time demand of the business for drift detection, to ensure that both data dynamic changes can be captured and resource waste caused by frequent calculation can be avoided.

[0102] Data screening needs to follow the principle of time continuity, only data with timestamps completely covering the window interval are included, data exceeding the interval or with time gaps, missing data caused by transmission delay are excluded, to ensure the consistency of the time dimension of the data set. At the same time, preliminary preprocessing is performed: based on the unique data identifier (such as DATA ID), duplicate data is removed to avoid feature calculation bias; the non-emptiness of the core field is checked, such as standard indicator code, statistical period, standard value, and data with missing core fields are removed. After division, the current window data set is generated, which needs to include data entries, window identifiers (such as start time-end time) and other core meta information, to facilitate subsequent association and tracing with historical window data.

[0103] S22, performing statistical feature extraction on the current window data set to generate current data distribution features.

[0104] Optionally, the core of statistical feature extraction is to transform window data into quantifiable distribution indicators, providing a basis for comparing differences, focusing on two key features: distribution pattern and temporal correlation. Distribution pattern features focus on the central tendency and dispersion of the data, with core indicators including mean and variance.

[0105] The formula for calculating the mean is: ,

[0106] in, This is the mean of a numeric field (such as a standard number) in the current window. This is the number of valid data entries for this field. For the first The field values ​​of each data point are used to reflect the central trend of the data.

[0107] The formula for calculating variance is: ,

[0108] in, For variance, The mean, The number of valid data entries. For the first The field values ​​of each data point are used to reflect the degree of dispersion of the data from the mean.

[0109] Time-series correlation features target economic data with time-series attributes. The core indicator is the autocorrelation coefficient, and the formula is: ,

[0110] in, Lagging The autocorrelation coefficient of order, The lag order is set based on the data time granularity. The number of valid data entries. The mean, and The first Article, No. The field values ​​of each data point are used to reflect the degree of correlation between the data at different points in time.

[0111] In addition, basic indicators of dispersion (such as quartiles and range) are added, and all features are integrated to form the current data distribution feature set.

[0112] S23. Perform reference distribution modeling on historical economic data to generate a baseline data distribution.

[0113] Optionally, the core of reference distribution modeling is to build a comparison benchmark based on historical semantic drift-free data, ensuring the representativeness and stability of the benchmark. The historical data screening needs to meet three core conditions: no semantic drift verified by manual verification, covering multiple statistical periods (such as multiple months and multiple quarters), covering different data source scenarios, while ensuring data integrity (no missing core fields) and time continuity to avoid benchmark distortion.

[0114] The modeling adopts a core method of combining parameter method and non-parameter method: for fields that meet the theoretical distribution (such as Gaussian distribution), the parameter method is adopted, and for Gaussian distribution, the mean and variance are calculated by maximum likelihood estimation to build a benchmark model , The benchmark probability density function is , and the field value is ,

[0115] where is the benchmark probability density function , the number of valid historical data is the kernel function bandwidth, which is adaptively selected according to the data distribution is the Gaussian kernel function: ,

[0116] where is the field value to be estimated is the field value of the th historical data.

[0117] Set the benchmark update mechanism: accumulate a certain amount of new semantic drift-free data, re-model and compare the new and old benchmarks through the difference degree such as KL divergence, if the difference degree is less than the threshold, replace the old benchmark, if the difference degree is greater than the threshold, trigger manual audit to avoid the benchmark being outdated. The final generated benchmark data distribution model needs to include model type, core parameters (such as mean, bandwidth), data source, etc. information, and is stored in the benchmark library for calling.

[0118] S24, calculate the difference degree of the current data distribution characteristics and the benchmark data distribution, and generate a distribution difference degree value.

[0119] Optionally, the core of difference degree calculation is to quantify the deviation degree of the current distribution and the benchmark distribution, and select the core index for different distribution types. For benchmarks constructed by parameter method, such as Gaussian distribution, the parameter difference degree index is adopted, and the core is the relative difference of mean and variance:

[0120] Mean relative difference degree: , is the current mean, is the reference mean value;

[0121] is the relative difference of variance; is the current variance, is the reference variance.

[0122] For the reference constructed by the non-parametric method, the core uses the KL divergence to quantify the overall difference between the distributions, and the formula is:

[0123] where, is the KL divergence of the current distribution relative to the reference distribution, and are the probability density functions of the current and reference distributions, respectively, used to reflect the information difference between the two.

[0124] After all the indicators are calculated, the comprehensive distribution difference value is generated by weighted fusion, and the core formula is:

[0125] where, is the comprehensive difference value, is the number of indicators, is the weight of the th indicator, is set according to the sensitivity of the indicator to drift, is the difference value of the th indicator. The final result needs to be accompanied by an explanation of the composition and weight of the indicators to ensure interpretability.

[0126] S25, according to the distribution difference value and the preset drift judgment condition, threshold judgment processing is performed to generate a drift detection result.

[0127] Optionally, the core of threshold judgment is to realize drift grading judgment based on the comprehensive difference value, to ensure that the result adapts to business needs. The preset judgment condition is constructed through offline experiments: calculate the difference value based on different sample sets of drift degree, such as no drift, slight drift, and severe drift, combined with business misjudgment cost, such as the risk of misjudging severe drift as no drift, and use statistical methods to calibrate multi-level thresholds, such as ROC curve analysis.

[0128] The core judgment logic is divided into three levels:

[0129] No drift: when , is the first threshold value, it is determined that there is no semantic drift, and it is necessary to verify whether a single indicator meets the no drift condition, and after confirmation, the "no drift" result is generated;

[0130] Slight drift: when ,​​​​ For the secondary threshold, determine the slight drift, analyze the main contribution indicators and adjust the information verification according to the data source to determine whether it is a reasonable fluctuation, generate a "slight drift" result and trigger trend monitoring;

[0131] When the threshold is met, a serious drift is determined, the core drift indicator is located and the reason is traced back, such as statistical standard change, and a "serious drift" result is generated and subsequent processing suggestions are proposed, such as triggering semantic alignment.

[0132] A dynamic adjustment mechanism is introduced: when the business scenario changes (such as adding a new data source) or the data accumulates to a certain scale, the threshold is re-optimized; consistency verification is performed on the results of multiple consecutive windows to avoid misjudgment of a single window fluctuation. The final generated drift detection result needs to include the result type, difference value, threshold basis, generation time and other core information.

[0133] In the above embodiment, the standardized data is divided into a time window, the statistical characteristics of the data in the window are extracted, the baseline distribution is constructed based on historical data, the difference between the current distribution and the baseline is calculated, and whether the drift is determined according to the difference. The originally general detection of drift becomes clear and quantifiable steps, which can capture the changes in data semantics in real time and accurately, such as statistical aperture fine-tuning and index definition modification, reducing the misjudgment that may occur in a single link detection.

[0134] In one of the embodiments, S24 includes:

[0135] S241, probability density estimation is performed on the current data distribution characteristics and the baseline data distribution to generate a first probability distribution and a second probability distribution.

[0136] Optionally, the core purpose of probability density estimation is to convert the current data distribution characteristics and the baseline data distribution into a probability distribution form that can be directly used for difference calculation, providing basic data for JS divergence and KS test. The processing objects are respectively the current data distribution characteristics (including statistical indicators such as mean and variance) and the baseline data distribution: for characteristics that conform to a theoretical distribution (such as Gaussian distribution), parameter method is used to estimate the probability density, for example, Gaussian probability density function is constructed based on the mean and variance of the current data distribution characteristics, and the corresponding baseline probability density function is constructed based on the parameters of the baseline data distribution; for characteristics that do not conform to the theoretical distribution, non-parametric method is used, such as kernel density estimation, which directly estimates the probability density from sample data without pre-setting the distribution type. Finally, through the estimation process, a first probability distribution (denoted as P) corresponding to the current data distribution and a second probability distribution (denoted as Q) corresponding to the baseline data distribution are generated, which need to maintain consistent dimensions.

[0137] S242, JS divergence calculation is performed on the first probability distribution and the second probability distribution to generate an initial difference value; the initial difference value calculation formula is:​

[0138]

[0139] wherein, is an initial difference value, P is a first probability distribution, Q is a second probability distribution, M is a mean distribution of P and Q, is a KL divergence calculation function.

[0140] Optionally, the core of the JS divergence calculation is to quantify the overall difference between the first probability distribution P and the second probability distribution Q, solve the non-symmetry problem of the KL divergence, and ensure the objectivity of the difference degree calculation. The calculation process takes the preset formula as the core basis, and the formula is expressed as: ,

[0141] wherein, is an initial difference value, used to reflect the overall information difference between P and Q; P is a first probability distribution, a probability distribution corresponding to current data; Q is a second probability distribution, a probability distribution corresponding to reference data; M is a mean distribution of P and Q, and the construction manner is , that is, the probability density values of P and Q are averaged point by point to form an intermediate transition distribution; is a KL divergence calculation function, used to calculate the information entropy difference between two distributions, such as is the KL divergence from P to M). Through the formula, the KL divergence of P and Q with the mean distribution M is calculated and averaged to obtain the initial difference degree value, and the value range is [0, 1], and the larger the value is, the more significant the difference between P and Q is.

[0142] S243, the first probability distribution and the second probability distribution are converted into cumulative distribution functions to generate a first cumulative distribution function and a second cumulative distribution function.

[0143] Optionally, the cumulative distribution function conversion is to convert the probability density distribution into a cumulative probability distribution, which provides adaptive input data for the KS test. The conversion logic is based on the definition of the cumulative distribution function: for the first probability distribution P, the corresponding first cumulative distribution function is denoted as , which represents the probability that the random variable takes a value less than or equal to x, and the calculation formula is: ,

[0144] wherein, t is an integral variable, is the probability density value of the first probability distribution at t; similarly, the second cumulative distribution function corresponding to the second probability distribution Q is denoted as , and the calculation formula is: . The two cumulative distribution functions obtained after conversion have a value range of [0, 1] and have the characteristics of monotone non-decrease, and can intuitively reflect the cumulative probability difference between the two distributions at different value points.

[0145] S244, KS test calculation is performed on the first cumulative distribution function and the second cumulative distribution function to generate a distribution deviation value.

[0146] Optionally, the KS test calculation is performed by comparing the maximum deviation of the first cumulative distribution function and the second cumulative distribution function to quantify the position and shape difference of the two distributions, and to supplement the deficiency of JS divergence in capturing local differences. The test process takes the two cumulative distribution functions as input, and the core calculation logic is to find the maximum value of the absolute difference between the two function values at all x values, i.e. the distribution deviation value, denoted as K, which is expressed as: ,

[0147] where is the supremum, i.e. the maximum possible value. The value of the distribution deviation value ranges from 0 to 1, and the larger the value, the more significant the deviation of the two cumulative distributions at a certain value point, which can effectively detect the local distribution difference that JS divergence may ignore, such as the case where the overall difference between the two distributions is small but the deviation in a specific interval (such as the high value segment) is large.

[0148] S245, the distribution difference value and the distribution deviation value are weighted and fused to generate a distribution difference value.

[0149] Optionally, the weighted fusion integrates the overall difference information of JS divergence and the local deviation information of KS test to obtain a more comprehensive and accurate final distribution difference value. The fusion process takes the initial difference value and the distribution deviation value as input. First, according to the sensitivity of the two indicators to semantic drift, the weights are set: since JS divergence can better reflect the overall information difference of the distribution, the weight is usually higher than that of KS test distribution deviation value, for example, JS divergence weight = 0.6, distribution deviation value weight = 0.4, and it needs to satisfy = 1, the weight setting needs to be determined in combination with the contribution of the two indicators to the drift detection accuracy in the offline experiment. The fusion calculation formula is ,

[0150] where D is the final generated distribution difference value. Through this fusion process, both the core information of the overall distribution difference and the supplementary information of the local deviation are retained, ensuring that the distribution difference value can fully reflect the deviation of the current data distribution from the reference distribution.

[0151] In the above embodiments, two probability distributions are obtained by performing probability density estimation on the current data and the reference data, and then the initial difference is calculated by JS divergence and the distribution deviation is calculated by KS test, and finally the final difference degree is obtained by weighted fusion of the two. Compared with the traditional calculation method of only looking at the overall difference or local difference, the technical scheme can consider both the overall characteristics and local details of the distribution, such as finding the shift of the overall distribution of the data and capturing the subtle deviation of a certain interval, so that the difference degree calculation is more accurate.

[0152] In one of the embodiments, S3 includes:

[0153] S31, according to the economic indicators in the drift detection result, generating a concept association graph through knowledge graph query processing.

[0154] Optionally, relying on the economic field knowledge graph, the semantic association relationship of the economic indicators in the drift detection result is established. First, the core components of the knowledge graph are determined, which include economic indicator nodes (such as GDP, CPI, import and export volume, etc.), indicator attributes (such as statistical caliber, calculation method, and field of application), and association edges between nodes (such as containing, deriving, and statistical basis). When querying, the economic indicators involved in the drift detection result (such as the "industrial added value" indicator whose semantic change is detected) are used as query keywords, the query interface of the knowledge graph is called, and the node information and relationship data of the indicators and their associated indicators are extracted, such as industrial added value associated with industrial output value, industrial intermediate input, etc. At the same time, the statistical caliber description, historical mapping record, and other attributes of each indicator are obtained. The finally generated concept association graph presents the economic indicators in the form of nodes and the semantic relationship between the indicators in the form of edges with attributes, which intuitively reflects the semantic context of the indicators.

[0155] S32, performing graph embedding processing on the concept association graph to generate a concept vector representation.

[0156] Optionally, the graph embedding processing is to convert the structured information of the concept association graph into a low-dimensional dense vector form, so as to measure the semantic similarity through numerical calculation. In the processing process, a graph embedding algorithm suitable for the association characteristics of economic indicators is adopted: taking Node2Vec as an example, the neighbor sequence of each indicator node is generated by traversing the concept association graph through a random walk strategy, simulating the local structural characteristics of the node in the graph; then the vector representation of the node is learned through the Skip-gram model, and the high-dimensional graph structure information is mapped to a low-dimensional vector space, such as 128 dimensions, 256 dimensions. The generated concept vector representation needs to preserve the semantic association characteristics between indicators, for example, the consumer price index and the commodity retail price index with similar semantics have a closer distance in the vector space, while the financial revenue and the commodity export volume with large semantic difference have a farther distance.

[0157] S33, performing similarity calculation on the concept vector representation to generate a semantic similarity matrix.

[0158] Optionally, the semantic association degree between different concept vectors is quantified by numerical calculation. For concept vectors, a suitable similarity algorithm is selected for pairwise calculation. The commonly used algorithm is cosine similarity. The core logic is to calculate the cosine value of the angle between two vectors to measure the consistency of the vector direction. The closer the direction, the higher the similarity. In the calculation process, all concept vectors form a vector set, and the similarity values of any two vectors in the set are calculated in turn, such as the similarity value of vector A (corresponding to GDP) and vector B (corresponding to regional gross product), the similarity value of vector A and vector C (corresponding to fixed asset investment), etc. Finally, the above similarity values are organized in matrix form to generate a semantic similarity matrix: the rows and columns of the matrix correspond to different economic indicator concepts, and each element value in the matrix is the similarity of the corresponding two concept vectors, with a value range of [0, 1], clearly presenting the semantic similarity degree between all indicators.

[0159] S34, comparing each element value in the semantic similarity matrix with a preset similarity threshold to generate a preliminary alignment decision.

[0160] Optionally, based on the semantic similarity matrix, the indicator pairs that can be semantically aligned are screened in combination with the preset threshold. First, the preset similarity threshold needs to be set in combination with the economic business scenario. For example, for core economic indicators such as GDP and fiscal deficit, the threshold is set higher to ensure alignment accuracy; for secondary auxiliary indicators such as industry sentiment index, the threshold can be appropriately reduced to balance flexibility. When comparing and judging, each element in the semantic similarity matrix is traversed: if the element value (i.e., the semantic similarity of the two indicators) is greater than or equal to the preset threshold, it is determined that the two indicators meet the semantic alignment conditions, and the preliminary decision is that they can be aligned, and the element value is recorded as the alignment confidence; if the element value is less than the preset threshold, it is determined that the semantic difference between the two is significant, and the preliminary decision is that they cannot be aligned. The final preliminary alignment decision presents all alignable indicator pairs and their confidence, unalignable indicator pairs and difference reasons, such as too low similarity value.

[0161] S35, performing signal generation condition judgment processing on the preliminary alignment decision through predefined business rules to obtain a dynamic alignment signal.

[0162] Optionally, the preliminary alignment decision is checked and corrected by business rules to ensure that the final alignment strategy meets the actual business needs. The pre-defined business rules need to cover the key constraints of economic data processing, such as the "core indicator priority alignment rule" (if the core indicator conflicts with the non-core indicator in the preliminary decision, the core indicator alignment result is retained), the "data consistency rule" (the index pair that can be aligned needs to meet the consistency of statistical period and regional range, otherwise the alignment method needs to be adjusted), and the "historical mapping rule" (if there is a historical alignment record for the index pair, the historical strategy needs to be referred to adjust the current decision). When judging and processing, the preliminary alignment decision is matched with the business rules one by one: if a "alignable" decision meets all relevant business rules, it is confirmed that the alignment strategy is valid and is included in the dynamic alignment signal; if it does not meet the rules, such as inconsistent statistical periods of the index pair, the alignment method is adjusted (such as aligning after unifying the statistical period) or marked as "to be manually reviewed"; if the "unalignable" decision meets the business rules, such as two indicators belonging to completely different statistical fields, the decision is maintained. The final generated dynamic alignment signal explicitly includes the valid alignment strategy (such as which index pair needs to be aligned, the alignment method), the to-be-processed items (such as the index pair that needs to be manually reviewed), the execution priority, and other information.

[0163] In the above embodiment, the concept association relationship of economic indicators is queried through the knowledge graph, the association graph is converted into a concept vector, the vector similarity is calculated to obtain a matrix, and then the dynamic alignment signal is generated in combination with a threshold and business rules. The limitations of traditional alignment relying only on data features are broken, professional knowledge in the economic field is introduced, the alignment result is more suitable for actual business scenarios, and the rationality and adaptability of semantic alignment are improved.

[0164] In one of the embodiments, S33 includes:

[0165] S331, performing vector normalization processing on the concept vector representation to generate a standardized vector.

[0166] Optionally, the vector normalization is to eliminate the similarity calculation deviation caused by the magnitude difference of the concept vector, and to ensure the fairness of comparison. Taking the concept vector representation as the processing object, the L2 normalization method is adopted: the L2 norm of each concept vector is calculated, each element of the vector is divided by the norm, the L2 norm of the normalized vector is 1, and finally the standardized vector is generated, avoiding the interference of the vector length on the subsequent similarity result.

[0167] S332, performing improved similarity calculation on the standardized vector to generate a concept similarity value; the calculation formula of the concept similarity value is:

[0168]

[0169] wherein, is the concept similarity value, is the i-th concept vector, is the j-th concept vector, is the distance weight adjustment parameter, is the interaction weight adjustment parameter, represents the Hadamard product operator, is the L2 norm of a vector.

[0170] Optionally, the improved similarity calculation combines vector distance difference and element interaction information to improve accuracy. The standardized vector is input, and a preset formula is used for calculation. The formula is expressed as: ,

[0171] wherein, is the similarity value of the i-th and j-th concepts, taking a value in (0, 1], the closer to 1, the more similar the semantics; , is the corresponding standardized vector; controls the influence of the vector Euclidean distance ( ) on the similarity, controls the influence of the L2 norm of the vector Hadamard product ( ) on the similarity, is the L2 norm calculation. Through the formula, two types of information are fused to generate a concept similarity value that fits the semantic association in the economic field.

[0172] S333, construct a matrix for the concept similarity value to generate an initial similarity matrix.

[0173] Optionally, the matrix construction organizes the scattered similarity values into a structured matrix. According to the number of concepts n, an initial similarity matrix of dimensions is constructed: the row and column indices correspond to each concept, and the matrix elements are At the same time, the symmetry of the matrix is checked, because the similarity has symmetry, to ensure that the element values are consistent, and the initial similarity matrix is generated to realize structured storage.

[0174] S334, correct the initial similarity matrix according to the field knowledge to generate a corrected similarity matrix.

[0175] Optionally, the field knowledge correction is to combine economic field professional knowledge, such as index hierarchical relationship and statistical caliber rules, to modify the values in the initial matrix that do not conform to the business logic. For example, increase the low similarity value of the total GDP and the gross national product, and decrease the high similarity value of the industrial added value and the consumer price index. The basis for recording adjustment is ensured to be traceable, and a corrected similarity matrix that conforms to the business logic is generated.

[0176] S335, perform sparse processing on the corrected similarity matrix to generate a semantic similarity matrix.

[0177] Optionally, sparsification removes redundant low-similarity elements to reduce computational complexity. A preset similarity threshold is set, determined through offline experiments based on the definition of semantic association in the economic field. The correction matrix is ​​traversed: element values ​​below the threshold are set to 0 to determine weak / no association; values ​​above or equal to the threshold are retained, ultimately generating a semantic similarity matrix containing only key semantic associations.

[0178] In the above embodiments, the concept vectors are normalized, similarity is calculated using an improved formula, an initial matrix is ​​constructed, the matrix is ​​then corrected using domain knowledge, and finally sparsification is performed to remove useless information, resulting in a semantic similarity matrix. This avoids similarity calculation bias caused by differences in vector magnitude and also eliminates redundant information with low relevance, making the similarity results more accurate. This provides high-quality data support for semantic alignment decisions and reduces alignment bias.

[0179] In an optional embodiment, such as Figure 2 As shown, S4 includes:

[0180] S41. Data is acquired from the training sample library based on the dynamic alignment signal to generate an incremental training set.

[0181] Optionally, data acquisition involves selecting suitable samples based on the requirements of the dynamic alignment signal to ensure the relevance of incremental training. The dynamic alignment signal includes the direction of semantic alignment (such as adjustment of indicator definition or adaptation of statistical caliber) and data range requirements (such as samples from specific economic fields or statistical periods). Relevant samples are extracted from the training sample library based on these criteria: for example, if the signal points to "semantic adjustment of R&D expenditure indicator," then samples in the sample library containing "R&D expenditure" related annotations and covering the semantics before and after the adjustment are selected. At the same time, basic verification is performed on the extracted samples to remove invalid samples with incorrect annotations or missing core fields, ultimately forming the incremental training set to avoid the impact of irrelevant samples on training efficiency.

[0182] S42. Perform feature enhancement processing on the incremental training set to generate enhanced feature representations.

[0183] Optionally, feature enhancement improves the ability of sample features to represent semantic changes, adapting to dynamic alignment requirements. Based on the original features of the incremental training set, such as indicator semantic vectors and statistical attribute features, enhancement methods adapted to the economic domain are adopted: for example, adding correlation features by combining hierarchical relationships between indicators (e.g., "R&D expenditure" belongs to "fixed asset investment"), or generating caliber-adapted features based on differences in statistical calibers, such as pricing units in different regions; normalizing and discretizing numerical features, and semantically expanding textual features, such as adding synonym vectors. Through the above processing, the generated enhanced feature representation more accurately reflects the key information required for semantic alignment.

[0184] S43. Update the model parameters of the enhanced feature representation to generate an intermediate model.

[0185] Optionally, the core of parameter updating is to incrementally optimize the model based on enhanced features, avoiding full retraining. Based on the existing cross-domain mapping model, enhanced feature representations are input into the model, and gradient descent-like algorithms are used to update parameters: parameters are adjusted only for model layers related to the dynamic alignment signal, such as the semantic mapping layer and feature adaptation layer, while parameters of irrelevant layers, such as the basic feature extraction layer, are fixed to reduce computational overhead. During the update process, the goal is to minimize the mapping error, such as minimizing the difference between the standard semantics output by the model and the semantics of the sample annotations. Iterative updates continue until parameters converge, generating an intermediate model to ensure that the model parameters adapt to the new semantic alignment requirements.

[0186] S44. Evaluate the performance of the intermediate model and generate model performance indicators.

[0187] Optionally, performance evaluation examines the semantic mapping effectiveness of the intermediate model to determine whether it meets business requirements. Core metrics suitable for the economic domain are selected, such as mapping accuracy (the percentage of samples whose model output semantics match the labeled semantics) and semantic consistency score (the average similarity between the model's output vector and the standard semantic vector). Validation samples from the incremental training set are allocated to the intermediate model, accounting for approximately 10%-20%, and the above metrics are calculated. Simultaneously, the performance difference between the intermediate model and the model before the update is compared to clarify the improvement in model performance due to parameter updates, generating model performance metrics.

[0188] S45. Based on the model performance indicators, perform early stopping judgment processing to generate an updated cross-domain mapping model.

[0189] Optionally, the core of early stopping is to avoid model overfitting and determine the optimal model state. The early stopping rule is set as follows: if the model performance metrics show no improvement for several consecutive rounds (e.g., 3-5 rounds), or show a downward trend, the model is considered to have reached its optimal state, and parameter updates are stopped; if the metrics continue to improve, the process returns to S43 to continue parameter updates. After the early stopping judgment is passed, the intermediate model in the current state is determined as the final model. Model parameters, performance metrics, training logs, and other information are recorded, and an updated cross-domain mapping model is generated to ensure that the model maintains good generalization ability while adapting to semantic alignment requirements.

[0190] In the above embodiments, incremental data is obtained from the sample library based on the dynamic alignment signal, feature enhancement is performed on the data, parameters are updated to obtain an intermediate model, model performance is evaluated, and the final updated model is determined through early stopping. The model does not need to be fully retrained every time; it only needs to be trained on the parts of the data whose semantics have changed. This saves computing resources, shortens update time, and can adapt to semantic changes in the data in a timely manner, ensuring that the model is always in a stable and usable state.

[0191] In one embodiment, S43 includes:

[0192] S431. Perform mini-batch partitioning on the enhanced feature representation to generate training batch data.

[0193] Optionally, mini-batch partitioning is used to balance the efficiency and stability of model training, avoiding the risk of computational resource overload or overfitting caused by using all augmented feature representations at once. Taking the augmented feature representations (including feature data and corresponding labels) as the processing object, a fixed batch size is set according to the hardware resources and convergence requirements of model training. The augmented feature representations are then divided into multiple non-overlapping training batches, either sequentially or randomly. Each training batch contains an equal number of feature samples and matching label information, ensuring that the model can gradually learn the mapping relationship between features and labels during subsequent iterations of training. Simultaneously, batch randomization reduces the interference of data order on the training results.

[0194] S432. Perform forward propagation calculations on the training batch data to generate prediction results.

[0195] Optionally, forward propagation computation involves inputting the training batch data into an existing cross-domain mapping model. The model's computational logic across each layer generates prediction results, achieving a preliminary mapping from features to labels. After the enhanced features from the training batch data are input into the model, they sequentially pass through a feature extraction layer (such as a Transformer encoder) and a mapping layer (such as a fully connected layer): the feature extraction layer performs high-level semantic representation refinement on the enhanced features, and the mapping layer converts the refined features into output values ​​consistent with the label dimension, ultimately generating the prediction result for each sample. The dimension of the prediction result remains consistent with the label dimension; for example, if the label is a single-dimensional standard indicator value, the prediction result will also be a single-dimensional value.

[0196] S433. Calculate the loss based on the prediction results and generate the loss value.

[0197] Optionally, loss calculation quantifies the deviation between the predicted result and the true labels in the training batch data, providing direction for model parameter optimization. Using the predicted result and the true labels as input, an appropriate loss function is selected based on the business scenario; for example, mean squared error loss is used for regression tasks, and cross-entropy loss is used for classification tasks. The loss value for each sample is obtained by calculating the difference between the two, and then the loss values ​​of all samples within the batch are averaged to generate the loss value corresponding to that training batch. A larger loss value indicates a greater deviation between the predicted result and the actual situation under the current model parameters, requiring parameter adjustment to reduce the loss.

[0198] S434. Optimize the parameters based on the loss value to generate updated model parameters. The formula for calculating the updated model parameters is:

[0199]

[0200] in, For the updated model parameters, Let be the model parameters for the t-th iteration. For adaptive learning rate, For first-order moment estimation, For second-order moment estimation, These are numerically stable parameters.

[0201] Optionally, parameter optimization adjusts model parameters based on the loss value to gradually reduce the model's prediction bias. An adaptive optimization algorithm is used to achieve efficient parameter updates, and the core calculation formula is: ,

[0202] in, For the updated model parameters, such as the weights of the feature extraction layer and the biases of the mapping layer, For the first Model parameters at the next iteration; To achieve adaptive learning rate, it is dynamically adjusted during the iteration process to balance the step size of parameter updates, avoiding oscillations caused by excessively large step sizes or slow convergence caused by excessively small step sizes. It is a first-order moment estimate, reflecting the first-order momentum of the parameter gradient, and smoothing the gradient change trend; It is a second-order moment estimate, reflecting the second-order momentum of the parameter gradient, and adaptively adjusts the update step size for different parameters; To ensure numerical stability and avoid zero denominators, the parameters are typically set to their minimum values. Using this formula, the gradients calculated from the loss values ​​are used to update the first and second moments, resulting in updated model parameters and achieving adaptive optimization of the model parameters.

[0203] S435. Smooth the updated model parameters to generate an intermediate model.

[0204] Optionally, parameter smoothing is used to avoid model fluctuations caused by single parameter updates and improve model stability. Using the updated model parameters as input, a moving average strategy is employed: the currently updated parameters are merged with historical smoothed parameters according to preset weights, such as by setting a smoothing coefficient. Then the smoothed parameters are: ,

[0205] in, These are the historical parameters after the previous smoothing. Smoothing reduces the impact of single-batch training on the parameters, making parameter changes more gradual. Finally, the smoothed parameters are loaded into the model structure to generate an intermediate model.

[0206] In the above embodiments, the enhanced features are divided into mini-batches, prediction results are obtained through forward propagation, loss values ​​are calculated, parameters are updated using the formula for the adaptive learning rate, and finally, the parameters are smoothed to obtain an intermediate model. This solves the problems of large fluctuations and slow convergence in traditional parameter updates. The adaptive learning rate can adjust the update step size according to changes in loss, and parameter smoothing can avoid the influence of extreme values, making parameter updates more stable and efficient, and ensuring that the performance of the intermediate model meets expectations.

[0207] The aforementioned method, system, equipment, and medium for processing business data establish a processing scheme where each step supports the others and is optimized layer by layer. This scheme involves breaking down the semantic drift detection steps, optimizing the distribution difference calculation logic, introducing knowledge graphs to assist semantic alignment, accurately calculating concept similarity, using incremental training to optimize the model, and stably updating model parameters. This technical solution can efficiently process multi-source, heterogeneous, cross-domain economic data, achieving data standardization and semantic unification. It also solves the problems of lagging rules, slow model updates, and large calculation biases inherent in traditional methods, making data processing both real-time and accurate, conforming to business logic, and highly efficient and energy-saving, thus meeting the core requirements for data quality and processing efficiency in economic data analysis.

[0208] It should be understood that although the steps in the flowcharts of the embodiments described above are shown sequentially according to the arrows, these steps are not necessarily executed in the order indicated by the arrows. Unless explicitly stated herein, there is no strict order restriction on the execution of these steps, and they can be executed in other orders. Moreover, at least some steps in the flowcharts of the embodiments described above may include multiple steps or multiple stages. These steps or stages are not necessarily completed at the same time, but can be executed at different times. The execution order of these steps or stages is not necessarily sequential, but can be performed alternately or in turn with other steps or at least some of the steps or stages of other steps.

[0209] Based on the same inventive concept, this application also provides a system for implementing the above-described method for processing business data. The solution provided by this system is similar to the implementation described in the above method; therefore, the specific limitations in one or more embodiments of the business data processing system provided below can be found in the limitations of the business data processing method described above, and will not be repeated here.

[0210] In one exemplary embodiment, such as Figure 3 As shown, a business data processing system 10 is provided, including:

[0211] The data preprocessing module 11 is used to preprocess cross-domain economic data from multiple heterogeneous data sources to generate standardized economic data streams.

[0212] The semantic drift detection module 12 is used to perform semantic drift detection on the standardized economic data stream and generate drift detection results.

[0213] The semantic alignment judgment module 13 is used to perform semantic alignment judgment based on the drift detection result and generate a dynamic alignment signal.

[0214] The mapping model incremental training module 14 is used to perform incremental training of the mapping model based on the dynamic alignment signal to generate an updated cross-domain mapping model.

[0215] The mapping and transformation processing module 15 is used to perform mapping and transformation processing on standardized economic data streams based on the updated cross-domain mapping model, and generate semantically unified business data.

[0216] In one embodiment, the semantic drift detection module includes:

[0217] The time window partitioning unit is used to perform time window partitioning on the standardized economic data stream and generate the current window dataset.

[0218] The statistical feature extraction unit is used to extract statistical features from the current window dataset and generate the current data distribution features;

[0219] The reference distribution modeling unit is used to perform reference distribution modeling on historical economic data to generate a baseline data distribution.

[0220] The difference calculation unit is used to calculate the difference between the current data distribution characteristics and the benchmark data distribution, and generate a distribution difference value.

[0221] The threshold judgment unit is used to perform threshold judgment processing based on the distribution difference value and the preset drift judgment conditions to generate drift detection results.

[0222] In one embodiment, the difference calculation unit includes:

[0223] The probability density estimation subunit is used to estimate the probability density of the current data distribution characteristics and the baseline data distribution, and generate a first probability distribution and a second probability distribution.

[0224] The JS divergence calculation subunit is used to perform JS divergence calculation on the first probability distribution and the second probability distribution to generate an initial divergence value; the formula for calculating the initial divergence value is:

[0225]

[0226] in, Let P be the initial difference value, Q be the first probability distribution, and M be the mean distribution of P and Q. Here is the function for calculating KL divergence;

[0227] The cumulative distribution function transformation subunit is used to transform the first probability distribution and the second probability distribution using the cumulative distribution function to generate the first cumulative distribution function and the second cumulative distribution function;

[0228] The KS test calculation subunit is used to perform KS test calculations on the first cumulative distribution function and the second cumulative distribution function to generate the distribution deviation value.

[0229] The weighted fusion subunit is used to perform weighted fusion processing on the distribution difference value and the distribution deviation value to generate the distribution difference value.

[0230] In one embodiment, the semantic alignment determination module includes:

[0231] The knowledge graph query unit is used to generate a concept association graph based on the economic indicators in the drift detection results through knowledge graph query processing.

[0232] The graph embedding processing unit is used to perform graph embedding processing on the concept association graph to generate concept vector representations;

[0233] The similarity calculation unit is used to calculate the similarity of concept vector representations and generate a semantic similarity matrix;

[0234] The alignment decision judgment unit is used to compare and judge the values ​​of each element in the semantic similarity matrix with a preset similarity threshold to generate a preliminary alignment decision;

[0235] The signal generation condition judgment unit is used to perform signal generation condition judgment processing on the preliminary alignment decision according to predefined business rules, so as to obtain the dynamic alignment signal.

[0236] In one embodiment, the similarity calculation unit includes:

[0237] The vector normalization subunit is used to perform vector normalization on the concept vector representation to generate standardized vectors;

[0238] An improved similarity calculation subunit is used to perform improved similarity calculations on standardized vectors, generating concept similarity values; the formula for calculating concept similarity values ​​is:

[0239]

[0240] in, This represents the concept similarity value. Let i be the vector of the i-th concept. Let j be the vector of the concept. This is a distance weighting parameter. For interaction weight adjustment parameters, This represents the Hadamard product operator. Let L2 be the norm of the vector;

[0241] The matrix construction subunit is used to construct a matrix from the concept similarity values ​​to generate an initial similarity matrix;

[0242] The domain knowledge correction subunit is used to perform domain knowledge correction on the initial similarity matrix and generate a corrected similarity matrix;

[0243] The matrix sparsification sub-unit is used to sparsify the calibration similarity matrix to generate a semantic similarity matrix.

[0244] In one embodiment, the mapping model incremental training module includes:

[0245] The data acquisition unit is used to acquire data from the training sample library based on the dynamic alignment signal and generate an incremental training set.

[0246] The feature enhancement processing unit is used to perform feature enhancement processing on the incremental training set to generate enhanced feature representations;

[0247] The model parameter update unit is used to update the model parameters of the enhanced feature representation and generate an intermediate model.

[0248] The performance evaluation unit is used to evaluate the performance of the intermediate model and generate model performance metrics.

[0249] The early stopping judgment unit is used to perform early stopping judgment processing based on model performance indicators and generate an updated cross-domain mapping model.

[0250] In one embodiment, the model parameter update unit includes:

[0251] Mini-batch partitioning is used to partition the enhanced feature representation into mini-batch units to generate training batch data.

[0252] The forward propagation computation subunit is used to perform forward propagation computation on the training batch data and generate prediction results;

[0253] The loss calculation subunit is used to calculate the loss from the prediction results and generate the loss value.

[0254] The parameter optimization subunit is used to optimize the parameters based on the loss value, generating updated model parameters. The formula for calculating the updated model parameters is as follows:

[0255]

[0256] in, For the updated model parameters, Let be the model parameters for the t-th iteration. For adaptive learning rate, For first-order moment estimation, For second-order moment estimation, For numerical stability parameters;

[0257] The parameter smoothing subunit is used to smooth the updated model parameters and generate an intermediate model.

[0258] In one embodiment, a computer device is provided, including a memory and a processor, the memory storing a computer program, the processor executing the computer program to implement the steps of a business data processing method as described above.

[0259] In one embodiment, a computer-readable storage medium is provided having a computer program stored thereon, which, when executed by a processor, implements the steps of a business data processing method as described above.

[0260] For the device embodiments, since they basically correspond to the method embodiments, the relevant parts can be referred to in the description of the method embodiments. The device embodiments described above are merely illustrative. The components described as separate parts may or may not be physically separate, and the components shown as units may or may not be physical units, that is, they may be located in one place or distributed across multiple network units. Some or all of the modules can be selected to achieve the purpose of this disclosure according to actual needs. Those skilled in the art can understand and implement this without creative effort.

[0261] The above-described embodiments are merely illustrative of several implementation methods of the embodiments of this application, and their descriptions are relatively specific and detailed. However, they should not be construed as limiting the scope of the patent application. It should be noted that those skilled in the art can make various modifications and improvements without departing from the concept of the embodiments of this application, and these modifications and improvements all fall within the protection scope of the embodiments of this application.

Claims

1. A method for processing business data, characterized in that, The method includes: S1. Preprocess cross-domain economic data from multiple heterogeneous data sources to generate standardized economic data streams; S2. Perform semantic drift detection on the standardized economic data stream and generate drift detection results; S3. Based on the drift detection results, perform semantic alignment judgment and generate a dynamic alignment signal; S4. Based on the dynamic alignment signal, perform incremental training of the mapping model to generate an updated cross-domain mapping model; S5. Based on the updated cross-domain mapping model, perform mapping and transformation processing on the standardized economic data stream to generate semantically unified business data.

2. The method according to claim 1, characterized in that, S2 includes: S21. Perform time window segmentation processing on the standardized economic data stream to generate the current window dataset; S22. Statistical feature extraction is performed on the current window dataset to generate current data distribution features; S23. Perform reference distribution modeling on historical economic data to generate a baseline data distribution; S24. Calculate the difference between the current data distribution characteristics and the baseline data distribution to generate a distribution difference value; S25. Perform threshold judgment processing based on the distribution difference value and the preset drift judgment condition to generate the drift detection result.

3. The method according to claim 2, characterized in that, S24 includes: S241. Estimate the probability density of the current data distribution characteristics and the baseline data distribution to generate a first probability distribution and a second probability distribution; S242. Perform JS divergence calculation on the first probability distribution and the second probability distribution to generate an initial dissimilarity value; the formula for calculating the initial dissimilarity value is: in, Let P be the initial difference value, Q be the first probability distribution, and M be the mean distribution of P and Q. Here is the function for calculating KL divergence; S243. Perform a cumulative distribution function transformation on the first probability distribution and the second probability distribution to generate a first cumulative distribution function and a second cumulative distribution function; S244. Perform a KS test on the first and second cumulative distribution functions to generate the distribution deviation value; S245. Perform weighted fusion processing on the distribution difference value and the distribution deviation value to generate the distribution difference value.

4. The method according to claim 1, characterized in that, S3 includes: S31. Based on the economic indicators in the drift detection results, generate a concept association graph through knowledge graph query processing; S32. Perform graph embedding processing on the concept association graph to generate a concept vector representation; S33. Calculate the similarity of the concept vector representations to generate a semantic similarity matrix; S34. Based on the comparison and judgment of each element value in the semantic similarity matrix with the preset similarity threshold, a preliminary alignment decision is generated; S35. Using predefined business rules, perform signal generation condition judgment processing on the preliminary alignment decision to obtain the dynamic alignment signal.

5. The method according to claim 4, characterized in that, S33 includes: S331. Perform vector normalization on the concept vector representation to generate a standardized vector; S332. Improve the similarity calculation of the standardized vector to generate a concept similarity value; the formula for calculating the concept similarity value is: in, This represents the concept similarity value. Let i be the vector of the i-th concept. Let j be the vector of the concept. This is a distance weighting parameter. For interaction weight adjustment parameters, This represents the Hadamard product operator. Let L2 be the norm of the vector; S333. Construct a matrix from the concept similarity values ​​to generate an initial similarity matrix; S334. Perform domain knowledge correction on the initial similarity matrix to generate a corrected similarity matrix; S335. The corrected similarity matrix is ​​sparsified to generate the semantic similarity matrix.

6. The method according to claim 1, characterized in that, S4 includes: S41. Data is acquired from the training sample library based on the dynamic alignment signal to generate an incremental training set; S42. Perform feature enhancement processing on the incremental training set to generate enhanced feature representations; S43. Perform model parameter update processing on the enhanced feature representation to generate an intermediate model; S44. Evaluate the performance of the intermediate model and generate model performance indicators; S45. Based on the model performance indicators, perform early stopping judgment processing to generate the updated cross-domain mapping model.

7. The method according to claim 6, characterized in that, S43 includes: S431. Perform mini-batch partitioning on the enhanced feature representation to generate training batch data; S432. Perform forward propagation calculation on the training batch data to generate prediction results; S433. Calculate the loss based on the prediction result and generate a loss value; S434. Perform parameter optimization based on the loss value to generate updated model parameters. The formula for calculating the updated model parameters is as follows: in, For the updated model parameters, Let be the model parameters for the t-th iteration. For adaptive learning rate, For first-order moment estimation, For second-order moment estimation, For numerical stability parameters; S435. Smooth the updated model parameters to generate the intermediate model.

8. A business data processing system, characterized in that, The system includes: The data preprocessing module is used to preprocess cross-domain economic data from multiple heterogeneous data sources to generate standardized economic data streams. A semantic drift detection module is used to perform semantic drift detection on the standardized economic data stream and generate drift detection results. The semantic alignment judgment module is used to perform semantic alignment judgment based on the drift detection result and generate a dynamic alignment signal; The mapping model incremental training module is used to perform incremental training of the mapping model based on the dynamic alignment signal to generate an updated cross-domain mapping model. The mapping and transformation processing module is used to perform mapping and transformation processing on the standardized economic data stream based on the updated cross-domain mapping model, and generate semantically unified business data.

9. A computer device comprising a memory and a processor, wherein the memory stores a computer program, characterized in that, When the processor executes the computer program, it implements the method of any one of claims 1 to 7.

10. A computer-readable storage medium having a computer program stored thereon, characterized in that, When the computer program is executed by a processor, it implements the method of any one of claims 1 to 7.

Citation Information

Cited By

  • Industrial model fine tuning and reasoning method and device and electronic equipment

    CN121920544A

  • Big model data governance method based on cross-domain semantic drift

    CN122433747A