Missing data completion methods, devices, equipment and storage media

By employing generative adversarial networks and joint loss fine-tuning, the problem of missing data in logistics data is solved, achieving high-precision and logically consistent data completion, adapting to complex and ever-changing logistics business scenarios.

CN123068786APending Publication Date: 2026-11-10SHANGHAI DONGPU INFORMATION TECH CO LTD
View PDF 0 Cites 0 Cited by

Patent Information

Application Number
CN202611337730.X
Authority / Receiving Office
CN · China
Patent Type
Applications(China)
Current Assignee / Owner
Filing Date
2026-08-31
Publication Date
2026-11-10

AI Technical Summary

Technical Problem

There are data gaps in logistics data. Existing technologies struggle to capture non-linear change patterns, resulting in low completion accuracy, inconsistent field logic, insufficient generalization ability, and an inability to adapt to complex and ever-changing logistics business scenarios.

Method used

A missing data completion model is constructed using generative adversarial networks. Through two-stage pre-training and joint loss fine-tuning, combined with the loss of the feature layer and the business rule layer, the missing data completion model is optimized to achieve structured mapping of data and consistency of business logic.

Benefits of technology

It improves the accuracy of data completion, ensures the consistency of business logic, adapts to changing logistics business scenarios, and enhances the generalization ability of the model and the reliability of completion results.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure CN123068786A_ABST
    Figure CN123068786A_ABST
Patent Text Reader

Abstract

This invention relates to the field of data completion technology, and more particularly to a method, apparatus, device, and storage medium for missing data completion. The method first preprocesses the acquired raw multi-source heterogeneous data, then performs structured mapping on the preprocessed data to obtain a three-dimensional raw data tensor and a mask tensor, constructing an initial missing data completion model. Based on the three-dimensional raw data tensor and the mask tensor, the initial missing data completion model undergoes a two-stage pre-training process to obtain an optimized missing data completion model. Based on the optimized missing data completion model, feature layer loss and business rule layer loss are calculated. Based on the feature layer loss and business rule layer loss, the optimized missing data completion model undergoes joint loss fine-tuning to obtain a target missing data completion model. Data to be completed is then acquired and input into the target missing data completion model for missing value completion processing to obtain completed data. The aim is to improve data completion accuracy.
Need to check novelty before this filing date? Find Prior Art

Description

Technical Field

[0001] This invention relates to the field of data completion technology, and in particular to a method, apparatus, device, and storage medium for completing missing data. Background Technology

[0002] In logistics data processing scenarios, multi-source heterogeneous data commonly suffers from missing data, which directly impacts freight calculation, business auditing, and subsequent data analysis. Currently, common methods for missing data completion include traditional approaches such as mean imputation, linear interpolation, and simple statistical model prediction. However, these methods have significant limitations in practical applications. Traditional completion methods struggle to capture the complex non-linear patterns within logistics data. They often fail to effectively learn from seasonal fluctuations, sudden business changes during peak sales periods, and data feature variations caused by special transportation routes, ultimately resulting in low overall accuracy of the completed data. Furthermore, traditional solutions often perform independent completion for each business field, ignoring the strong coupling relationships between different fields within the logistics business. Business parameters such as weight, volume, transportation distance, and service level have business-level constraints. Multiple parameters participate in the calculation of basic freight costs, and independent completion can easily lead to internal logical contradictions in the completed data. The resulting data cannot support accurate billing and auditing work and fails to meet the actual needs of business operations. In addition, many existing completion models have limited adaptability to specific scenarios, and most can only adapt to fixed missing patterns. When faced with continuously changing business scenarios and new missing types during business iterations, the models lack generalization ability, making it difficult to achieve stable and reliable missing completion effects, and thus failing to adapt well to complex and ever-changing actual logistics business. Summary of the Invention

[0003] In order to overcome the shortcomings of the prior art, the present invention aims to provide a method, apparatus, device and storage medium for missing data completion, which aims to solve the technical problems of low accuracy of logistics data completion, inconsistent field logic and insufficient generalization ability, so as to improve the accuracy of data completion, ensure the consistency of business logic and adapt to the technical effects of changing business scenarios.

[0004] The first aspect of this invention provides a method for missing data completion, comprising: acquiring raw multi-source heterogeneous data; performing data preprocessing on the raw multi-source heterogeneous data to obtain preprocessed data; performing structured mapping processing on the preprocessed data to obtain a three-dimensional raw data tensor and a mask tensor; constructing an initial missing data completion model; performing two-stage pre-training processing on the initial missing data completion model based on the three-dimensional raw data tensor and the mask tensor to obtain an optimized missing data completion model; calculating feature layer loss and business rule layer loss based on the optimized missing data completion model; performing joint loss fine-tuning processing on the optimized missing data completion model based on the feature layer loss and the business rule layer loss to obtain a target missing data completion model; acquiring data to be completed; inputting the data to be completed into the target missing data completion model for missing value completion processing to obtain completed data.

[0005] Optionally, in a first implementation of the first aspect of the present invention, the step of preprocessing the original multi-source heterogeneous data to obtain preprocessed data includes: using a data cleaning algorithm to clean the original multi-source heterogeneous data to obtain cleaned data; using a format correction algorithm to correct the timestamp format of the cleaned data to obtain corrected data; and using a missing value identification algorithm to mark missing values ​​in the corrected data to obtain the preprocessed data.

[0006] Optionally, in a second implementation of the first aspect of the present invention, the step of performing structured mapping processing on the preprocessed data to obtain a three-dimensional original data tensor and a mask tensor includes: obtaining a preset time window; performing slicing processing on the time series in the preprocessed data based on the time window to obtain time-series slice data; performing multi-type field encoding embedding processing on the time-series slice data to obtain encoded time-series data; and generating the three-dimensional original data tensor and the mask tensor based on the encoded time-series data.

[0007] Optionally, in a third implementation of the first aspect of the present invention, the step of performing a two-stage pre-training process on the initial missing data completion model based on the three-dimensional original data tensor and the mask tensor to obtain an optimized missing data completion model includes: acquiring a historical dataset; performing field masking processing on the historical dataset according to a preset ratio to construct unsupervised training samples; performing a first-stage unsupervised pre-training process on the initial missing data completion model based on the unsupervised training samples, the three-dimensional original data tensor, and the mask tensor to alternately optimize the generator and discriminator of the initial missing data completion model, and calculating a first adversarial loss, a first reconstruction loss, and a first feature matching loss; iteratively updating the weight parameters of the initial missing data completion model based on the first adversarial loss, the first reconstruction loss, and the first feature matching loss; outputting a pre-trained intermediate model when a preset first iteration stopping condition is met; and performing a second-stage supervised fine-tuning process on the pre-trained intermediate model to obtain the optimized missing data completion model.

[0008] Optionally, in a fourth implementation of the first aspect of the present invention, the step of performing a second-stage supervised fine-tuning process on the pre-trained intermediate model to obtain the optimized missing data completion model includes: obtaining a supervised fine-tuning dataset carrying true missing labels; performing a second-stage supervised fine-tuning process on the pre-trained intermediate model based on the supervised fine-tuning dataset and calculating the supervised loss; iteratively updating the weight parameters of the pre-trained intermediate model based on the supervised loss; and outputting the optimized missing data completion model when a preset second iteration stopping condition is met.

[0009] Optionally, in a fifth implementation of the first aspect of the present invention, the step of calculating the feature layer loss and the business rule layer loss based on the optimized missing data completion model includes: obtaining simulated completed data output by the optimized missing data completion model and real complete data corresponding to the simulated completed data; obtaining a pre-trained autoencoder, performing weight freezing processing on the autoencoder to obtain a frozen encoder; inputting the simulated completed data and the real complete data into the frozen encoder for feature extraction processing to obtain simulated hidden layer feature vectors and real hidden layer feature vectors; performing feature vector distance metric calculation processing on the simulated hidden layer feature vectors and the real hidden layer feature vectors to obtain the feature layer loss; constructing a differentiable billing rule calculation graph, inputting the simulated completed data into the differentiable billing rule calculation graph for billing logic forward deduction processing to obtain at least one predicted billing result; obtaining a standard billing result corresponding to the predicted billing result; performing billing result difference metric calculation processing on the predicted billing result and the standard billing result to obtain the business rule layer loss.

[0010] Optionally, in a sixth implementation of the first aspect of the present invention, the step of performing joint loss fine-tuning on the optimized missing data completion model based on the feature layer loss and the business rule layer loss to obtain the target missing data completion model includes: obtaining joint fine-tuning samples; performing forward inference processing on the optimized missing data completion model based on the joint fine-tuning samples to calculate a second adversarial loss, a second reconstruction loss, and a second feature matching loss; performing weighted summation processing on the second adversarial loss, the second reconstruction loss, the second feature matching loss, the feature layer loss, and the business rule layer loss to obtain a joint total loss; iteratively updating the weight parameters of the optimized missing data completion model using a backpropagation algorithm based on the joint total loss; and outputting the target missing data completion model when a preset joint fine-tuning stopping condition is met.

[0011] A second aspect of the present invention provides a missing data completion device, comprising: a data preprocessing module for acquiring raw multi-source heterogeneous data and performing data preprocessing on the raw multi-source heterogeneous data to obtain preprocessed data; a structured mapping module for performing structured mapping processing on the preprocessed data to obtain a three-dimensional raw data tensor and a mask tensor; a two-stage pre-training module for constructing an initial missing data completion model and performing two-stage pre-training processing on the initial missing data completion model based on the three-dimensional raw data tensor and the mask tensor to obtain an optimized missing data completion model; a loss calculation module for calculating feature layer loss and business rule layer loss based on the optimized missing data completion model; a joint loss fine-tuning module for performing joint loss fine-tuning processing on the optimized missing data completion model based on the feature layer loss and the business rule layer loss to obtain a target missing data completion model; and a missing value completion module for acquiring data to be completed and inputting the data to be completed into the target missing data completion model for missing value completion processing to obtain completed data.

[0012] A third aspect of the present invention provides a missing data completion device, the missing data completion device comprising: a memory and at least one processor, the memory storing instructions; the at least one processor calling the instructions in the memory to cause the missing data completion device to perform the steps of the missing data completion method described in any of the preceding claims.

[0013] A fourth aspect of the present invention provides a computer-readable storage medium storing instructions that, when executed by a processor, implement the steps of the missing data completion method described in any of the preceding claims.

[0014] In the technical solution of this invention, firstly, raw multi-source heterogeneous data is acquired, and preprocessed to obtain preprocessed data. Then, structured mapping is performed on the preprocessed data to obtain a three-dimensional raw data tensor and a mask tensor. Next, an initial missing data completion model is constructed. Based on the three-dimensional raw data tensor and the mask tensor, the initial missing data completion model is pre-trained in two stages to obtain an optimized missing data completion model. Based on the optimized missing data completion model, feature layer loss and business rule layer loss are calculated. Based on the feature layer loss and business rule layer loss, the optimized missing data completion model is fine-tuned with joint loss to obtain a target missing data completion model. Finally, the data to be completed is acquired and input into the target missing data completion model for missing value completion processing to obtain completed data. This invention aims to solve the technical problems of low accuracy, inconsistent field logic, and insufficient generalization ability in logistics data completion, thereby improving data completion accuracy, ensuring business logic consistency, and adapting to diverse business scenarios. Attached Figure Description

[0015] Figure 1 This is a first flowchart of the missing data completion method provided in an embodiment of the present invention; Figure 2 This is a second flowchart of the missing data completion method provided in the embodiments of the present invention; Figure 3 This is a third flowchart of the missing data completion method provided in the embodiments of the present invention; Figure 4 This is a fourth flowchart of the missing data completion method provided in the embodiments of the present invention; Figure 5 This is a fifth flowchart of the missing data completion method provided in the embodiments of the present invention; Figure 6 This is a sixth flowchart of the missing data completion method provided in the embodiments of the present invention; Figure 7 The seventh flowchart of the missing data completion method provided in the embodiments of the present invention; Figure 8 This is a schematic diagram of the missing data completion device provided in an embodiment of the present invention; Figure 9 This is a schematic diagram of the missing data completion device provided in an embodiment of the present invention. Detailed Implementation

[0016] This invention provides a method, apparatus, device, and storage medium for completing missing data. In this invention, the terms "first," "second," "third," "fourth," etc. (if present)," in the specification, claims, and accompanying drawings are used to distinguish similar objects and are not necessarily used to describe a specific order or sequence. It should be understood that such data can be interchanged where appropriate so that the embodiments described herein can be implemented in orders other than those illustrated or described herein. Furthermore, the terms "comprising" or "having," and any variations thereof, are intended to cover non-exclusive inclusion; for example, a process, method, system, product, or device that includes a series of steps or units is not necessarily limited to those steps or units explicitly listed, but may include other steps or units not explicitly listed or inherent to such processes, methods, products, or devices.

[0017] For ease of understanding, the specific process of the embodiments of the present invention is described below. Please refer to [link / reference]. Figure 1 One embodiment of the missing data completion method in this invention includes: 101. Obtain raw multi-source heterogeneous data, and perform data preprocessing on the raw multi-source heterogeneous data to obtain preprocessed data; In this embodiment, the original multi-source heterogeneous data comes from business data generated by multiple business systems such as transportation management, warehousing management, positioning, electronic waybills, and financial settlement. Data acquisition is completed by connecting to the data source interfaces of each business system. Data preprocessing is carried out on the acquired original multi-source heterogeneous data. First, data cleaning is performed to remove outliers and duplicate records to obtain cleaned data. Then, the timestamp format of the cleaned data is corrected to unify the timestamp expression form to obtain corrected data. Subsequently, missing data identification processing is performed on the corrected data to identify and mark missing values. Finally, preprocessed data is output. This preprocessing process can complete the normalization of multi-source business data, filter out invalid noise in the data, unify the time dimension expression, and mark and retain missing information. This provides a reliable input foundation for subsequent structured mapping and model training, reduces the interference of dirty data on the subsequent model learning process, and ensures the reliability of the data foundation for subsequent missing data completion tasks.

[0018] 102. Perform structured mapping processing on the preprocessed data to obtain the three-dimensional original data tensor and mask tensor; In this embodiment, the structured mapping process first reads a preset time window and then performs slicing processing on the time series within the preprocessed data based on this time window. The entire lifecycle business data of the same waybill is aligned along the time axis and divided into multiple continuous time-series slices according to the window granularity. For time slices with missing data, the corresponding fields are marked as needing completion. Subsequently, multi-type field encoding and embedding processing is performed on the time-series slice data. Standardization, vector encoding, periodic feature extraction, and semantic vector conversion are performed on different types of fields to complete the mapping of various heterogeneous fields to a model-recognizable vector space. Simultaneously, field relationships are constructed using a business knowledge base, and a three-dimensional original data tensor and a mask tensor are generated based on the encoded time-series data. The three-dimensional original data tensor is a structured tensor carrying the three dimensions of batch size, number of time slices, and total number of fields, used to store all business feature information after encoding and conversion. The mask tensor is a binary tensor matching the dimensions of the three-dimensional original data tensor. The values ​​within the tensor are used to distinguish whether the corresponding field is a true observation or a missing field to be completed, serving as an important input for subsequent model training and completion calculations. This process converts unstructured logistics business data into a standardized tensor form that the model can directly read. It takes into account both time dimension information and business dependencies between fields, and completes a unified representation of different types of heterogeneous fields. The mask tensor can accurately distinguish between observed data and missing locations, avoid interference from missing samples on model parameter learning, and ensure that the model can fully explore the temporal change patterns and business relationships between fields, providing a standardized and complete input carrier for subsequent model pre-training.

[0019] 103. Construct an initial missing data completion model. Based on the three-dimensional original data tensor and the mask tensor, perform a two-stage pre-training process on the initial missing data completion model to obtain an optimized missing data completion model. In this embodiment, the initial missing data completion model is built based on a generative adversarial network. The model contains two main network modules: a generator and a discriminator. The generator is constructed using convolutional gated recurrent units and consists of an encoder and a decoder. The encoder receives the original data with spliced ​​random noise and a mask tensor. It extracts contextual information of the time and feature dimensions layer by layer using multiple one-dimensional convolutions and convolutional gated recurrent units, and compresses it into a low-dimensional latent space vector. The decoder uses this latent space vector as input and completes data reconstruction with the help of deconvolution and convolutional gated recurrent units, outputting virtual complete data with matching dimensions. The discriminator uses a convolutional neural network and outputs the probability that the sample comes from the real data distribution after receiving the complete sample. The two-stage pre-training process relies on the 3D raw data tensor and the mask tensor. First, a historical dataset is acquired, and fields within the dataset are masked according to a preset ratio to simulate data loss scenarios in real-world business scenarios and construct unsupervised training samples. These unsupervised training samples, the 3D raw data tensor, and the mask tensor are then input into the initial missing data completion model for the first stage of unsupervised pre-training. Parameter optimization is performed alternately on the generator and discriminator, and the first adversarial loss, first reconstruction loss, and first feature matching loss are calculated simultaneously. The first adversarial loss drives the generator to produce data that closely approximates the real distribution; the first reconstruction loss constrains the generated results to not modify existing real observation fields; and the first feature matching loss prompts the generated samples to closely approximate real samples. Maintaining consistency at the high-level feature distribution level, the model weight parameters are iteratively updated based on the three types of losses mentioned above until the first iteration stopping condition is met, outputting the pre-trained intermediate model. Subsequently, a supervised fine-tuning dataset carrying true missing labels is obtained. This dataset is used to perform the second stage of supervised fine-tuning on the pre-trained intermediate model, calculating the supervised loss. The supervised loss calculates the deviation between the generated result and the true value for the missing field. The model weight parameters are iteratively updated based on the supervised loss. When the second iteration stopping condition is met, an optimized missing data completion model is output. This optimized missing data completion model is a generative adversarial network model that has achieved parameter convergence after two stages of pre-training. It has learned the distribution patterns of logistics business data and possesses basic data missing completion capabilities. This two-stage pre-training process fully leverages the data patterns contained in massive historical data through the unsupervised stage, completes the model's basic capability learning with manually constructed missing samples, and then conducts supervised fine-tuning using a small dataset with true labels. This guides the model to focus on the task of completing missing fields, reducing the model's dependence on the number of labeled samples, alleviating the instability problem of generative adversarial network training, improving the model's learning effect on temporal dependencies and field association features, and providing a basic model foundation for subsequent joint loss fine-tuning.

[0020] 104. Calculate the feature layer loss and business rule layer loss based on the optimized missing data completion model; In this embodiment, when calculating the feature layer loss and business rule layer loss based on the optimized missing data completion model, the simulated completed data output by the optimized missing data completion model and the corresponding real complete data are first obtained. The autoencoder, which has been trained unsupervised on a large amount of historical complete data, is called. The autoencoder is subjected to weight freezing to obtain a frozen encoder. The simulated completed data and the real complete data are then fed into the frozen encoder to complete deep feature extraction. The simulated hidden layer feature vector and the real hidden layer feature vector are output respectively. The distance metric is calculated on the two sets of hidden layer feature vectors to obtain the feature layer loss. The feature layer loss is used to characterize the deviation between the simulated completed data and the real complete data in the latent feature space, driving the generated result to closely approximate the distribution of real data at the high-level semantic level, and is not limited to the matching of surface field values. Subsequently, a differentiable billing rule calculation graph is constructed, transforming all internal tiered pricing, volumetric weight conversion, service level coefficients, and various surcharge calculation logics within the logistics business into differentiable operations. Simulation completion data is input into the differentiable billing rule calculation graph to perform forward deduction of billing logic, outputting one or more predicted billing results. Then, standard billing results matching the predicted billing results are obtained from historical business data. A difference measurement calculation is performed on the predicted billing results and standard billing results to obtain the business rule layer loss. The business rule layer loss represents the deviation between the billing results derived from the completion fields and the business standard billing results. Based on this loss constraint, the business logic that conforms to logistics billing is completed and output. This calculation process establishes dual constraints from two dimensions: feature semantics and business rules. The feature layer loss mines the deep distribution patterns of the data and suppresses the false results generated by fitting only field values. The business rule layer loss embeds business domain knowledge into the model loss calculation process, connecting the completed fields with the actual billing logic. This ensures that the model completion output is not only numerically reasonable but also meets the constraints of the actual business. It effectively improves the problem of internal logical contradictions in the completed data and enhances the usability of the completion results for billing and auditing business scenarios.

[0021] 105. Based on the feature layer loss and the business rule layer loss, perform joint loss fine-tuning on the optimized missing data completion model to obtain the target missing data completion model; In this embodiment, when performing joint loss fine-tuning on the optimized missing data completion model based on feature layer loss and business rule layer loss to obtain the target missing data completion model, firstly, joint fine-tuning samples are obtained. The joint fine-tuning samples are used to drive the optimization of the missing data completion model to complete forward inference and calculate the second adversarial loss, the second reconstruction loss, and the second feature matching loss. Then, the second adversarial loss, the second reconstruction loss, the second feature matching loss, the feature layer loss, and the business rule layer loss are weighted and summed to generate the joint total loss. The weights corresponding to each loss are adjustable hyperparameters, and parameter tuning can be completed through experiments. The weights corresponding to the business rule layer loss are usually set to higher values ​​to ensure the priority constraint status of business logic in the model optimization process. Based on the joint total loss, the backpropagation algorithm is used to iteratively update the weight parameters of the missing data completion model. The business rule layer loss can participate in gradient backpropagation, allowing gradient information to be propagated back to the model, continuously correcting the model parameters until the training process reaches the preset joint fine-tuning stopping condition. After the training process terminates, the target missing data completion model is output. The target missing data completion model is the final missing data completion model with business logic constraints after joint fine-tuning of multi-dimensional losses. This tuning process integrates adversarial constraints, reconstruction constraints, feature matching constraints, feature layer consistency constraints, and business rule constraints. It no longer relies solely on single-dimensional numerical fitting for model training. Gradient backpropagation can simultaneously drive the model to consider the restoration effect of sample values, alignment of deep feature distribution, and compliance with business rules. This ensures that the completed data output by the model is not only realistic at the field numerical level but also closely matches the real data in terms of high-level feature distribution. Moreover, after business logic deduction, the generated completed fields can output calculation results that conform to the actual business, effectively mitigating the defect of ordinary generated model completion results that seem reasonable but violate business rules, and improving the reliability and application value of missing data completion results in actual business scenarios.

[0022] 106. Obtain the data to be completed, input the data to be completed into the target missing data completion model to complete the missing values, and obtain the completed data.

[0023] In this embodiment, after acquiring the data to be completed, the data to be completed in real time is first preprocessed and structured mapped to convert it into tensors and missing masks that can be recognized by the model. The processed tensors and missing masks are then fed into the target missing data completion model to perform forward inference operations. The model generates a complete hypothetical data tensor. The system uses the missing mask to complete field filtering and extracts the generated fields corresponding to the missing positions in the hypothetical data tensor. The original non-missing fields retain their original observation values. The two are then fused to obtain the completed data.

[0024] After obtaining the completed data, a confidence assessment is required. The completed data is fed into a pre-trained discriminator, which outputs a probability value representing the data's authenticity. Simultaneously, the consistency loss at the business rule layer is calculated. The probability value and the reciprocal of the consistency loss at the business rule layer are weighted according to a set fusion weight to output a comprehensive confidence score. Based on the confidence score, a tiered handling logic is executed. When the confidence score is higher than 0.9, the completed data is directly pushed to the downstream billing audit and data analysis system for automatic application. When the confidence score is between 0.7 and 0.9, the completed data is pushed to business personnel as a reference suggestion for manual review and confirmation. When the confidence score is lower than 0.7, the system automatically generates a manual review work order, which is submitted to business experts for verification and correction. All completion results, confidence scores, and corresponding handling decisions are uniformly written to the audit log database to support business traceability and model iteration optimization. The system continuously collects correction samples generated during the manual review process and billing anomaly information from downstream business systems to build a feedback sample library. Feedback samples are extracted periodically and combined with some historical pre-training samples to construct an incremental fine-tuning dataset. The joint loss function is used to incrementally train the model, and regularization constraints are introduced during training to suppress catastrophic forgetting. After incremental training is completed, a new version model is generated. A / B testing is conducted, allocating a small portion of business traffic to the new version model. The performance of the new and old models is compared in key metrics such as average confidence level, proportion of manual reviews, and billing dispute rate. The new version model is only officially launched after all metrics show significant improvement. Simultaneously, the system continuously monitors the confidence level distribution of overall business data. When a significant drop in average confidence level or a rapid increase in the proportion of low-confidence samples occurs, an alarm is triggered, prompting operations personnel to intervene and analyze changes in business patterns. Model pre-training updates or adjustments to the business rule calculation graph are then performed as needed to ensure that online completion capabilities can continuously adapt to the dynamic evolution of the business. The entire solution provides end-to-end intelligent completion of missing waybill data in real-time production environments. It goes beyond simply outputting the completed values ​​and introduces a confidence quantification mechanism to measure the reliability of the completion results. Human and machine tasks are assigned according to the confidence level, ensuring business processing efficiency while avoiding the risk of unreliable completion results directly entering the business chain. Relying on feedback loops and incremental iteration mechanisms, it continuously mines real business samples, drives continuous model iteration and optimization, and combines online indicator monitoring and alarms to promptly perceive changes in business models. This effectively improves the stability and long-term adaptability of the missing data completion solution in logistics billing business scenarios.

[0025] Please see Figure 2 In this embodiment of the invention, the step of preprocessing the original multi-source heterogeneous data to obtain preprocessed data includes: 201. The original multi-source heterogeneous data is cleaned using a data cleaning algorithm to obtain cleaned data; 202. The cleaned data is corrected using a format correction algorithm to obtain corrected data; 203. The missing value is marked on the corrected data using a missing value identification algorithm to obtain the preprocessed data.

[0026] In this embodiment, a data cleaning algorithm is used to clean the original multi-source heterogeneous data to obtain cleaned data. The original multi-source heterogeneous data comes from multiple business systems such as transportation management, warehousing management, positioning, electronic waybills, and financial settlement. It is collected by the data source interface of each business system. The data cleaning algorithm can be implemented by using a statistical outlier detection algorithm and a duplicate record matching algorithm. The statistical outlier detection algorithm uses statistical thresholds to identify abnormal samples that obviously violate business physical constraints, such as negative weight or volume exceeding the physical limit. The duplicate record matching algorithm identifies waybill records that are completely duplicated at the business semantic level by comparing key fields. During the processing, all original multi-source heterogeneous data is traversed, outliers and duplicate records are filtered out, and invalid noise samples are removed. After filtering, cleaned data is output. The cleaned data is an intermediate dataset that has removed outliers and duplicate records and retains valid business information. Next, a format correction algorithm is used to correct the timestamp format of the cleaned data to obtain corrected data. This algorithm, implemented using a time format standardization parsing algorithm, supports parsing and converting time fields with various encodings and writing paradigms. The process involves reading various heterogeneous timestamp fields within the cleaned data, parsing them, and then uniformly converting them into a standard time expression format. This eliminates data inconsistencies caused by inconsistent time field formats output by different business systems, resulting in corrected data. The corrected data is an intermediate dataset that has undergone time field standardization and whose time dimension expression is fully aligned. Then, a missing value detection algorithm is used to mark missing values ​​in the corrected data to obtain preprocessed data. This algorithm, implemented using a null value and pattern matching detection algorithm, can identify various forms of missing data, such as empty placeholders and default business markers. During processing, all fields of the corrected data are traversed, identifying various missing conditions within each field, distinguishing the data type and missing occurrence pattern, and marking the identified missing locations to retain the information. Finally, the preprocessed data is output. The entire processing flow completes the standardization of business data from multiple sources, filters out invalid noise in the original data, unifies the time dimension representation of the full data, and retains all missing information. This provides a reliable input foundation for subsequent structured mapping and model training, reduces the interference of dirty data on the subsequent model learning process, and ensures the reliability of the data foundation for the missing data completion task from the data source level.

[0027] Please see Figure 3In this embodiment of the invention, the step of performing structured mapping processing on the preprocessed data to obtain a three-dimensional original data tensor and a mask tensor includes: 301. Obtain a preset time window, and slice the time series in the preprocessed data based on the time window to obtain time series slice data; 302. Perform multi-type field encoding embedding processing on the time-series slice data to obtain encoded time-series data; 303. Generate the three-dimensional original data tensor and the mask tensor based on the encoded time-series data.

[0028] In this embodiment, the preset time window is a fixed-duration time granularity unit, which can be set according to the logistics waybill business flow rate, data collection frequency, and business analysis needs. A common value is 30 minutes. When slicing the time series in the preprocessed data based on the time window, all business data belonging to the same waybill are first aligned along the time axis. Then, the complete life cycle of the waybill is divided into several continuous segments according to the fixed time window. For time segments without valid business data, all fields in the segment are marked as pending completion. After processing, time-series slice data is obtained. The time-series slice data is a time-series business dataset of waybills that is segmented and aligned according to the time window and marked with missing slices pending completion. When performing multi-type field encoding and embedding processing on time-series slice data, for numerical fields, max-min normalization or Z-score standardization is used to complete the numerical interval mapping. For categorical fields, one-hot encoding or learned embedding is used to convert them into dense vectors. For time-series fields, periodic features such as weekdays, holidays, and hours are extracted and encoded. For textual fields, a pre-trained word embedding model is called to generate fixed-length semantic vectors. In this way, different types of business fields are uniformly converted into vector expressions that can be recognized by the model. After this encoding and conversion, encoded time-series data is obtained. Encoded time-series data is an intermediate dataset that completes the vector representation of multi-category fields and retains the time-series dimension information. In the process of generating 3D raw data tensors and mask tensors based on coded time-series data, a field relationship graph is first built using a business knowledge base. Nodes correspond to billing-related business fields, and edges are used to depict the business dependency logic between fields. Then, the vector information of the coded time-series data is organized according to three dimensions: batch size, number of time slices, and total number of fields, and assembled to form a 3D raw data tensor. Simultaneously, a binary mask tensor that perfectly matches the dimensions of the 3D raw data tensor is constructed. The element values ​​inside the mask tensor are used to identify whether the corresponding field belongs to a true observation or a missing field to be filled. The entire processing flow transforms the raw logistics business data into a standardized tensor format that the model can directly read, fully preserving the time-series evolution information and the business dependency logic between fields, achieving a unified representation of heterogeneous business fields. The mask tensor can accurately divide the observed data and missing positions, reducing the negative impact of missing samples on model parameter learning, enabling the model to fully explore the time-series change patterns and field business relationships, and providing a standardized and complete input carrier for subsequent model pre-training.

[0029] Please see Figure 4 In this embodiment of the invention, the step of performing a two-stage pre-training process on the initial missing data completion model based on the three-dimensional original data tensor and the mask tensor to obtain an optimized missing data completion model includes: 401. Obtain historical datasets and perform field masking on the historical datasets according to a preset ratio to construct unsupervised training samples; 402. Based on the unsupervised training samples, the three-dimensional original data tensor, and the mask tensor, perform a first-stage unsupervised pre-training process on the initial missing data completion model to alternately optimize the generator and discriminator of the initial missing data completion model, and calculate the first adversarial loss, the first reconstruction loss, and the first feature matching loss. 403. The weight parameters of the initial missing data completion model are iteratively updated based on the first adversarial loss, the first reconstruction loss, and the first feature matching loss; 404. When the preset first iteration stopping condition is met, output the pre-trained intermediate model; 405. Perform a second-stage supervised fine-tuning process on the pre-trained intermediate model to obtain the optimized missing data completion model.

[0030] In this embodiment, the historical dataset is a massive historical complete logistics business dataset obtained through multi-source data access and cleaning, structured time-series slice alignment, and field encoding embedding. When performing field masking processing on the historical dataset according to a preset ratio, complete samples in the historical dataset are selected, and some fields inside the samples are randomly masked according to a preset ratio to artificially create the effect of missing fields. This simulates the random data missing phenomenon that occurs in the logistics business scenario. After the masking processing is completed, unsupervised training samples are obtained. The unsupervised training samples are logistics business samples that are artificially constructed with simulated missing fields and can be used for unsupervised mode training. The initial missing data completion model undergoes a first-stage unsupervised pre-training process based on unsupervised training samples, a 3D raw data tensor, and a mask tensor. During the alternating optimization of the generator and discriminator of the initial missing data completion model, which is a missing data completion model built on a generative adversarial network, the generator uses a convolutional gated recurrent unit (CGN) as its base network, consisting of an encoder and a decoder. The encoder receives raw data with concatenated random noise and a mask tensor, extracts temporal and feature dimension contextual information through multiple layers of 1D convolutions and CGNs, and compresses it to obtain a low-dimensional latent space vector. The decoder, based on this latent space vector, reconstructs virtual complete data with the same output dimension as the input through deconvolutions and CGNs. The discriminator is a convolutional neural network (CNN) that outputs a scalar probability after receiving complete samples, representing the likelihood that the sample originates from the real data distribution. During the training phase, the parameters of one network are alternately fixed while the parameters of the other network are updated. Unsupervised training samples, a 3D raw data tensor, and a mask tensor are fed into the model for forward propagation, simultaneously calculating the three types of losses.

[0031] The first adversarial loss is the fundamental loss of the generative adversarial network. The generator aims to increase the probability of misjudgment by the discriminator, while the discriminator aims to improve its ability to distinguish between real and generated samples. The formula for calculating the first reconstruction loss is as follows: , in, This represents the total number of non-missing fields observed. , These represent the sample index and the field index, respectively. This indicates the value at the corresponding position in the virtual complete data output by the generator. This indicates the value at the corresponding position in the mask matrix. This represents the value at the corresponding position of the real sample. This loss only calculates the reconstruction error for the fields that were originally observed. The constraint generator cannot modify the existing real observation values. The formula for calculating the first feature matching loss is: , in, This represents the feature output of a certain intermediate layer of the discriminator. This indicates that the generator outputs virtual complete data. Representing real and complete data, this loss constraint ensures that the generated samples and real samples maintain consistency at the high-level semantic feature distribution level, thus stabilizing training and improving the quality of generated data. When iteratively updating the weight parameters of the initial missing data completion model based on the first adversarial loss, first reconstruction loss, and first feature matching loss, the three types of losses are weighted and summed to obtain the total loss for the first stage. The gradient of the total loss is then backpropagated to each layer of the network via the backpropagation algorithm, updating the gradients of the weight parameters within the generator and discriminator respectively. This completes the training process of forward propagation, loss calculation, and backpropagation gradient update in a loop.

[0032] The preset first iteration stopping condition can be set based on the dynamic equilibrium state of the model's adversarial training, the iteration round threshold, and the loss convergence threshold. Termination conditions can be selected as follows: the generator and discriminator losses tend to stabilize; the network reaches the preset iteration round; and the total loss decreases to the set threshold. When the training process meets the first iteration stopping condition, a pre-trained intermediate model is output. This pre-trained intermediate model is a generative adversarial network model that has completed unsupervised pre-training, learned the general distribution pattern of logistics data, and possesses basic completion capabilities, but has not yet been fine-tuned with real missing samples. Then, the pre-trained intermediate model undergoes a second stage of supervised fine-tuning to obtain an optimized missing data completion model. The entire processing flow relies on a massive historical complete dataset to enable the model to learn the general temporal patterns and field relationships of logistics business data through unsupervised pre-training. It relies on manually simulating missing samples to complete the basic capability construction, and then uses a small amount of dataset with ground truth labels to carry out supervised fine-tuning. This guides the model to focus on the task of completing the real missing fields, effectively reducing the model's dependence on the size of labeled samples, alleviating the inherent defects of oscillation and instability in the training of generative adversarial networks, strengthening the model's ability to learn temporal dependencies and business field relationship features, and the output optimized missing data completion model can serve as the model base for further joint loss fine-tuning.

[0033] Please see Figure 5 In this embodiment of the invention, the second-stage supervised fine-tuning of the pre-trained intermediate model to obtain the optimized missing data completion model includes: 501. Obtain a supervised fine-tuning dataset carrying true missing labels, perform a second-stage supervised fine-tuning process on the pre-trained intermediate model based on the supervised fine-tuning dataset, and calculate the supervised loss; 502. Iteratively update the weight parameters of the pre-trained intermediate model based on the supervised loss; 503. When the preset second iteration stopping condition is met, output the optimized missing data completion model.

[0034] In this embodiment, the supervised fine-tuning dataset carrying true missing labels is a small-scale logistics dataset where samples simultaneously possess both actual missing business location markers and corresponding ground truth values ​​for the missing fields. This dataset is collected from actual business systems, filtering for fields that actually occurred during business operations. Furthermore, waybill samples with actual missing location values ​​can be obtained through business backtracking and business ledger verification. After sample collection and organization, the supervised fine-tuning dataset is formed. When performing the second stage of supervised fine-tuning on the pre-trained intermediate model based on the supervised fine-tuning dataset, samples from the supervised fine-tuning dataset are fed into the pre-trained intermediate model to complete forward inference. The model's internal generator outputs the completed virtual data. During this process, the supervised loss is calculated. The supervised loss specifically calculates the error between the generated completion result and the actual ground truth value for the truly missing fields. The corresponding calculation formula is: , in, This indicates that the generator outputs virtual complete data. This indicates the true value corresponding to the missing field. Represents the mask matrix, This function acts as a location filter, only including truly missing fields in the loss calculation. This loss does not constrain the originally complete observation fields; it only drives the model to optimize the completion effect of missing positions. When iteratively updating the weight parameters of the pre-trained intermediate model based on supervised loss, the supervised loss is weighted and fused with the model's original adversarial loss, reconstruction loss, and feature matching loss to obtain the total loss for the fine-tuning stage. The gradient of this total loss is then backpropagated to each layer of the network using the backpropagation algorithm to update the gradients of the network weights and bias parameters of the generator and discriminator. The training process of forward inference of samples, loss calculation, and backpropagation of gradient updates is executed iteratively. The preset second iteration stopping condition can be set according to the convergence degree of model loss, the maximum number of iterations, and the validation set completion index. Termination criteria can be selected as the validation set completion error converging to a threshold range, the number of training iterations reaching a preset upper limit, and the loss no longer significantly decreasing over multiple consecutive iterations. After the training process meets the second iteration stopping condition, the optimized missing data completion model is output. This fine-tuning stage relies on a small number of business samples with ground truth values. Based on the model foundation laid by unsupervised pre-training, further parameter convergence is performed. The supervised loss specifically focuses on the completion effect of the real missing fields, avoiding the model from simply maintaining the original observed fields and ignoring the learning of missing positions. This can fully leverage the learning results of unsupervised pre-training on massive data patterns, significantly reduce the model's dependence on large-scale labeled samples, further correct the model's completion output, strengthen the model's adaptability to real business missing scenarios, and provide a more reliable model foundation for subsequent model fine-tuning.

[0035] Please see Figure 6In this embodiment of the invention, the calculation of feature layer loss and business rule layer loss based on the optimized missing data completion model includes: 601. Obtain the simulation completion data output by the optimized missing data completion model and the real complete data corresponding to the simulation completion data; 602. Obtain the pre-trained autoencoder, and perform weight freezing on the autoencoder to obtain the frozen encoder; 603. Input the simulated completed data and the real complete data into the freeze encoder for feature extraction processing to obtain the simulated hidden layer feature vector and the real hidden layer feature vector; 604. Perform feature vector distance metric calculation on the simulated hidden layer feature vector and the real hidden layer feature vector to obtain the feature layer loss; 605. Construct a differentiable billing rule calculation graph, input the simulation completion data into the differentiable billing rule calculation graph for forward deduction of billing logic, and obtain at least one predicted billing result; 606. Obtain the standard billing result corresponding to the predicted billing result; 607. Perform billing result difference measurement calculation on the predicted billing result and the standard billing result to obtain the business rule layer loss.

[0036] In this embodiment, the optimized missing data completion model receives missing logistics samples marked with a mask. After forward computation by the model's internal generator, it outputs simulated completed data. The corresponding real complete data comes from the business dataset. The original state of this sample is a complete sample. During the training process, the missing data is simulated by a mask to provide a ground truth reference. The pre-trained autoencoder is a lightweight, independent network specifically designed to mine deep semantic features of logistics data. Its training process relies on a large amount of historical complete logistics data for unsupervised training. The encoder part performs dimensionality reduction and compression on the input data to obtain hidden layer features. Then, the decoder reconstructs the original input from the hidden layer features. The reconstruction error is used as the training objective to iteratively update all weights of the network. After training, the pre-trained autoencoder is obtained. When performing weight freezing on the autoencoder, the weights and bias parameters of all layers of the network are locked. Gradient updates are no longer performed on the network during the training phase. Only the forward inference capability of the network is retained. After weight freezing, a frozen encoder is obtained. The frozen encoder is only responsible for feature extraction. Its parameters are not affected by subsequent fine-tuning processes, ensuring that the feature extraction logic remains stable. When inputting simulated completed data and real complete data into the freeze encoder for feature extraction, the two sets of data are fed into the encoding branches of the freeze encoder respectively. After multi-layer network operations, dimensionality reduction and compression are completed, stripping away the surface field numerical information and mining deep latent patterns within the samples. Simulated hidden layer feature vectors and real hidden layer feature vectors are output. The simulated hidden layer feature vector is the vector representation of the simulated completed data mapped to the latent feature space, and the real hidden layer feature vector is the vector representation of the real complete data mapped to the latent feature space. A feature vector distance metric is calculated between the simulated and real hidden layer feature vectors. The L2 norm is used to calculate the distance between the two sets of vectors to obtain the feature layer loss. The calculation formula is as follows: , in, This represents the feature vector of the simulated hidden layer. This represents the true hidden feature vector. This loss measures the degree of deviation between the generated sample and the true sample in the latent feature space, and constrains the generated result to align with the distribution of the true data at a high-level semantic level.

[0037] When constructing a differentiable billing rule calculation graph, all business billing logic in logistics operations, such as tiered pricing, volumetric weight conversion, service level coefficients, and various surcharges, is transformed into differentiable mathematical operations. Business condition judgments and table lookup conversions are replaced with differentiable approximate operators, encapsulated into a calculation graph. This differentiable billing rule calculation graph is a business rule operation unit capable of participating in neural network gradient backpropagation, receiving completed business fields to complete billing derivation. When inputting simulation-completed data into the differentiable billing rule calculation graph for forward derivation of billing logic, relevant fields such as weight, volume, distance, and service level from the simulation-completed data are sent to the calculation graph. The graph sequentially executes operations such as volumetric weight conversion, maximizing the chargeable weight, matching the unit price, and accumulating freight costs. Forward propagation outputs at least one predicted billing result, which is the freight output derived from the fields obtained by simulation completion through business billing rule derivation. The standard billing result corresponding to the predicted billing result is obtained from historical business data. Historical bills with the same origin and destination, service level, and weight range as the current bill are selected. The average, median, or business contract benchmark price is used as the standard billing result, which serves as the benchmark billing reference value recognized by the business for the same type of bill. A billing result difference measurement calculation is performed between the predicted and standard billing results. The L1 loss is used to calculate the average absolute error of multiple billing items. The business rule layer loss calculation formula is as follows: , in, This indicates the total number of billing items, including basic freight, various surcharges, and other different billing items. Indicates the first The projected billing results, Indicates the first The standard billing results, with L1 loss exhibiting stronger robustness to outliers, are used to quantify the discrepancy between the billing results derived from the completed fields and the business benchmark billing. The entire process implements dual-layer consistency constraints at the feature layer and business rule layer. The feature layer loss constrains the completed data from the latent semantic space level, preventing the model from generating semantically spurious samples by merely fitting surface-level field values. The business rule layer loss embeds billing knowledge from the logistics domain into the loss calculation, bridging the constraint relationship between the completed fields and the business billing logic. This ensures that the completed data output by the model is not only numerically reasonable but also meets real business logic, effectively mitigating the shortcomings of traditional generative adversarial networks where the completed results contradict each other within the business logic. This significantly improves the practical usability of the completed data in billing and auditing business scenarios.

[0038] Please see Figure 7 In this embodiment of the invention, the joint loss fine-tuning of the optimized missing data completion model based on the feature layer loss and the business rule layer loss to obtain the target missing data completion model includes: 701. Obtain joint fine-tuning samples, and perform forward inference processing on the optimized missing data completion model based on the joint fine-tuning samples to calculate the second adversarial loss, the second reconstruction loss, and the second feature matching loss; 702. The second adversarial loss, the second reconstruction loss, the second feature matching loss, the feature layer loss, and the business rule layer loss are weighted and summed to obtain the joint total loss; 703. Based on the joint total loss, the weight parameters of the optimized missing data completion model are iteratively updated using the backpropagation algorithm; 704. When the preset joint fine-tuning stop condition is met, output the target missing data completion model.

[0039] In this embodiment, the joint fine-tuning sample is a dataset sample used for supervised fine-tuning in the second stage. It is obtained by masking the original complete logistics business samples to simulate field missingness. The sample simultaneously carries the missing input, mask label, and true complete value. This sample is selected from the business dataset and is used to support model training in the multi-loss joint fine-tuning stage. When performing forward inference processing on the optimized missing data completion model based on the joint fine-tuning sample, the joint fine-tuning sample with missing mask is input into the optimized missing data completion model. The model generator performs forward operation and outputs the completion result. The discriminator simultaneously distinguishes between the generated completed sample and the real sample, completing the forward inference of the entire network and providing intermediate variables such as generated samples, discriminant output, and reconstruction output for subsequent multi-class loss calculation. The second adversarial loss originates from the adversarial training branch and is used to measure the discriminator's ability to distinguish between generated and real samples, constraining the generator to produce simulated samples that can confuse the discriminator. The second reconstruction loss uses real complete samples as ground truth and measures the field reconstruction deviation between the model's completed output and the real samples, ensuring the fitting effect at the numerical level of the basic fields. The second feature matching loss performs matching calculations on the output features of the discriminator's intermediate layer, constraining the generated samples and real samples to align at the discriminator's feature level, mitigating the collapse problem of adversarial mode. The three types of losses are calculated and solved according to their respective loss calculation formulas, based on the generated samples, discriminator intermediate features, and ground truth samples obtained from forward inference. When performing a weighted summation of the second adversarial loss, second reconstruction loss, second feature matching loss, feature layer loss, and business rule layer loss, an independent and adjustable weight hyperparameter is assigned to each type of loss. The weight hyperparameter values ​​are tuned based on multiple sets of comparative experiments. The weight corresponding to the business rule layer loss is generally configured higher to strengthen the priority of business logic constraints in the overall optimization objective. After weighted summation, the joint total loss is obtained. The joint total loss is a comprehensive optimization objective that integrates adversarial constraints, reconstruction constraints, feature matching constraints, feature layer consistency constraints, and business rule constraints, integrating multi-dimensional model optimization guidance from numerical values, features, and business rules.

[0040] Based on the joint total loss, the backpropagation algorithm is used to iteratively update the weight parameters of the model for optimizing missing data completion. The loss of the business rule layer relies on the differentiable billing rule computation graph to achieve gradient backpropagation. The loss gradient can be propagated back along the computation graph to the generator network to solve the gradients corresponding to the parameters of each layer of the network. Then, the optimizer updates all trainable weights of the generator according to the set learning rate. In each iteration, it is simultaneously constrained by five types of loss, continuously correcting the model generation behavior, so that the model can simultaneously take into account the accuracy of field reconstruction, the realism of sample adversarial, the consistency of high-level features, and the compliance of business billing logic. The joint fine-tuning stopping condition is the basis for determining the termination of training. It can be set according to the number of training rounds, the convergence of the joint total loss of the validation set, and the performance of the validation set business indicators. It can be configured to stop when the preset maximum number of iterations is reached, or it can monitor the validation set loss and terminate training when the validation set loss no longer decreases for several consecutive rounds or the fluctuation enters a stable range. Alternatively, it can be combined with downstream business indicators and end fine-tuning when the compliance rate of the billing logic of the completed samples on the validation set reaches the business threshold. When any preset joint fine-tuning stopping condition is met, the training process ends and the target missing data completion model is output. This joint fine-tuning process integrates multi-dimensional constraints into a single optimization objective, moving beyond simply relying on field numerical fitting to drive model learning. Gradient backpropagation can simultaneously impose multiple constraints on the generator, including adversarial, reconstruction, feature, and business rule constraints. This ensures that the completed samples output by the model not only have field values ​​that closely resemble real business data, but also that the high-level feature distribution is aligned with the real samples. Furthermore, the billing results derived from the completed fields also conform to the logistics business billing rules. This effectively improves the problem of traditional generative missing value completion schemes where the numerical values ​​appear reasonable but the business logic is contradictory, thereby enhancing the reliability of the model in real business scenarios such as logistics billing and auditing.

[0041] The missing data completion method in the embodiments of the present invention has been described above. The missing data completion device in the embodiments of the present invention is described below. Please refer to [link / reference]. Figure 8 One embodiment of the missing data completion device in this invention includes: Data preprocessing module 801: used to acquire raw multi-source heterogeneous data, perform data preprocessing on the raw multi-source heterogeneous data, and obtain preprocessed data; Structured mapping module 802: used to perform structured mapping processing on the preprocessed data to obtain a three-dimensional original data tensor and a mask tensor; Two-stage pre-training module 803: used to construct an initial missing data completion model, and to perform two-stage pre-training on the initial missing data completion model based on the three-dimensional original data tensor and the mask tensor to obtain an optimized missing data completion model; Loss calculation module 804: used to calculate feature layer loss and business rule layer loss based on the optimized missing data completion model; Joint loss fine-tuning module 805: used to perform joint loss fine-tuning on the optimized missing data completion model based on the feature layer loss and the business rule layer loss, to obtain the target missing data completion model; Missing value completion module 806: used to obtain the data to be completed, input the data to be completed into the target missing data completion model to perform missing value completion processing, and obtain the completed data.

[0042] Based on the same ideas as the methods in the above embodiments, the apparatus provided in this application can implement the methods in the above embodiments.

[0043] above Figure 8 The missing data completion device in the embodiments of the present invention will be described in detail from the perspective of modular functional entities. The missing data completion device in the embodiments of the present invention will be described in detail from the perspective of hardware processing.

[0044] Figure 9 This is a schematic diagram of a missing data completion device 900 provided in an embodiment of the present invention. The missing data completion device 900 can vary significantly due to different configurations or performance. It may include one or more central processing units (CPUs) 910 (e.g., one or more processors) and a memory 920, and one or more storage media 930 (e.g., one or more mass storage devices) storing application programs 933 or data 932. The memory 920 and storage media 930 can be temporary or persistent storage. The program stored in the storage media 930 may include one or more modules (not shown in the diagram), each module including a series of instruction operations on the missing data completion device 900. Furthermore, the processor 910 may be configured to communicate with the storage media 930 and execute a series of instruction operations on the storage media 930 on the missing data completion device 900 to implement the steps of the missing data completion method provided in the above-described method embodiments.

[0045] The missing data completion device 900 may also include one or more power supplies 940, one or more wired or wireless network interfaces 950, one or more input / output interfaces 960, and / or one or more operating systems 931, such as Windows Server, Mac OS X, Unix, Linux, FreeBSD, etc. Those skilled in the art will understand that... Figure 9The missing data completion device structure shown does not constitute a limitation on the missing data completion device, and may include more or fewer components than shown, or combine certain components, or have different component arrangements.

[0046] The present invention also provides a computer-readable storage medium, which can be a non-volatile computer-readable storage medium or a volatile computer-readable storage medium, wherein the computer-readable storage medium stores instructions that, when executed on a computer, cause the computer to perform the steps of a missing data completion method.

[0047] Those skilled in the art will clearly understand that, for the sake of convenience and brevity, the specific working process of the system, device, or unit described above can be referred to the corresponding process in the foregoing method embodiments, and will not be repeated here.

[0048] If the integrated unit is implemented as a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present invention, in essence, or the part that contributes to the prior art, or all or part of the technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute all or part of the steps of the methods described in the various embodiments of the present invention. The aforementioned storage medium includes various media capable of storing program code, such as USB flash drives, portable hard drives, read-only memory (ROM), random access memory (RAM), magnetic disks, or optical disks.

[0049] Finally, it should be noted that the above descriptions are merely preferred embodiments of the present invention and are not intended to limit the present invention. Although the present invention has been described in detail with reference to the foregoing embodiments, those skilled in the art can still modify the technical solutions described in the foregoing embodiments or make equivalent substitutions for some of the technical features. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of the present invention should be included within the protection scope of the present invention.

Claims

1. A method for completing missing data, characterized in that, include: Obtain raw multi-source heterogeneous data, and perform data preprocessing on the raw multi-source heterogeneous data to obtain preprocessed data; The preprocessed data is subjected to structured mapping processing to obtain a three-dimensional original data tensor and a mask tensor; An initial missing data completion model is constructed, and a two-stage pre-training process is performed on the initial missing data completion model based on the three-dimensional original data tensor and the mask tensor to obtain an optimized missing data completion model. Based on the optimized missing data completion model, calculate the feature layer loss and the business rule layer loss; Based on the feature layer loss and the business rule layer loss, the optimized missing data completion model is subjected to joint loss fine-tuning to obtain the target missing data completion model. Obtain the data to be completed, input the data to be completed into the target missing data completion model to complete the missing values, and obtain the completed data.

2. The missing data completion method according to claim 1, characterized in that, The process of preprocessing the original multi-source heterogeneous data to obtain preprocessed data includes: The original multi-source heterogeneous data is cleaned using a data cleaning algorithm to obtain cleaned data. The cleaned data is corrected by applying a format correction algorithm to correct the timestamp format, resulting in corrected data. The missing value identification algorithm is used to mark the missing values ​​in the corrected data to obtain the preprocessed data.

3. The missing data completion method according to claim 1, characterized in that, The step of performing structured mapping processing on the preprocessed data to obtain a three-dimensional original data tensor and a mask tensor includes: A preset time window is obtained, and the time series in the preprocessed data is sliced ​​based on the time window to obtain time series slice data. The time-series slice data is subjected to multi-type field encoding embedding processing to obtain encoded time-series data; The three-dimensional original data tensor and the mask tensor are generated based on the encoded temporal data.

4. The missing data completion method according to claim 1, characterized in that, The process of performing a two-stage pre-training process on the initial missing data completion model based on the original 3D data tensor and the mask tensor to obtain an optimized missing data completion model includes: Obtain a historical dataset and perform field masking on the historical dataset according to a preset ratio to construct unsupervised training samples; The initial missing data completion model is subjected to a first-stage unsupervised pre-training process based on the unsupervised training samples, the three-dimensional original data tensor, and the mask tensor, so as to alternately optimize the generator and discriminator of the initial missing data completion model, and calculate the first adversarial loss, the first reconstruction loss, and the first feature matching loss. The weight parameters of the initial missing data completion model are iteratively updated based on the first adversarial loss, the first reconstruction loss, and the first feature matching loss. When the preset first iteration stopping condition is met, the pre-trained intermediate model is output; The pre-trained intermediate model is subjected to a second stage of supervised fine-tuning to obtain the optimized missing data completion model.

5. The missing data completion method according to claim 4, characterized in that, The second stage of supervised fine-tuning of the pre-trained intermediate model to obtain the optimized missing data completion model includes: Obtain a supervised fine-tuning dataset carrying true missing labels, perform a second-stage supervised fine-tuning process on the pre-trained intermediate model based on the supervised fine-tuning dataset, and calculate the supervised loss; The weight parameters of the pre-trained intermediate model are iteratively updated based on the supervised loss. When the preset second iteration stopping condition is met, the optimized missing data completion model is output.

6. The missing data completion method according to claim 1, characterized in that, The calculation of feature layer loss and business rule layer loss based on the optimized missing data completion model includes: Obtain the simulation-completed data output by the optimized missing data completion model and the real complete data corresponding to the simulation-completed data; Obtain a pre-trained autoencoder, and perform weight freezing on the autoencoder to obtain a frozen encoder; The simulated complete data and the real complete data are input into the frozen encoder for feature extraction processing to obtain the simulated hidden layer feature vector and the real hidden layer feature vector. The feature vector distance metric is calculated on the simulated hidden layer feature vector and the real hidden layer feature vector to obtain the feature layer loss. Construct a differentiable billing rule calculation graph, input the simulation completion data into the differentiable billing rule calculation graph for forward deduction of billing logic, and obtain at least one predicted billing result; Obtain the standard billing result corresponding to the predicted billing result; The predicted billing result and the standard billing result are processed by billing result difference measurement calculation to obtain the business rule layer loss.

7. The missing data completion method according to claim 1, characterized in that, The step of performing joint loss fine-tuning on the optimized missing data completion model based on the feature layer loss and the business rule layer loss to obtain the target missing data completion model includes: Obtain joint fine-tuning samples, and perform forward inference processing on the optimized missing data completion model based on the joint fine-tuning samples to calculate the second adversarial loss, the second reconstruction loss, and the second feature matching loss; The second adversarial loss, the second reconstruction loss, the second feature matching loss, the feature layer loss, and the business rule layer loss are weighted and summed to obtain the joint total loss; Based on the joint total loss, the weight parameters of the optimized missing data completion model are iteratively updated using the backpropagation algorithm; When the preset joint fine-tuning stopping condition is met, the target missing data completion model is output.

8. A missing data completion device, characterized in that, include: Data preprocessing module: used to acquire raw multi-source heterogeneous data, perform data preprocessing on the raw multi-source heterogeneous data, and obtain preprocessed data; Structured mapping module: used to perform structured mapping processing on the preprocessed data to obtain the three-dimensional original data tensor and mask tensor; Two-stage pre-training module: used to construct an initial missing data completion model, and to perform two-stage pre-training on the initial missing data completion model based on the three-dimensional original data tensor and the mask tensor to obtain an optimized missing data completion model; Loss calculation module: used to calculate feature layer loss and business rule layer loss based on the optimized missing data completion model; Joint loss fine-tuning module: used to perform joint loss fine-tuning on the optimized missing data completion model based on the feature layer loss and the business rule layer loss, to obtain the target missing data completion model; Missing value completion module: used to obtain the data to be completed, input the data to be completed into the target missing data completion model to perform missing value completion processing, and obtain the completed data.

9. A missing data completion device, characterized in that, The missing data completion device includes: a memory and at least one processor, wherein the memory stores instructions; At least one of the processors invokes the instructions in the memory to cause the missing data completion device to perform the steps of the missing data completion method as described in any one of claims 1-7.

10. A computer-readable storage medium storing instructions thereon, characterized in that, When the instructions are executed by the processor, they implement the steps of the missing data completion method as described in any one of claims 1-7.