Data processing model training method and device and data processing method and device

Abstract

Embodiments of the invention provide a data processing model training method and apparatus, and a data processing method and apparatus. The data processing model training method comprises the steps of determining a first conditional probability corresponding to first experimental data and a second conditional probability corresponding to second experimental data; training according to a target calibration weight calculated according to the first conditional probability and the second conditional probability to obtain a calibration estimator; inputting the target confusion data and the corresponding observation result into a calibration estimator to obtain a first calibration weight, and inputting the target confusion data and the estimation result into the calibration estimator to obtain a second calibration weight; processing the estimation result according to the two calibration weights to obtain a calibration result, inputting the target confusion data and the target intervention data into an initial data processing model to obtain a prediction result, and training according to the prediction result and the calibration result to obtain a target data processing model; the problem of selection deviation in survival analysis is solved by calibrating the estimator and adjusting the weight.

Description

Technical Field

[0001] This specification relates to the field of computer technology, and in particular to a data processing model training method. One or more embodiments of this specification also relate to a data processing model training device, a data processing method, a data processing apparatus, a computing device, a computer-readable storage medium, and a computer program product. Background Technology

[0002] During the experimental phase of the project, the experimental window limitation may cause positive samples that exceed the time limit to be lost or unobservable. That is, after the experiment, we will obtain uncensored data and censored data, while censored data outside the time window may complete the endpoint event.

[0003] In survival analysis (used to analyze survival time data, i.e. the time from a specific point in time to a certain endpoint event), directly using uncensored data for data analysis can lead to selection bias. That is, when censored data is present, the collected uncensored data cannot represent the overall research population of the project experiment. Directly using these uncensored data for analysis will result in biased estimations and inaccurate analysis results. Summary of the Invention

[0004] In view of the above, embodiments of this specification provide a data processing model training method. One or more embodiments of this specification also relate to a data processing model training device, a data processing method, a data processing apparatus, a computing device, a computer-readable storage medium, and a computer program product.

[0005] According to a first aspect of the embodiments of this specification, a data processing model training method is provided, comprising: Determine the first conditional probability corresponding to the first experimental data and the second conditional probability corresponding to the second experimental data, wherein the first experimental data includes the confusion data, the intervention data and the observation results within the target time window, the second experimental data includes the confusion data and the intervention data, and the first conditional probability and the second conditional probability are determined based on the preset confusion data; The target calibration weights are calculated based on the first conditional probability and the second conditional probability, and a calibration estimator is trained based on the target calibration weights. The target confusion data and the corresponding observation results in the first experimental data are input into the calibration estimator to obtain the first calibration weight. The target confusion data and the estimation results are input into the calibration estimator to obtain the second calibration weight. The estimation results are determined based on the target confusion data, the corresponding target intervention data, and the target result estimator. The estimation results are calibrated according to the first calibration weight and the second calibration weight to obtain calibration results. The target confusion data and the target intervention data are input into the initial data processing model to obtain prediction results. The initial data processing model is trained according to the prediction results and the calibration results to obtain the target data processing model.

[0006] According to a second aspect of the embodiments of this specification, a data processing model training apparatus is provided, comprising: The probability determination module is configured to determine a first conditional probability corresponding to the first experimental data and a second conditional probability corresponding to the second experimental data, wherein the first experimental data includes the confusion data, the intervention data and the observation results within the target time window, the second experimental data includes the confusion data and the intervention data, and the first conditional probability and the second conditional probability are determined based on the preset confusion data. The calibration estimator training module is configured to calculate target calibration weights based on the first conditional probability and the second conditional probability, and train a calibration estimator based on the target calibration weights. The weight determination module is configured to input the target confusion data and the corresponding observation results in the first experimental data into the calibration estimator to obtain a first calibration weight, and input the target confusion data and the estimation results into the calibration estimator to obtain a second calibration weight, wherein the estimation results are determined based on the target confusion data, the corresponding target intervention data, and the target result estimator. The model training module is configured to calibrate the estimation result according to the first calibration weight and the second calibration weight to obtain a calibration result, input the target confusion data and the target intervention data into the initial data processing model to obtain a prediction result, and train the initial data processing model according to the prediction result and the calibration result to obtain the target data processing model.

[0007] According to a third aspect of the embodiments of this specification, a data processing method is provided, comprising: Acquire first experimental data and second experimental data, wherein the first experimental data includes confounded data, intervention data and observation results within the target time window, and the second experimental data includes confounded data and intervention data; Input the obfuscated data in the second experimental data and the intervention data into the target data processing model to obtain the target result corresponding to the obfuscated data in the second experimental data; Based on the second experimental data and the target result, updated experimental data is obtained; based on the first experimental data and the updated experimental data, the target dataset is obtained. Based on the target dataset and the analysis estimator, data analysis results are obtained.

[0008] According to a fourth aspect of the embodiments of this specification, a data processing apparatus is provided, comprising: The data acquisition module is configured to acquire first experimental data and second experimental data, wherein the first experimental data includes confounding data, intervention data and observation results within the target time window, and the second experimental data includes confounding data and intervention data. The result acquisition module is configured to input the obfuscated data in the second experimental data and the intervention data into the target data processing model to obtain the target result corresponding to the obfuscated data in the second experimental data. The dataset acquisition module is configured to obtain updated experimental data based on the second experimental data and the target result, and to obtain the target dataset based on the first experimental data and the updated experimental data. The data analysis module is configured to obtain data analysis results based on the target dataset and the analysis estimator.

[0009] According to a fifth aspect of the embodiments of this specification, a computing device is provided, comprising: Memory and processor; The memory is used to store computer programs / instructions, and the processor is used to execute the computer programs / instructions. When the computer programs / instructions are executed by the processor, they implement the steps of the above-mentioned data processing model training method and data processing method.

[0010] According to a sixth aspect of the embodiments of this specification, a computer-readable storage medium is provided that stores a computer program / instructions, which, when executed by a processor, implement the steps of the above-described data processing model training method and data processing method.

[0011] According to a seventh aspect of the embodiments of this specification, a computer program product is provided, including a computer program / instructions that, when executed by a processor, implement the steps of the above-described data processing model training method and data processing method.

[0012] This specification provides a data processing model training method in one embodiment, which determines the first conditional probability and the second conditional probability of the first experimental data and the second experimental data based on preset confused data, and calculates the target calibration weight. The target calibration weight precisely quantifies how many samples in the first experimental data (including observations) need to represent the same confused data from the second experimental data (without observations). The target calibration weight is learned by training a calibration estimator, which essentially constructs a compensation mechanism. That is, when some confused data in the first experimental data has the same characteristics as the confused data in a large number of censored second experimental data, the calibration estimator will assign a higher weight to it. In this way, by assigning a high weight, the missing observations in the second experimental data due to time window limitations are compensated, and the selection bias is offset.

[0013] By inputting the same target-confused data into a pre-trained calibration estimator along with the actual observations and the estimation results generated by the target result estimator, first and second calibration weights are obtained. The difference between these two calibration weights accurately reflects the deviation between the actual observed distribution and the predicted distribution. Using this difference to perform targeted calibration on the estimation results can directly correct the cognitive bias of the target result estimator caused by learning the first experimental data (biased data). Using the corrected calibration results and the prediction results of the initial data processing model, the target data processing model is trained. This model learns the ability to infer the potential results corresponding to the full dataset (including the first and second experimental data), significantly improving the generalization performance and estimation accuracy of the target data processing model in the presence of censored data (second experimental data that does not contain the observation results), making its output a more accurate unbiased estimate of the true population distribution. Attached Figure Description

[0014] Figure 1 This is a schematic diagram of a data processing model training method provided in one embodiment of this specification; Figure 2 This is a flowchart illustrating a data processing model training method provided in one embodiment of this specification; Figure 3 This is a flowchart illustrating a data processing method provided in one embodiment of this specification; Figure 4 This is a flowchart illustrating the processing procedure of a data processing method provided in one embodiment of this specification. Figure 5 This is a schematic diagram of a survival analysis model provided in one embodiment of this specification; Figure 6a This is a causal diagram illustrating a survival analysis provided in one embodiment of this specification; Figure 6bThis is a causal diagram illustrating a modified survival analysis provided in one embodiment of this specification; Figure 7 This is a schematic diagram of the structure of a data processing model training device provided in one embodiment of this specification; Figure 8 This is a schematic diagram of the structure of a data processing apparatus provided in one embodiment of this specification; Figure 9 This is a structural block diagram of a computing device provided in one embodiment of this specification. Detailed Implementation

[0015] Many specific details are set forth in the following description to provide a full understanding of this specification. However, this specification can be implemented in many other ways than those described herein, and those skilled in the art can make similar extensions without departing from the spirit of this specification. Therefore, this specification is not limited to the specific implementations disclosed below.

[0016] The terminology used in one or more embodiments of this specification is for the purpose of describing particular embodiments only and is not intended to be limiting of the one or more embodiments of this specification. The singular forms “a,” “described,” and “the” as used in one or more embodiments of this specification and the appended claims are also intended to include the plural forms unless the context clearly indicates otherwise. It should also be understood that the term “and / or” as used in one or more embodiments of this specification refers to and includes any or all possible combinations of one or more associated listed items.

[0017] It should be understood that although the terms first, second, etc., may be used to describe various information in one or more embodiments of this specification, such information should not be limited to these terms. These terms are only used to distinguish information of the same type from one another. For example, first may also be referred to as second without departing from the scope of one or more embodiments of this specification, and similarly, second may also be referred to as first. Depending on the context, the word "if" as used herein may be interpreted as "when," "when," or "in response to a determination."

[0018] Furthermore, it should be noted that the user information (including but not limited to user device information, user personal information, etc.) and data (including but not limited to data used for analysis, stored data, displayed data, etc.) involved in one or more embodiments of this specification are all information and data authorized by the user or fully authorized by all parties. Moreover, the collection, use and processing of related data must comply with the relevant laws, regulations and standards of the relevant countries and regions, and corresponding operation entry points are provided for users to choose to authorize or refuse.

[0019] First, the terms and concepts used in one or more embodiments of this specification will be explained.

[0020] Subscription: The act of a user purchasing a financial product.

[0021] Redemption: The act of a user reselling their held financial products to realize cash.

[0022] Balance: The amount held in wealth management products. Balance = Subscriptions - Redemptions.

[0023] Intervention variables: External actions or changes (such as marketing strategies / products) imposed on research subjects (such as users) with the aim of observing the impact of such actions on outcome variables of interest (such as project metrics).

[0024] Confounding variables: Variables that affect both intervention variables and outcome variables.

[0025] CATE (Conditional Average Treatment Effect) / HTE (Heterogeneous Treatment Effect): A commonly used target estimator in causal inference, which is the difference between the outcome after applying intervention variable A and the outcome after applying intervention variable B, given the characteristics of a set of samples.

[0026] VAE: Variational Autoencoder, an unsupervised probabilistic generative model. It maps input data to the distribution parameters of latent variables through an encoder, and then reconstructs the data from the latent variables through a decoder. It is often used to learn the unconditional distribution of input data and to recover latent variables.

[0027] In project strategy decision-making scenarios, it is necessary to determine the outcome indicators obtained after implementing project strategy interventions on research subjects during the experimental phase. Analyzing these outcome indicators allows for the determination of the causal effects of the project strategy. Specifically, in scenarios such as loans and wealth management, indicators such as spending amount, balance increase, and subscription amount are often the outcome indicators to be observed. However, the distribution of these indicators exhibits an extremely right-skewed characteristic (zero-inflation right-skewed). This distribution is mainly due to the presence of a large number of zero values, indicating that users have no spending / deposits (zero inflation); and that only a small portion of deposit / loan users have large amounts of deposits / spending (long tail). Therefore, a long experimental period and a large sample size are required to obtain reliable analytical conclusions, leading to problems such as excessively long decision-making times and wasted resources.

[0028] When a survival analysis framework is introduced for modeling during the experimental phase of a project, an event can be considered a user's first purchase, and the occurrence of an event is regarded as a user conversion. The key performance indicator is the time of the event or the user conversion time. During the experiment, uncensored data can be obtained based on the users who have been observed to have converted, while censored data can be obtained based on the remaining unconverted users. However, the existing solution ignores the real-world problem of selection bias between censored and uncensored data, so the analysis results are still inaccurate due to bias.

[0029] This specification provides a data processing model training method. One or more embodiments of this specification also relate to a data processing model training device, a data processing method, a data processing apparatus, a computing device, a computer-readable storage medium, and a computer program product, which will be described in detail in the following embodiments.

[0030] See Figure 1 , Figure 1 A schematic diagram of a data processing model training method according to an embodiment of this specification is shown.

[0031] Specifically, the data processing model training method is applied to a data processing system, which includes an edge device 102 and a server 104. The edge device 102 is used to send first experimental data and second experimental data to the server 104.

[0032] The target data processing model was trained on server 104 based on the first and second experimental data.

[0033] Specifically, the process involves determining a first conditional probability corresponding to the first experimental data and a second conditional probability corresponding to the second experimental data. The first experimental data includes the obfuscated data, intervention data, and observations within the target time window. The second experimental data includes obfuscated data and intervention data. The first and second conditional probabilities are determined based on preset obfuscated data. Target calibration weights are calculated based on the first and second conditional probabilities, and a calibration estimator is trained based on these target calibration weights. The target obfuscated data and corresponding observations from the first experimental data are input into the calibration estimator to obtain the first calibration weight. The target obfuscated data and the estimation results are then input into the calibration estimator to obtain the second calibration weight. The estimation results are determined based on the target obfuscated data, the corresponding target intervention data, and the target result estimator. The estimation results are calibrated based on the first and second calibration weights to obtain a calibration result. The target obfuscated data and the target intervention data are input into an initial data processing model to obtain a prediction result. The initial data processing model is then trained based on the prediction result and the calibration result to obtain a target data processing model.

[0034] The target data processing model can be returned to the edge device 102 for deployment on the edge device 102, or the calling interface of the target data processing model can be returned to the edge device 102 so that the edge device 102 can apply the target data processing model for data processing through the calling interface. No limitation is made here.

[0035] The edge device 102 may include a browser, an app (application), or a web application such as an H5 (Hypertext Markup Language 5) application, a lightweight application (also known as a mini-program), or a cloud application. The edge device can be developed based on a software development kit (SDK) provided by the server, such as a real-time communication (RTC) SDK. The edge device can be deployed in an electronic device and depends on the device's operation or certain apps within the device to run. The electronic device may have a display screen and support information browsing, such as a personal mobile terminal like a mobile phone, tablet, or personal computer. Various other types of applications can also be configured in the electronic device, such as human-computer interaction applications, model training applications, data processing model training applications, web browser applications, shopping applications, search applications, instant messaging tools, email clients, and social media platform software.

[0036] Server 104 can be understood as a server providing various services, including physical servers and cloud servers. Examples include servers providing communication services to multiple clients, servers supporting backend training of models used on clients, and servers processing data sent by clients. It's important to note that Server 104 can be implemented as a distributed server cluster composed of multiple servers, or as a single server. Server 104 can also be a server in a distributed system, or a server integrated with blockchain. Server 104 can also be a cloud server providing basic cloud computing services such as cloud services, cloud databases, cloud computing, cloud functions, cloud storage, network services, cloud communication, middleware services, domain name services, security services, content delivery networks (CDNs), and big data and artificial intelligence platforms, or an intelligent cloud computing server or intelligent cloud host with artificial intelligence technology.

[0037] See Figure 2 , Figure 2 A flowchart of a data processing model training method provided in one embodiment of this specification is shown, which specifically includes the following steps.

[0038] Step 202: Determine the first conditional probability corresponding to the first experimental data and the second conditional probability corresponding to the second experimental data, wherein the first experimental data includes the confusion data, the intervention data and the observation results within the target time window, the second experimental data includes the confusion data and the intervention data, and the first conditional probability and the second conditional probability are determined based on the preset confusion data.

[0039] The first experimental data can be understood as data containing observation results that has been observed within the target time window. The target time window can be understood as the experimental window. In this embodiment of the specification, the first experimental data is uncensored data. The second experimental data can be understood as data in which the final result was not observed within the target time window. In this embodiment of the specification, the second experimental data is censored data whose observation results are missing due to the expiration of the time window.

[0040] Confounded data can be understood as confounding variables. In user conversion scenarios, confounding variables can be understood as user characteristics that can influence which intervention variables a user will accept (such as which marketing strategy they are assigned to) and the final conversion result (such as placing an order). For example, confounding variables include historical spending amount, user level (ordinary or member), activity level, etc. Intervention data can be understood as intervention variables. In user conversion scenarios, intervention variables can be preferential strategies, marketing methods, etc., applied to users.

[0041] The first conditional probability and the second conditional probability can be understood as the predicted probability of an event occurring under given conditions. In the embodiments of this specification, the first conditional probability and the second conditional probability are probabilities calculated under the same preset confusion data.

[0042] Specifically, taking the user conversion scenario as an example, in survival analysis, the sample feature (i.e., user feature) is defined as X, which includes a confounding variable, an intervention variable (i.e., intervention data) as W, the actual / potential event occurrence time (i.e., user conversion time) as T, the target time window as C, and the observation time (i.e., the observation result) as Y. The observation time is determined by the minimum of the event occurrence time and the target time window. For each user in the project experiment, the corresponding user feature, intervention variable, observation time, and censoring flag δ (δ=0 represents that the data is censored, and δ=1 represents that the data is not censored) are obtained. That is, each experimental data (including censored data and uncensored data) consists of {X, W, Y, δ}.

[0043] For example, with a target time window of 7 days, if user a is observed to have converted on day 5, then the actual event time for user a is 5, and the observation time is 5. However, user b does not convert within the target time window but converts on day 9, so the potential event time for user b is 9. The minimum value between the target time window and the potential event time is taken to determine the observation time for user b as 7. In reality, it is not known that user b converted on day 9 when the experiment ends, so the target time window is used as the observation time in the censored data.

[0044] It should be noted that the experimental data must meet the following assumptions: Intervention consistency, i.e., one intervention variable W corresponds to one version, and the observed event occurrence time equals its potential outcome; Negligibility, i.e., it is assumed that there are no unknown confounding variables, thus ensuring that after considering the influence of user characteristics, the correlation between the captured intervention variable and the event occurrence time is a causal relationship; Intervention overlap, i.e., each sample (user) has a probability of receiving intervention from any intervention variable, thus supporting the comparison of results between different intervention groups, thereby obtaining a causal effect; Censorship overlap, i.e., each sample has a probability of being censored or not censored; Negligible censoring, i.e., there are no unknown confounding variables that can simultaneously affect the event occurrence time and the target time window.

[0045] In specific implementation, the first experimental data in the embodiments of this specification can be uncensored data, and the second experimental data can be censored data. The first conditional probability and the second conditional probability are both calculated based on the preset confusion data. The purpose is to quantify the conditional distribution of the shadow variable in different experimental data when given the same preset confusion data, so as to lay the foundation for subsequent calculation of the weight used to correct the bias.

[0046] In practical applications, in a 7-day promotional strategy experiment, the first experimental data includes user data that clearly makes a purchase decision within 7 days; the second experimental data includes user data that did not purchase at the end of 7 days but may still convert in the future (i.e., censored data). The first conditional probability can be the probability that a user will complete a purchase decision within 7 days given the user's activity level (pre-mixed data), and the second conditional probability is the probability that the user's decision result cannot be observed after 7 days given the same user activity level.

[0047] In one or more embodiments of this specification, shadow data is related to the outcome variable (i.e., the time of event occurrence) but is unrelated to whether the data has been censored. Therefore, shadow data can serve as a measure of the bias caused by censoring. Specifically, a first distribution estimator is trained to learn the first conditional probability corresponding to the first experimental data, and a second distribution estimator is trained to learn the second conditional probability corresponding to the second experimental data. Based on the first and second conditional probabilities, the data distribution differences between different experimental data (such as censored and uncensored data) are quantified. Subsequently, the distribution information of the shadow data can be used to directly infer and correct the data distribution bias between censored and uncensored data. The specific implementation method is as follows: Determine the first conditional probability corresponding to the first experimental data, including: Determine the initial obfuscated data in the first experimental data, and the first initial shadow data corresponding to the initial obfuscated data; The initial confusion data in the first experimental data is input into the first distribution estimator to obtain the first predicted shadow data. The first distribution estimator is trained based on the first initial shadow data and the first predicted shadow data to obtain the trained first distribution estimator. The first conditional probability corresponding to the first experimental data is determined based on the first distribution estimator after training.

[0048] Shadow data and obfuscated data are determined from sample features. Shadow data (i.e. shadow variables) are related to the time T of the event, but are not related to whether the data has been censored δ.

[0049] In specific implementation, the sample features in the first experimental data are decoupled to obtain the decoupled initial confused data and the corresponding first target shadow data. By inputting the preset confused data into the first distribution estimator, the predicted first predicted shadow data is obtained. The decoupled first target shadow data is used as label data, and the loss between the first predicted shadow data and the first target shadow data is calculated. The first distribution estimator is then trained based on the calculated loss to obtain the trained first distribution estimator.

[0050] In fact, through the training process described above, the trained first distribution estimator learns the first conditional probability of the shadow data given the confused data in the first experimental data. This conditional probability represents the conditional distribution of the shadow data in the first experimental data. Therefore, when the trained first distribution estimator is used to determine the corresponding shadow data for the input confused data, it does not directly make a deterministic prediction of a single value, but instead samples or calculates the expectation from the learned conditional distribution to generate representative shadow data that matches the statistical characteristics of each input confused data. Thus, the first conditional probability corresponding to the first experimental data can be determined based on the trained first distribution estimator.

[0051] In practical applications, the first distribution estimator after training is itself an implicit or parameterized representation of the first conditional probability.

[0052] In one or more embodiments of this specification, the second conditional probability is determined by a trained second distribution estimator. Similar to training the first distribution estimator in the above embodiments, the second distribution estimator is trained based on the second initial shadow data corresponding to the initial confusion data in the second experimental data and the predicted second predicted shadow data. Specific implementation methods are as follows: Determining the second conditional probability corresponding to the second experimental data includes: Determine the initial confusion data in the second experimental data, and the corresponding second initial shadow data; The initial confusion data in the second experimental data is input into the second distribution estimator to obtain the second predicted shadow data. The second distribution estimator is trained based on the second initial shadow data and the second predicted shadow data to obtain the trained second distribution estimator. The second conditional probability corresponding to the second experimental data is determined based on the second distribution estimator.

[0053] The specific implementation method is similar to that described above and will not be repeated here; the trained second distribution estimator learns the second conditional probability of the shadow data when given the confused data in the second experimental number. This conditional probability represents the conditional distribution of the shadow data in the second experimental data.

[0054] The data processing model training method provided in the embodiments of this specification trains a first distribution estimator and a second distribution estimator respectively, so that the trained first distribution estimator can learn a first conditional probability and the trained second distribution estimator can learn a second conditional probability. When the conditional probability represents the conditional distribution of shadow data, the data distribution differences of different experimental data (such as censored data and uncensored data) can be quantified.

[0055] In one or more embodiments of this specification, obtaining confused data and shadow data from sample features is achieved through a target decoupling model. Specifically, the encoder in the initial decoupling model decouples the predicted confused data and predicted shadow data. Based on the mutual information loss between the predicted confused data and the predicted shadow data, and the reconstruction loss, the initial decoupling model is trained to obtain the target decoupling model. Detailed implementation methods are as follows: Before determining the first initial shadow data corresponding to the initial obfuscated data in the first experimental data, the method further includes: The original object data from the first experimental data and / or the second experimental data is input into the initial decoupling model. The encoder in the initial decoupling model is used to decouple the predicted confusing data and the predicted shadow data from the original object data. The original object data includes confusing data and shadow data. Calculate the mutual information loss between the predicted confused data and the predicted shadow data, and use the decoder in the initial decoupling model to reconstruct the predicted confused data and the predicted shadow data to obtain the reconstructed object data; Calculate the reconstruction loss between the original object data and the reconstructed object data, train the initial decoupling model based on the mutual information loss and the reconstruction loss, and obtain the target decoupling model.

[0056] The decoupling model can be understood as a neural network model that separates the original object data into independent data with clear semantic information. This decoupling model is implemented through a VAE model with an encoder-decoder structure. The initial decoupling model has the same model structure as the target decoupling model. Mutual information loss can be understood as a loss function used to measure and minimize the correlation between two data points, which is used to enable the decoupling model to learn mutually independent data representations. Reconstruction loss can be understood as a loss function that measures the decoupling model's ability to reconstruct the input data, which is used to ensure that the original object information is not lost during the decoupling process.

[0057] The original object data can be understood as the sample features mentioned above, representing the relevant features of the research object. This original object data contains obfuscated data and shadow data. The reconstructed object data can be understood as the object data obtained by reconstructing and restoring the information output by the encoder through the decoder in the decoupled model.

[0058] Specifically, the original object data derived from the first and / or second experimental data is determined. An initial decoupling model is trained based on this original object data to obtain the target decoupling model. In practice, the encoder in the initial decoupling model is responsible for decoupling the original object data into two independent representations (i.e., predicted confusing data and predicted shadow data), and ensuring the independence between the predicted confusing data and the predicted shadow data by calculating mutual information loss. The decoder in the initial decoupling model is responsible for reconstructing the original data based on the predicted confusing data and predicted shadow data obtained from decoupling, thereby obtaining reconstructed object data. The integrity of information is ensured by the reconstruction loss between the original object data and the reconstructed object data. The initial decoupling model is trained based on the mutual information loss and the reconstruction loss to obtain the target decoupling model.

[0059] The trained target decoupling model can automatically decouple obfuscated data and shadow data from complex raw object data.

[0060] The data processing model training method provided in the embodiments of this specification trains an initial decoupling model by calculating the mutual information loss between the two decoupled data and the reconstruction loss between the original object data and the reconstructed object data to obtain a target decoupling model. This allows the target decoupling model to automatically extract clean, independent confused data and shadow data from the mixed original object data, providing effective label data for training the first distribution estimator and the second distribution estimator.

[0061] In one or more embodiments of this specification, after training to obtain a target decoupling model, the target object data from experimental data (including first experimental data and second experimental data) is input into the target decoupling model to obtain initial confused data and decoupling shadow data. Based on this, a conditional independence test is performed on the decoupling shadow data to ensure that the obtained initial shadow data conforms to the premise assumptions of causal inference. Specific implementation methods are described below: Determining the first initial shadow data corresponding to the initial obfuscated data in the first experimental data includes: Input the target object data in the first experimental data into the target decoupling model to obtain the initial confused data and the first decoupling shadow data in the first experimental data. Perform a conditional independence test on the first decoupling shadow data to obtain the first initial shadow data. Determine the second initial shadow data corresponding to the initial confusion data in the second experimental data, including: Input the target object data from the second experimental data into the target decoupling model to obtain the initial confused data and the second decoupling shadow data from the second experimental data. Perform a conditional independence test on the second decoupling shadow data to obtain the second initial shadow data.

[0062] As mentioned in the above embodiments, shadow data is related to the outcome variable (i.e., the time of event occurrence) but is unrelated to whether the data is censored. Therefore, shadow data can be used as a measure of bias caused by censoring. That is, the distribution difference of shadow data in different experimental data (censored data and uncensored data) can purely reflect the overall difference in the time of event occurrence, which is unrelated to the censoring symbol δ. Therefore, by comparing the conditional distribution (i.e. conditional probability) of shadow data in different experimental data, the selection bias caused by censoring can be corrected unbiasedly without introducing new bias.

[0063] When determining the first and second conditional probabilities by training the first and second distribution estimators, and considering the conditional distribution of the shadow data given the confused data, it is necessary to ensure that the first initial shadow data used to train the first distribution estimator and the second initial shadow data used to train the second distribution estimator satisfy the aforementioned preconditions that are related to the outcome variable but independent of whether the data has been censored. Therefore, after obtaining the decoupled shadow data based on the target decoupling model, it is necessary to verify the conditional independence of the decoupled shadow data to ensure that the obtained initial shadow data is independent of the censoring flag δ.

[0064] Among them, the conditional independence test can be understood as a statistical hypothesis testing method, which is used to verify whether shadow data is independent of other variables in the experiment (in this embodiment, other variables are censoring markers) given confounded data and intervention data, thereby ensuring that shadow data meets the preconditions for causal inference.

[0065] Specifically, the target object data in the first experimental data (uncensored data) and the second experimental data (censored data) are input into the target decoupling model to obtain the corresponding first decoupling shadow data and second decoupling shadow data. Conditional independence tests are performed on the decoupled first decoupling shadow data and second decoupling shadow data to rigorously verify whether the first decoupling shadow data and second decoupling shadow data satisfy the theoretical premise that they are unrelated to the censoring label. Only the first decoupling shadow data and second decoupling shadow data that pass the conditional independence test will be determined as the first initial shadow data and the second initial shadow data, thus laying a reliable data foundation for subsequent unbiased distribution estimation and bias correction.

[0066] For example, an e-commerce platform conducts a 7-day promotional experiment to evaluate the effect of a new personalized recommendation algorithm (intervention variable) on user purchase conversion (outcome variable). The first experimental data includes user data that clearly shows a purchase behavior within 7 days; the second experimental data consists of user data that remains active but has not converted at the end of 7 days. The raw behavioral data (i.e., target object data, such as browsing history and clickstream) of these two types of user data are input into the target decoupling model, which outputs initial obfuscated data (such as "high-value users" and "price-sensitive users") and decoupling shadow data (such as "high-frequency visitors" and "nighttime active users"). The decoupling shadow data obtained from decoupling is subjected to a conditional independence test to ensure that behavioral patterns such as "high-frequency visits" are not directly related to whether the user's behavior was observed within 7 days (whether it was censored). After passing the conditional independence test, these pure behavioral patterns can be identified as initial shadow data for subsequent analysis.

[0067] The data processing model training method provided in this specification efficiently decouples decoupled shadow data from complex target object data through automated decoupling of the target decoupling model. The conditional independence test is used to ensure that the initial shadow data used to measure and correct biases satisfies the key assumptions of causal inference, making the subsequent calculations (conditional probability, calibration weights) theoretically rigorous. This ensures that the subsequent calculations can accurately quantify the real causal effects of the recommendation algorithm, which are not affected by the observation window, thereby avoiding making incorrect strategy decisions based on biased data and ultimately making the data analysis results more accurate and reliable.

[0068] Step 204: Calculate the target calibration weights based on the first conditional probability and the second conditional probability, and train a calibration estimator based on the target calibration weights.

[0069] Specifically, a calibration estimator is trained to evaluate data representativeness. In practice, a target calibration weight is calculated using the first and second conditional probabilities determined in the above embodiments (representing the distribution of corresponding shadow data given confused data in uncensored and censored data, respectively). This target calibration weight precisely quantifies how many censored samples with the same characteristics should be represented by each sample in the uncensored data. Using this target calibration weight as a learning objective, the calibration estimator is trained to automatically determine the statistical representativeness of any sample in the uncensored data.

[0070] For example, continuing with the previous example, for a "high-value, low-frequency visitor" who completed the conversion within 7 days, the second conditional probability determines that this type of user is more common in censored data (high second conditional probability) but less common in uncensored data (low first conditional probability). Therefore, if this type of user is observed in the first experimental data, this type of user needs to represent many users with the same characteristics but whose results have not been observed. Therefore, a higher weight (e.g., a target calibration weight of 5) is assigned to this type of user by using the target calibration weight calculated by the first conditional probability and the second conditional probability (target calibration weight = second conditional probability / first conditional probability).

[0071] When a calibration estimator is obtained through training with target calibration weights, this calibration estimator, as an automated tool to correct selection bias, can automatically assign a weight to each sample in the first experimental data in subsequent application stages. Among them, samples in the uncensored data that have the same characteristics as the censored data will be given higher weights. By assigning weights, the data distribution of the uncensored data is adjusted so that it can represent the unbiased overall data distribution, thereby learning more generalized and realistic patterns that cover the entire user group. In turn, based on biased observation data, an unbiased causal conclusion that is closer to the real situation of the entire user group can be drawn.

[0072] In one or more embodiments of this specification, preset scrambled data and corresponding observation results from the first experimental data are input into an initial estimator to obtain predicted calibration weights. The loss between these predicted calibration weights and the target calibration weights determined based on conditional probability is calculated to train the initial estimator and obtain a calibration estimator. Specific implementation methods are described below: A calibration estimator is obtained by training based on the target calibration weights, including: The preset obfuscated data and the corresponding observation results are input into the initial estimator to obtain the prediction calibration weights. The initial estimator is trained based on the target calibration weights and the predicted calibration weights to obtain the calibration estimator.

[0073] Specifically, the target calibration weights are calculated using the first conditional probability and the second conditional probability based on the pre-set confused data. Therefore, when training the initial estimator based on the target calibration weights, it is necessary to ensure that the input of the initial estimator and the calculation source of the target calibration weights are consistent, that is, the pre-set confused data is used to obtain the predicted calibration weights.

[0074] As mentioned in the above embodiments, target calibration weights can be assigned to samples, thereby changing the relative importance of samples in the data distribution, achieving the effect of adjusting the data distribution so that the adjusted data distribution is closer to the empirical distribution of the overall true situation (i.e., achieving calibration for the data distribution).

[0075] In practice, when censored data lacks observations compared to uncensored data, pre-set scrambled data and corresponding true observations are input into the initial estimator to obtain predicted calibration weights for the observations. The loss between these predicted calibration weights and the target calibration weights is calculated, and the initial estimator is trained. By reducing the gap between the predicted calibration weights and the target calibration weights, the predicted calibration weights, when assigned to the observations in the sample data, can correct the empirical distribution of the true observations through weighted adjustment, thus obtaining an unbiased empirical distribution of the observations. After the distribution is corrected, a reliable statistical basis is provided for accurately estimating the observation time corresponding to the censored data.

[0076] Once a calibration estimator is trained, it can output calibration weights representing the importance of the distribution of observations, based on the confusing data and the observations.

[0077] The data processing model training method provided in this specification obtains a calibration estimator through training, making it a plug-and-play tool that can quickly assign appropriate calibration weights to any new data, greatly improving the efficiency of the entire analysis process. Furthermore, based on the calibration weights learned by the calibration estimator, the data distribution can be calibrated, thereby ultimately obtaining reliable and unbiased data analysis results.

[0078] Step 206: Input the target confusion data and the corresponding observation results from the first experimental data into the calibration estimator to obtain the first calibration weight. Input the target confusion data and the estimation results into the calibration estimator to obtain the second calibration weight. The estimation results are determined based on the target confusion data, the corresponding target intervention data, and the target result estimator.

[0079] The first calibration weight represents the true importance of the sample in the real world, while the second calibration weight represents the importance of the output estimation result of the target result estimator.

[0080] The target confusion data (sample features) and their corresponding real observations (such as conversion time) in the first experimental data are input into the calibration estimator. Since the calibration estimator has learned the mapping relationship of the data representativeness weights, the calibration estimator will output the first calibration weight, which represents the statistical importance that this type of data should have in the whole data in the real world.

[0081] The same target obfuscated data, along with the target outcome estimator's estimation result (model-predicted conversion time) based on the target obfuscated data and corresponding target intervention data (such as marketing strategies), are input into the calibration estimator. The calibration estimator outputs a second calibration weight, which, based on the target outcome estimator's corresponding estimation result, reflects how the target outcome estimator recognizes and evaluates the importance of this type of data.

[0082] By comparing the first calibration weight and the second calibration weight, the systematic deviation between the actual data distribution and the predicted distribution can be accurately quantified.

[0083] For example, based on the target confusion data X1 and the corresponding observation T1, the calibration estimator determines that users containing the target confusion data X1 belong to a scarce but important group among all users, and therefore outputs a high first calibration weight, such as 3. Next, the target confusion data X1 with the same characteristics and the corresponding target intervention data W1 are input into a target outcome estimator that predicts whether a user will convert. This target outcome estimator outputs an estimate (such as the predicted conversion probability) for each user. The same target confusion data X1 and the predicted estimate are then input into the same calibration estimator. At this point, based on the predicted estimate, the calibration estimator determines that this group of users is not so scarce, and therefore outputs a low second calibration weight, such as 1. This second calibration weight represents the target outcome estimator's perception of the importance of these users.

[0084] In fact, although the target outcome estimator is obtained through training, even if the prediction accuracy of the target outcome estimator is high, the data distribution learned internally may differ from the real world. This difference will be amplified in the presence of censored data, leading to biased causal effect estimation. Therefore, by comparing the first calibration weight (realistic benchmark) with the second calibration weight (the target outcome estimator's cognition), the bias of the target outcome estimator at the distribution level can be accurately diagnosed.

[0085] By calculating the first and second calibration weights, we can accurately identify which user groups the target result estimator exhibits cognitive biases and the extent of those biases. This provides a direct and quantitative basis for subsequently calibrating the output of the target result estimator. By correcting the target result estimator's erroneous perceptions, we can significantly improve its generalization ability and ensure that it can make accurate predictions on the full dataset.

[0086] In one or more embodiments of this specification, the initial result estimator obtains a predicted result based on the initial confusion data and initial intervention data in the first experimental data, calculates the loss between the predicted result and the initial observation result corresponding to the initial confusion data, thereby training the initial result estimator to obtain the target result estimator. Specific implementation methods are described below: Before inputting the target obfuscation data and estimation results into the calibration estimator to obtain the second calibration weight, the process further includes: Input the initial confusion data and the corresponding initial intervention data from the first experimental data into the initial result estimator to obtain the prediction result; The initial result estimator is trained based on the initial observation results corresponding to the initial confused data and the prediction results to obtain the target result estimator; The target confusion data and the target intervention data are input into the target outcome estimator to obtain the estimation result.

[0087] Specifically, the initial obfuscated data and initial intervention data from the first experimental data (i.e., uncensored data) are used as input to an initial outcome estimator (an untrained neural network). This estimator outputs a predicted result (e.g., predicted user conversion time). By comparing this predicted result with the actual initial observation result (i.e., actual conversion time), the difference between the two (loss function) is calculated. This allows the parameters of the initial outcome estimator to be adjusted and optimized, resulting in a target outcome estimator. This target outcome estimator learns the mapping relationship from obfuscated and intervention data to observed results. When the target obfuscated data and target intervention data are input into the trained target outcome estimator, estimation results for subsequent calibration and comparison can be obtained.

[0088] In fact, since the target outcome estimator is trained on uncensored data, it inherits selection bias. The data distribution patterns it learns cannot be generalized to censored data. Therefore, this preliminary, biased target outcome estimator and its output estimation results provide a correction target for the process of calibrating the cognition of the target outcome estimator in the above embodiments. That is, the subsequent use of the first calibration weight and the second calibration weight to identify and correct the cognitive bias of the target outcome estimator caused by data censoring.

[0089] Step 208: The estimation result is calibrated according to the first calibration weight and the second calibration weight to obtain the calibration result. The target confusion data and the target intervention data are input into the initial data processing model to obtain the prediction result. The initial data processing model is trained according to the prediction result and the calibration result to obtain the target data processing model.

[0090] Specifically, the estimation results are calibrated using the first calibration weight and the second calibration weight to obtain the debiased calibration results. The initial data processing model is based on the target confusion data and the target intervention data to obtain the predicted results. The loss between the predicted results and the calibration results is calculated to train the initial data processing model and obtain the target data processing model.

[0091] In practice, by using calibration and prediction results to train the initial data processing model, the initial data processing model is made to obtain calibration results that are closer to the true population distribution. By adjusting the model parameters, the target data processing model with adjusted parameters learns the unbiased true population distribution, thus achieving calibration and alignment from a biased data distribution to an unbiased data distribution.

[0092] Using the previous example, target user characteristics (i.e., target obfuscation data) and target intervention strategies (i.e., target intervention data) are input into the initial data processing model to obtain the predicted conversion time, while the calibration result is the calibrated conversion time adjusted based on the first calibration weight and the second calibration weight, which corrects the selection bias.

[0093] Based on the target data processing model obtained through training, it is possible to make near-unbiased predictions on new data directly, and then make reliable causal inferences based on unbiased predictions (such as evaluating the real effect of the strategy), thereby effectively avoiding decision-making errors caused by selection bias.

[0094] In one or more embodiments of this specification, a target data processing model can be used to make accurate inferences about censored data (i.e., second experimental data) that originally could not yield clear observation results, thereby obtaining the target result corresponding to the target obfuscated data in the censored data. Specific implementation methods are described below: After obtaining the target data processing model, the following is also included: The target obfuscation data and the corresponding target intervention data in the second experimental data are input into the target data processing model to obtain the target result corresponding to the target obfuscation data in the second experimental data.

[0095] Specifically, the target confusion data and corresponding target intervention data in the second experimental data (i.e., censored data) are input into a pre-trained and calibrated target data processing model. This target data processing model uses its internally learned rules that have been corrected for selection bias to predict the corresponding target outcomes for these censored data (i.e., the possible event occurrence times if the experimental time is long enough), thereby completing the results for the missing and incomplete censored data.

[0096] Using the previous example, in a 7-day e-commerce promotion experiment, the second experimental data consists of active but non-converted user data that neither purchased nor churned at the end of the 7th day. The user characteristics (i.e., target obfuscation data, such as "low activity" and "new user") and the types of coupons assigned to them (i.e. target intervention data) are input into the target data processing model, and the predicted target result can be "predicted conversion time is the 12th day".

[0097] The data processing model training method provided in the embodiments of this specification can accurately predict the results of censored data through the target data processing model, thereby enabling data analysis based on unbiased data, making scientific and reliable decisions, and indirectly shortening the experimental time when the target results of censored data can be predicted.

[0098] See Figure 3 , Figure 3 A flowchart of a data processing method provided in one embodiment of this specification is shown, which specifically includes the following steps.

[0099] Step 302: Obtain first experimental data and second experimental data, wherein the first experimental data includes confounding data, intervention data and observation results within the target time window, and the second experimental data includes confounding data and intervention data.

[0100] Step 304: Input the obfuscated data in the second experimental data and the intervention data into the target data processing model to obtain the target result corresponding to the obfuscated data in the second experimental data.

[0101] Specifically, after obtaining the target data processing model through the above-described data processing model training method, the target result corresponding to the confused data in the second experimental data is obtained based on the target data processing model. Detailed implementation methods can be found in the above embodiments and will not be repeated here.

[0102] Step 306: Based on the second experimental data and the target result, obtain updated experimental data; based on the first experimental data and the updated experimental data, obtain the target dataset.

[0103] In practice, the target results obtained above are used to update the second experimental data (i.e., censored data). Specifically, the calibrated target results predicted by the target data processing model are added to the censored data to make up for the missing observations in the censored data. Thus, updated experimental data containing the target results are obtained.

[0104] The updated experimental data containing the target results are merged with the original complete first experimental data (uncensored data) to construct a complete and unbiased target dataset. The experimental data in this target dataset is equivalent to simulating the ideal scenario in which all research objects in the experiment are fully observed.

[0105] Step 308: Based on the target dataset and the analysis estimator, obtain the data analysis results.

[0106] The final causal analysis is performed based on a high-quality target dataset. Specifically, the target dataset is input into an analysis estimator specifically designed for causal inference. This analysis estimator analyzes the data based on unbiased experimental data and ultimately outputs reliable data analysis results, such as the differences in the true effects between different intervention strategies.

[0107] The data processing method provided in the embodiments of this specification enables the analysis estimator to perform analysis based on a sample representing the population by using a synthesized target dataset, thereby obtaining an unbiased estimate. The obtained data analysis results have high reliability and can also achieve an accurate assessment of long-term effects within a limited time window, thereby indirectly shortening the experimental duration.

[0108] See Figure 4 , Figure 4 A flowchart illustrating the processing procedure of a data processing method provided in one embodiment of this specification is shown.

[0109] Specifically, before detailing the data processing methods, the generative analysis framework provided in the embodiments of this specification will first be explained. See [link to documentation]. Figure 5 , Figure 5 This is a schematic diagram of a survival analysis model provided in one embodiment of this specification.

[0110] In scenarios such as loans / wealth management, users need a certain conversion time before they show the behavior of subscribing to wealth management products. This delayed response may not be captured within a limited time window. Furthermore, the balance indicators that need to be monitored are more complex and difficult to predict. For example, the data distribution of balance indicators is more extreme, with a very serious zero inflation right skew problem, and it is sensitive to outliers. Users with large deposits / withdrawals account for a very small portion of the total sample, but have a significant impact on the overall sample mean and variance. They are outliers / outliers (important customers) that cannot be removed.

[0111] To accurately address these issues, the research perspective shifted from changes in monetary value to user response time during product interactions, and a survival analysis framework was introduced for modeling. For example, the event is considered the user's first purchase; the occurrence of the event is considered a user conversion, and the observed result is the event occurrence time divided by the user conversion time. Conversions were observed during the experiment (…). Figure 5 Users with green nodes (representing conversion nodes) are the true samples (not censored); however, the remaining unconverted users (censored) are divided into false negative samples (those who entered the experiment too late or whose conversion time was too long, and continued intervention outside the time window will eventually lead to conversion) and true negative samples (never convert).

[0112] In survival analysis, covariates / user characteristics are defined as follows: (i.e., the object data in the above embodiments), intervention variables Time of actual / potential events Corresponding time window / censorship time Tag / Observation Time Has it been deleted? The dataset is defined as follows: The estimated target is CATE / THE: .

[0113] The experimental data under this survival analysis satisfies the assumptions mentioned in the above embodiments, such as intervention consistency, negligibility, intervention overlap, censoring overlap, and negligible censoring. It also implicitly assumes that the data are independent and identically distributed, that is, the censored data and the uncensored data are homogeneous, and the results corresponding to the uncensored data can be directly extrapolated to the censored data.

[0114] Based on these assumptions, a causal diagram for survival analysis can be obtained, such as... Figure 6a As shown, Figure 6a A causal diagram illustrating a survival analysis provided in one embodiment of this specification is shown.

[0115] This causal diagram shows that user characteristic X and intervention variable W affect time window C and event occurrence time (i.e., actual event time T). Intervention variable W is influenced by user characteristic X, and observation time Y is influenced by time window C and event occurrence time T. Whether δ is censored is affected by the time window (if no results are observed within the time window, the experimental data is determined to be censored data; otherwise, it is determined to be uncensored data).

[0116] In survival analysis, due to the presence of right censoring, survival analysis can be transformed into a "random missing" problem. Therefore, most existing survival analysis methods follow this framework: For positive samples or uncensored data, a series of methods such as inverse probability weighting, double robust estimation, meta-learners, or causal forests are used to directly estimate CATE; subsequently, as an optional step, for negative samples or right-censored data, the survival function is modeled and data imputation is performed using tree models, expectation-maximization algorithms, etc., and then the imputed data is merged with the uncensored data to form a complete dataset. Finally, common methods such as tree models, double robust estimation, meta-learners, or causal forests are used again to estimate CATE.

[0117] In reality, the event occurrence time T and the time window C both affect the censoring indicator δ. For example, when the time window C approaches 0, data will definitely be censored, while when the time window C approaches infinity, data will definitely not be censored. In survival analysis scenarios, as long as the conditions include T and C, the dependency path between any other variable (such as X) and δ can be blocked. When the event occurrence time T approaches 0, data will definitely not be censored, while when the event occurrence time T approaches infinity, data will definitely be censored.

[0118] In a causal diagram, the event occurrence time T and the time window C jointly influence the censoring flag δ, represented by an edge pointing from T and C to δ. This means that T and C simultaneously affect δ. (See [link to diagram]). Figure 6b , Figure 6b A causal diagram illustrating a modified survival analysis provided in one embodiment of this specification is shown.

[0119] In survival analysis, the event occurrence time T and the time window C jointly determine the censoring indicator δ. However, when δ=0, the event occurrence time T is unobservable. Traditional analytical methods often ignore this fact and fail to handle the selection bias between censored and uncensored data. This directly leads to the result that 𝔼[T|X, W, C, δ=0] ≠ 𝔼[T|X, W, C, δ=1]. Therefore, any direct comparison of different δ groups will produce serious selection bias, resulting in biased estimation conclusions.

[0120] In the embodiments of this specification, by means of Figure 4 The three-stage processing method yields accurate causal effect estimation results.

[0121] Phase 1 is used to construct shadow variables. Specifically, the shadow variable Z satisfies the following conditions: Condition (1) For uncapped data with δ=1, given user feature X and intervention variable W, the shadow variable Z is conditionally correlated with the event occurrence time T, i.e. , The condition (2) is independent of the given user characteristics X, intervention variables W, event occurrence time T, Z, and whether censoring is δ, i.e. , It signifies independence.

[0122] In practice, the VAE in phase 1 (i.e., the initial decoupling model in the above embodiment) is trained using uncapped data with δ=1. Specifically, the user feature X is input into the encoder in the VAE to decouple the confusion variable. And the initially screened shadow variable Z (i.e., decoupled shadow data); the confusion variable The initially screened shadow variable Z is input into the decoder in the VAE to reconstruct the features. (i.e., reconstructing object data).

[0123] The loss of this VAE includes reconstruction loss, which is the original input user feature data X and its reconstructed data. The differences between them; mutual information loss, i.e., maximizing conditional correlation. KL divergence, i.e., recovering latent variables The default loss, i.e., the conditional distribution With prior distribution Similarity between them; backpropagation based on calculated loss (i.e. Figure 4 (See the blue arrow in the image) Adjust the parameters of the VAE to obtain a trained VAE (i.e., the target decoupling model).

[0124] Among them, mutual information loss is mainly used to constrain the parameter learning of VAE, so that the separated If Z satisfies the two conditions in the definition of a shadow variable, then generally, if the mutual information between two variables is 0, the two variables are considered independent. Conversely, the greater the mutual information between two variables, the more correlated they are considered. Specifically, the core objective of a VAE is to maximize conditional correlation, that is, to enhance the conditional correlation between the shadow variable Z and the event occurrence time T. This is specifically achieved by training a variational distribution. To approximate the true conditional distribution Optimize parameters by maximizing log-likelihood This ensures that the model can accurately capture the mapping relationship from Z to T. Based on this, the upper bound of the conditional mutual information between Z and T is maximized, that is, its negative value is minimized. By simulating the constraints on the correlation between Z and T under the given conditions of X and W, it is ensured that the shadow variable Z can effectively capture information related to the time T of the event.

[0125] In survival analysis scenarios, due to the presence of right-censored data, the event occurrence time T is unobservable in this type of sample, making it impossible to directly verify or constrain the conditional independence hypothesis (Z and δ are independent given X, W, and T). To address this issue, mutual information is introduced as an alternative constraint tool to indirectly control the aforementioned condition (2). Specifically, by minimizing the upper bound of the mutual information between Z and δ, the shadow variable Z is made to carry as little information as possible related to the censored label δ during representation learning, thereby approximately satisfying the target independence.

[0126] The intervention overlap assumption serves as a fundamental premise in causal inference, ensuring that each sample has a probability of being assigned to a different intervention group. In the training of the VAE, this assumption is incorporated into the loss function design. By jointly optimizing the reconstruction loss, KL divergence, mutual information loss, and conditional independence constraints, the model is guided to learn a latent variable Z that satisfies both intervention overlap and conditional independence, thus providing a reliable feature representation for subsequent causal effect estimation.

[0127] By inputting both uncapped and uncensored data into the trained VAE, a preliminary shadow variable Z can be obtained. A conditional independence test is then performed on this preliminary shadow variable Z to obtain a strictly compliant shadow variable Z'. Considering that the representation learning of a VAE is not unique, it learns under multiple constraints... After representing Z, the shadow variable Z learned in the preliminary study is further examined, and a subset Z' of Z is selected to ensure that Z' satisfies the above conditions (1) and (2).

[0128] Phase 2 is used to impute censored data; specifically, it utilizes the data decoupled from the censored data at δ=0. Training the estimator (i.e., the second distribution estimator in the above embodiments) to learn This refers to the second conditional probability in the above embodiments. Specifically, The input is a confounding variable of δ=0 data. The output is the predicted shadow variable, and the monitoring signal is the shadow variable Z' of the data with δ=0. This can be achieved using distribution estimation methods such as kernel density estimation.

[0129] Decoupled using δ=1 uncensored data Training the estimator (i.e., the second distribution estimator in the above embodiments) to learn That is, the first conditional probability in the above embodiments. Specifically, The input is a confounding variable of δ=1 data. The output is the predicted shadow variable, and the monitoring signal is the shadow variable Z' of the data with δ=1. This can be achieved using distribution estimation methods such as kernel density estimation.

[0130] Using data with δ=1 Training the estimator (i.e., the calibration estimator in the above embodiments), for data with δ=1, the observation time Y is the event occurrence time T, and the actual goal is to learn the weight OR= , The input is a confounding variable of uncensored data. And the event occurrence time T, the output is Predicted weight values , The supervision signal is as described above. , The ratio of the corresponding predicted value, i.e. ; The goal is to make the predicted value To get as close as possible to the supervision signal, the loss function is minimized: .

[0131] In fact, it is also necessary to utilize the data from δ=1. Training the estimator (i.e., the target outcome estimator in the above embodiments) learns conditional probabilities Common estimation methods such as linear regression, meta-learners, and causal forests can be used to estimate the results. ,in This represents the predicted value output by the model. The data processing method provided in the embodiments of this specification aims to obtain an unbiased estimate of the conditional expectation E[T|X]. However, the event occurrence times in the uncensored data contain noise. Directly using the noisy T for subsequent global distribution correction would introduce unnecessary volatility. Therefore, the estimator is trained using data with δ=1. Come to learn The aim is to obtain a denoised and more robust conditional expectation estimate. This estimation result It represents the statistical regularity of the overall data and can more stably and reliably represent a specific user group (given). The typical timing of events provides a more reliable theoretical target for subsequent deviation correction steps.

[0132] After training and In this case, corresponding sample weights can be determined for samples in the uncensored data (based on the first calibration weights). Second calibration weight Sure), Multiply by this sample weight This allows the mathematical model for δ=1 data to be extrapolated to δ=0 data. It can be understood as the timing of potential events if δ=1 data becomes δ=0 (i.e., the calibration result in the above embodiment). .

[0133] The VAE (used to model the survival function) in phase 2 is trained using uncapped data with δ=1. The specific input... The encoder of the VAE outputs latent variables, i.e., the predicted potential time. .

[0134] The losses in Phase 2 of the VAE include: reconstruction loss, i.e., the loss from the original input data. and its reconstructed data The error between; the prediction error, i.e., the error obtained from the above. The calibration results are used as supervisory information to calculate the prediction error between the prediction result and the latent time of the encoder output in the VAE; KL divergence, i.e., the recovery of the latent true time in the VAE. The default loss, i.e., the conditional distribution With prior distribution The similarity between them.

[0135] The data with δ=0 will be used subsequently. The interpolated values ​​are obtained by inputting them into the trained VAE encoder. (i.e., the target result).

[0136] Phase 3 is used to estimate the causal effect. First, the complete dataset is integrated, specifically the complete dataset containing uncapped data with δ=1 (the original intervention variable W, and the decoupled confounding variables). The original label time Y) and censored data with δ=0 (original intervention variable W, decoupled confounding variable) Time after interpolation ).

[0137] Using the complete dataset, the results are fed into the outcome estimator to obtain the CATE (causal effect). The outcome estimator can be implemented using common estimation methods such as linear regression, meta-learners, and causal forests.

[0138] The data processing method provided in the embodiments of this specification, through survival analysis framework modeling, can directly deduce the occurrence time of events, more accurately estimate the causal effect of a certain strategy (intervention variable), and indirectly shorten the experimental duration. Specifically, it combines causal inference with survival analysis, and by modeling the potential occurrence time of events, it achieves the effect of shortening the experimental duration and calibrating experimental results within a limited time. Furthermore, by automatically learning the representation of shadow variables and using shadow variable theory to remove selection bias, it achieves unbiased imputation of censored data and unbiased estimation of causal effects, solving the problem of selection bias between censored and uncensored data.

[0139] Corresponding to the above method embodiments, this specification also provides embodiments of a data processing model training device. Figure 7 A schematic diagram of a data processing model training apparatus according to one embodiment of this specification is shown. Figure 7 As shown, the device includes: The probability determination module 702 is configured to determine a first conditional probability corresponding to the first experimental data and a second conditional probability corresponding to the second experimental data, wherein the first experimental data includes the confusion data, the intervention data and the observation results within the target time window, the second experimental data includes the confusion data and the intervention data, and the first conditional probability and the second conditional probability are determined based on the preset confusion data. The calibration estimator training module 704 is configured to calculate the target calibration weights based on the first conditional probability and the second conditional probability, and train a calibration estimator based on the target calibration weights. The weight determination module 706 is configured to input the target confusion data and the corresponding observation results in the first experimental data into the calibration estimator to obtain a first calibration weight, and input the target confusion data and the estimation results into the calibration estimator to obtain a second calibration weight, wherein the estimation results are determined based on the target confusion data, the corresponding target intervention data, and the target result estimator. The model training module 708 is configured to calibrate the estimation result according to the first calibration weight and the second calibration weight to obtain a calibration result, input the target confusion data and the target intervention data into the initial data processing model to obtain a prediction result, and train the initial data processing model according to the prediction result and the calibration result to obtain a target data processing model.

[0140] Optionally, the probability determination module 702 is further configured to: determine the initial confusion data and the corresponding first initial shadow data in the first experimental data; input the initial confusion data in the first experimental data into a first distribution estimator to obtain the first predicted shadow data; train the first distribution estimator based on the first initial shadow data and the first predicted shadow data to obtain the trained first distribution estimator; and determine the first conditional probability corresponding to the first experimental data based on the trained first distribution estimator.

[0141] Optionally, the probability determination module 702 is further configured to: determine the initial confusion data in the second experimental data and the corresponding second initial shadow data; input the initial confusion data in the second experimental data into the second distribution estimator to obtain the second predicted shadow data; train the second distribution estimator based on the second initial shadow data and the second predicted shadow data to obtain the trained second distribution estimator; and determine the second conditional probability corresponding to the second experimental data based on the second distribution estimator.

[0142] The apparatus further includes: a decoupling model acquisition module, configured to input the original object data from the first experimental data and / or the second experimental data into an initial decoupling model; use the encoder in the initial decoupling model to decouple the predicted confusing data and the predicted shadow data from the original object data, wherein the original object data includes confusing data and shadow data; calculate the mutual information loss between the predicted confusing data and the predicted shadow data; and use the decoder in the initial decoupling model to reconstruct the predicted confusing data and the predicted shadow data to obtain reconstructed object data; calculate the reconstruction loss between the original object data and the reconstructed object data; and train the initial decoupling model based on the mutual information loss and the reconstruction loss to obtain a target decoupling model.

[0143] Optionally, the probability determination module 702 is further configured to: input the target object data in the first experimental data into the target decoupling model to obtain the initial confused data and the first decoupling shadow data in the first experimental data; perform a conditional independence test on the first decoupling shadow data to obtain the first initial shadow data; input the target object data in the second experimental data into the target decoupling model to obtain the initial confused data and the second decoupling shadow data in the second experimental data; perform a conditional independence test on the second decoupling shadow data to obtain the second initial shadow data.

[0144] Optionally, the calibration estimator training module 704 is further configured to: input the preset scrambling data and the observation results corresponding to the preset scrambling data into an initial estimator to obtain predicted calibration weights; and train the initial estimator according to the target calibration weights and the predicted calibration weights to obtain the calibration estimator.

[0145] The apparatus further includes: a result estimator acquisition module, configured to input initial confusion data and corresponding initial intervention data from the first experimental data into an initial result estimator to obtain a prediction result; train the initial result estimator based on the initial observation results corresponding to the initial confusion data and the prediction result to obtain a target result estimator; and input the target confusion data and the target intervention data into the target result estimator to obtain the estimation result.

[0146] The device further includes a result acquisition module, configured to input the target obfuscation data and the corresponding target intervention data in the second experimental data into the target data processing model to obtain the target result corresponding to the target obfuscation data in the second experimental data.

[0147] The above is an illustrative scheme of a data processing model training device according to this embodiment. It should be noted that the technical solution of this data processing model training device and the technical solution of the data processing model training method described above belong to the same concept. For details not described in detail in the technical solution of the data processing model training device, please refer to the description of the technical solution of the data processing model training method described above.

[0148] Corresponding to the above method embodiments, this specification also provides data processing apparatus embodiments. Figure 8 A schematic diagram of the structure of a data processing apparatus according to one embodiment of this specification is shown. Figure 8 As shown, the device includes: The data acquisition module 802 is configured to acquire first experimental data and second experimental data, wherein the first experimental data includes confounding data, intervention data and observation results within the target time window, and the second experimental data includes confounding data and intervention data. The result acquisition module 804 is configured to input the obfuscated data in the second experimental data and the intervention data into the target data processing model to obtain the target result corresponding to the obfuscated data in the second experimental data. The dataset acquisition module 806 is configured to obtain updated experimental data based on the second experimental data and the target result, and to obtain the target dataset based on the first experimental data and the updated experimental data. The data analysis module 808 is configured to obtain data analysis results based on the target dataset and the analysis estimator.

[0149] The above is an illustrative scheme of a data processing apparatus according to this embodiment. It should be noted that the technical solution of this data processing apparatus and the technical solution of the data processing method described above belong to the same concept. For details not described in detail in the technical solution of the data processing apparatus, please refer to the description of the technical solution of the data processing method described above.

[0150] Figure 9 A structural block diagram of a computing device 900 according to one embodiment of this specification is shown. The components of the computing device 900 include, but are not limited to, a memory 910 and a processor 920. The processor 920 is connected to the memory 910 via a bus 930, and a database 950 is used to store data.

[0151] The computing device 900 also includes an access device 940, which enables the computing device 900 to communicate via one or more networks 960. Examples of these networks include Public Switched Telephone Network (PSTN), Local Area Network (LAN), Wide Area Network (WAN), Personal Area Network (PAN), or combinations of communication networks such as the Internet. The access device 940 may include one or more of any type of wired or wireless network interface (e.g., a network interface card (NIC)), such as an IEEE 802.11 Wireless Local Area Network (WLAN) wireless interface, a Wi-MAX (Worldwide Interoperability for Microwave Access) interface, an Ethernet interface, a Universal Serial Bus (USB) interface, a cellular network interface, a Bluetooth interface, or a Near Field Communication (NFC) interface.

[0152] In one embodiment of this specification, the above-described components of the computing device 900 and Figure 9 Other components, not shown, can also be connected to each other, for example, via a bus. It should be understood that... Figure 9 The block diagram of the computing device shown is for illustrative purposes only and is not intended to limit the scope of this specification. Those skilled in the art can add or replace other components as needed.

[0153] The computing device 900 can be any type of stationary or mobile computing device, including mobile computers or mobile computing devices (e.g., tablet computers, personal digital assistants, laptop computers, notebook computers, netbooks, etc.), mobile phones (e.g., smartphones), wearable computing devices (e.g., smartwatches, smart glasses, etc.) or other types of mobile devices, or stationary computing devices such as desktop computers or personal computers (PCs). The computing device 900 can also be a mobile or stationary server.

[0154] The processor 920 is used to execute the following computer program / instruction, which, when executed by the processor, implements the steps of the above-mentioned data processing model training method and data processing method.

[0155] The various embodiments in this specification are described in a progressive manner. Similar or identical parts between embodiments can be referred to interchangeably. Each embodiment focuses on its differences from other embodiments. In particular, the computing device embodiments are basically similar to the data processing model training method and data processing method embodiments, so the description is relatively simple. Relevant parts can be referred to the descriptions of the data processing model training method and data processing method embodiments.

[0156] An embodiment of this specification also provides a computer-readable storage medium storing a computer program / instructions that, when executed by a processor, implement the steps of the above-described data processing model training method and data processing method.

[0157] The various embodiments in this specification are described in a progressive manner. Similar or identical parts between embodiments can be referred to interchangeably. Each embodiment focuses on its differences from other embodiments. In particular, the computer-readable storage medium embodiments are relatively simple in description because they are fundamentally similar to the data processing model training method and data processing method embodiments; relevant parts can be referred to the descriptions of the data processing model training method and data processing method embodiments.

[0158] An embodiment of this specification also provides a computer program product, including a computer program / instructions, which, when executed by a processor, implement the steps of the above-described data processing model training method and data processing method.

[0159] The above is an illustrative scheme of a computer program product according to this embodiment. It should be noted that the technical solution of this computer program product belongs to the same concept as the technical solutions of the data processing model training method and the data processing method described above. For details not described in detail in the technical solution of the computer program product, please refer to the description of the technical solutions of the data processing model training method and the data processing method described above.

[0160] The foregoing has described specific embodiments of this specification. Other embodiments are within the scope of the appended claims. In some cases, the actions or steps recited in the claims may be performed in a different order than that shown in the embodiments and may still achieve the desired result. Furthermore, the processes depicted in the drawings do not necessarily require the specific or sequential order shown to achieve the desired result. In some embodiments, multitasking and parallel processing are possible or may be advantageous.

[0161] The computer instructions include computer program code, which may be in the form of source code, object code, executable file, or certain intermediate forms. The computer-readable medium may include: any entity or device capable of carrying the computer program code, recording media, USB flash drive, portable hard drive, magnetic disk, optical disk, computer memory, read-only memory (ROM), random access memory (RAM), electrical carrier signals, telecommunication signals, and software distribution media, etc. It should be noted that the content included in the computer-readable medium may be appropriately added or removed according to the requirements of patent practice. For example, in some regions, according to patent practice, computer-readable media may not include electrical carrier signals and telecommunication signals.

[0162] It should be noted that, for the sake of simplicity, the foregoing method embodiments are all described as a series of actions. However, those skilled in the art should understand that the embodiments in this specification are not limited to the described order of actions, because according to the embodiments in this specification, some steps can be performed in other orders or simultaneously. Furthermore, those skilled in the art should also understand that the embodiments described in this specification are all preferred embodiments, and the actions and modules involved are not necessarily essential to the embodiments in this specification.

[0163] In the above embodiments, the descriptions of each embodiment have different focuses. For parts not described in detail in a certain embodiment, please refer to the relevant descriptions in other embodiments.

[0164] The preferred embodiments disclosed above are merely illustrative of this specification. The optional embodiments do not exhaustively describe all details, nor do they limit the invention to the specific implementations described. Clearly, many modifications and variations can be made based on the embodiments described herein. These embodiments are selected and specifically described in this specification to better explain the principles and practical applications of the embodiments, thereby enabling those skilled in the art to better understand and utilize this specification. This specification is limited only by the claims and their full scope and equivalents.

Claims

1. A data processing model training method, comprising: determining a first conditional probability corresponding to first experimental data and a second conditional probability corresponding to second experimental data, wherein the first experimental data comprises the confounding data, intervention data, and observation results within a target time window, the second experimental data comprises the confounding data and the intervention data, and the first conditional probability and the second conditional probability are determined based on preset confounding data; calculating a target calibration weight according to the first conditional probability and the second conditional probability, and training a calibration estimator based on the target calibration weight; inputting target confounding data in the first experimental data and corresponding observation results into the calibration estimator to obtain a first calibration weight, and inputting the target confounding data and an estimated result into the calibration estimator to obtain a second calibration weight, wherein the estimated result is determined according to the target confounding data, corresponding target intervention data, and a target result estimator; performing calibration processing on the estimated result according to the first calibration weight and the second calibration weight to obtain a calibration result, inputting the target confounding data and the target intervention data into an initial data processing model to obtain a prediction result, and training the initial data processing model according to the prediction result and the calibration result to obtain a target data processing model.

2. The method of claim 1, wherein determining the first conditional probability corresponding to the first experimental data comprises: determining initial confounding data in the first experimental data and corresponding first initial shadow data; inputting the initial confounding data in the first experimental data into a first distribution estimator to obtain first predicted shadow data, training the first distribution estimator according to the first initial shadow data and the first predicted shadow data to obtain a trained first distribution estimator; and determining the first conditional probability corresponding to the first experimental data based on the trained first distribution estimator.

3. The method of claim 2, wherein determining the second conditional probability corresponding to the second experimental data comprises: determining initial confounding data in the second experimental data and corresponding second initial shadow data; inputting the initial confounding data in the second experimental data into a second distribution estimator to obtain second predicted shadow data, training the second distribution estimator according to the second initial shadow data and the second predicted shadow data to obtain a trained second distribution estimator; and determining the second conditional probability corresponding to the second experimental data based on the second distribution estimator.

4. The method of claim 3, wherein determining the initial confounding data in the first experimental data and the corresponding first initial shadow data further comprises: inputting original object data in the first experimental data and / or the second experimental data into an initial decoupling model, and decoupling predicted confounding data and predicted shadow data from the original object data by using an encoder in the initial decoupling model, wherein the original object data comprises confounding data and shadow data. calculating mutual information loss between the predicted confusion data and the predicted shadow data, and reconstructing the predicted confusion data and the predicted shadow data by using a decoder in the initial decoupling model to obtain reconstructed object data; calculating reconstruction loss between the original object data and the reconstructed object data, training the initial decoupling model based on the mutual information loss and the reconstruction loss to obtain a target decoupling model.

5. The method of claim 4, determining initial confusion data in the first experimental data and corresponding first initial shadow data comprises: inputting target object data in the first experimental data into the target decoupling model to obtain initial confusion data in the first experimental data and first decoupled shadow data, and performing conditional independence test on the first decoupled shadow data to obtain the first initial shadow data; determining initial confusion data in the second experimental data and corresponding second initial shadow data comprises: inputting target object data in the second experimental data into the target decoupling model to obtain initial confusion data in the second experimental data and second decoupled shadow data, and performing conditional independence test on the second decoupled shadow data to obtain the second initial shadow data.

6. The method of claim 1, training a calibration estimator based on the target calibration weight comprises: inputting the preset confusion data and observation corresponding to the preset confusion data into an initial estimator to obtain predicted calibration weight; training the initial estimator according to the target calibration weight and the predicted calibration weight to obtain the calibration estimator.

7. The method of claim 1, before inputting the target confusion data and estimation result into the calibration estimator to obtain second calibration weight, further comprising: inputting initial confusion data in the first experimental data and corresponding initial intervention data into an initial result estimator to obtain estimation result; training the initial result estimator according to initial observation corresponding to the initial confusion data and the estimation result to obtain a target result estimator; inputting the target confusion data and the target intervention data into the target result estimator to obtain the estimation result.

8. The method of any one of claims 1 to 7, after obtaining a target data processing model, further comprising: inputting target confusion data in the second experimental data and corresponding target intervention data into the target data processing model to obtain target result corresponding to the target confusion data in the second experimental data.

9. A data processing method, comprising: obtaining first experimental data and second experimental data, wherein the first experimental data contains confusion data, intervention data and observation in a target time window, and the second experimental data contains confusion data and intervention data; inputting confusion data and intervention data in the second experimental data into a target data processing model to obtain target result corresponding to the confusion data in the second experimental data; According to the second experimental data and the target result, updated experimental data is obtained, and according to the first experimental data and the updated experimental data, a target data set is obtained; According to the target data set and an analysis estimator, a data analysis result is obtained.

10. A data processing model training apparatus, comprising: a probability determination module configured to determine a first conditional probability corresponding to first experimental data and a second conditional probability corresponding to second experimental data, wherein the first experimental data comprises the confounding data, the intervention data, and the observation result within the target time window, the second experimental data comprises the confounding data and the intervention data, and the first conditional probability and the second conditional probability are determined based on preset confounding data; a calibration estimator training module configured to calculate a target calibration weight according to the first conditional probability and the second conditional probability, and train a calibration estimator based on the target calibration weight; a weight determination module configured to input target confounding data in the first experimental data and a corresponding observation result into the calibration estimator to obtain a first calibration weight, and input the target confounding data and an estimation result into the calibration estimator to obtain a second calibration weight, wherein the estimation result is determined according to the target confounding data, corresponding target intervention data, and a target result estimator; a model training module configured to calibrate the estimation result according to the first calibration weight and the second calibration weight to obtain a calibration result, input the target confounding data and the target intervention data into an initial data processing model to obtain a prediction result, and train the initial data processing model according to the prediction result and the calibration result to obtain a target data processing model.

11. A data processing apparatus, comprising: a data acquisition module configured to acquire first experimental data and second experimental data, wherein the first experimental data comprises confounding data, intervention data, and an observation result within a target time window, and the second experimental data comprises confounding data and intervention data; a result obtaining module configured to input confounding data in the second experimental data and the intervention data into a target data processing model to obtain a target result corresponding to the confounding data in the second experimental data; a data set obtaining module configured to obtain updated experimental data according to the second experimental data and the target result, and obtain a target data set according to the first experimental data and the updated experimental data; a data analysis module configured to obtain a data analysis result based on the target data set and an analysis estimator.

12. A computing device, comprising: a memory and a processor; the memory is configured to store computer programs / instructions, and the processor is configured to execute the computer programs / instructions, which realize the steps of the method in any one of claims 1 to 9 when executed by the processor.

13. A computer readable storage medium storing computer programs / instructions, which realize the steps of the method in any one of claims 1 to 9 when executed by a processor.

14. A computer program product comprising computer programs / instructions which, when executed by a processor, implement the steps of the method according to any one of claims 1 to 9.

Images