A heterogeneous table piece reuse method based on local variable relationship integration

Abstract

The application discloses a heterogeneous table artifact reuse method based on local variable relationship integration, constructs a global feature space through feature semantic alignment, integrates the reduction of multiple candidate artifacts according to the index alignment relationship between local features and global feature positions, forms a joint context, uses a table pre-training model or other models with table context reasoning capability to complete the features that do not match between the user task and the artifact, calculates the normalized entropy of the probability distribution of the missing feature predicted by the model output to measure the completion uncertainty, selects reliable artifacts for integrated prediction according to the uncertainty at the sample level, or combines the joint context with a small amount of user labeled data to perform end-to-end prediction. The application reduces the feature space adaptation cost in heterogeneous table artifact reuse and the prediction risk caused by low reliable completion, improves the model reuse efficiency, prediction stability and system scalability in small sample user tasks.

Description

Technical Field

[0001] This invention relates to a method for reusing heterogeneous table learning materials based on the integration of local variable relationships. It can be applied to scenarios such as healthcare, finance, industrial control, and public administration where there are inconsistencies in table feature spaces, insufficient labeled samples, or the need for cross-scenario model reuse. It can also support feature adaptation and reliability assessment when large language models, multimodal models, or intelligent agents call external table models. This invention belongs to the field of learning material reuse technology. Background Technology

[0002] Machine learning models are widely used in fields such as healthcare, finance, industry, and public administration. However, ordinary users often lack sufficient labeled data, computing resources, and model tuning experience, making it difficult to build high-quality models from scratch. The learning-materials paradigm encapsulates trained models and their specifications into shareable learning materials. Users can search and reuse existing models to complete their tasks through the learning-materials foundation system without accessing the developer's original training data. The specifications characterize the model's capabilities and the distribution of training data, serving as the core basis for searching and reusing learning materials.

[0003] Tabular data typically consists of numerical features, categorical features, and labels, and is extremely common in real-world business scenarios. Due to differences in data sources, feature engineering strategies, and domain knowledge among developers, different learning tools often possess different feature names, semantics, types, statistical distributions, and granularities. Even if multiple learning tools are designed for tabular learning tasks, they may exist in heterogeneous feature spaces. If user task samples are directly input into a learning tool model, the user samples may lack some features required by the learning tool, or the user features and learning tool features may not reliably correspond, significantly reducing model reuse effectiveness.

[0004] Furthermore, with the development of large language models, multimodal models, and agent technologies, more and more systems need to automatically call external tabular models to complete tasks such as classification, regression, risk assessment, recommendation, or control decisions. When such models or agents call tabular models, they can usually only obtain partial feature descriptions or local observation information of the target task, making it difficult to guarantee that their feature space is completely consistent with the input feature space of the callable tabular model.

[0005] Existing methods for reusing learning components in heterogeneous feature spaces often rely on subspace alignment or unified embedding space learning. These methods typically require manual partitioning of the feature space, or a unified embedding space that simultaneously possesses good representational capabilities and the ability to reconstruct the original feature space. When the feature dimension is high, the table structure is complex, the number of learning components continues to grow, and new tasks constantly emerge, it is often necessary to relearn the mapping relationship between feature blocks and the unified subspace for different scenarios. The resulting training requirements and computational overhead limit the scalability of the learning component base system. Furthermore, traditional simple average ensemble does not consider the differences in prediction reliability of different learning components on different test samples, and may include predictions from learning components that rely on a large number of low-quality imputed features in the final result, thus introducing noise. Summary of the Invention

[0006] Purpose of the Invention: Addressing the problem of learning materials being difficult to reuse directly due to different developers constructing them based on different table feature spaces, this invention provides a reuse method that integrates the local variable relationships of multiple learning materials within a heterogeneous table feature space, completes missing user task features, quantifies the reliability of the completion, and selects learning materials accordingly. This method should be able to generate shareable contextual representations using learning material specifications rather than the developer's original data, maintain stable performance even with limited user samples, and support two reuse approaches: calling existing learning material models and end-to-end prediction using pre-trained table models.

[0007] Technical Solution: A method for reusing heterogeneous table learning materials based on the integration of local variable relationships, applied to a computing environment including a learning material base system, a developer's end, and a user's end. The method includes the following steps: Step 100: The user terminal generates a user task specification based on the local table task data and submits it to the learning material base system; the learning material base system determines candidate learning materials from the existing learning materials in the system based on the specification information.

[0008] Step 200: The learning material base system constructs a global feature space based on user task characteristics and candidate learning material local features, and establishes an index alignment relationship between the local feature space and the global feature space.

[0009] Step 300: The learning material base system fills the global feature space with the candidate learning material specifications according to the index alignment relationship, and generates a joint context.

[0010] Step 400: Based on the joint context, the learning software base system determines the features to be completed that are not observed relative to the user samples in the input features of the candidate learning software model, completes the features to be completed, and calculates the completion uncertainty.

[0011] Step 500: The learning device base system calculates the sample-level learning device reliability based on the completion uncertainty, selects learning devices that meet the reliability requirements for prediction integration, or performs end-to-end prediction based on the joint context.

[0012] In step 100, candidate learning materials can be selected from all learning materials in the learning material base system through one or more of the following methods: semantic reduction screening, feature overlap ratio screening, task type screening, domain label screening, and causal feature screening. The candidate learning materials already have a set degree of overlap or semantic relevance with the feature space of the target task.

[0013] Furthermore, the core of step 200 is to organize user task features and candidate learning device local features into the same global feature space and record the index alignment relationship between the local features and the global feature positions; for user task features and candidate learning device local features that can be determined to correspond to the same variable, they are assigned to the same global feature position; for candidate learning device local features that fail to align with the existing global feature positions, the candidate learning device local features are added to the global feature space as new global features.

[0014] Further, step 300 includes the following steps: Step 301: Read the specification of the candidate learning materials.

[0015] Step 302: Fill the reduced samples into the global feature space according to the index alignment relationship from local features to global features.

[0016] Step 303: Set missing values ​​and missing masks for global feature locations not covered by the current candidate learning materials.

[0017] Step 304: Retain the reduced sample weights, source file identifiers, labels, or model output information.

[0018] Step 305: Summarize the reduction samples by row to obtain the joint context, which includes global feature values, missing mask, reduction sample weights and source component identifiers.

[0019] Further, step 400 includes the following steps: Step 401: For the sample to be predicted in the user task, compare the observed features of the sample to be predicted with the input features required by the candidate learning model, and determine the features required by the candidate learning model but not observed by the sample to be predicted, as the set of features to be completed.

[0020] Step 402: Filter the observed context samples of the feature to be completed from the joint context.

[0021] Step 403: Construct a table completion task using the features to be completed as temporary prediction targets.

[0022] Step 404: Call TabPFN or equivalent table pre-trained model to complete missing features.

[0023] Step 405: Obtain the missing feature completion values ​​and their predicted probability distribution.

[0024] Step 406: Calculate the normalized entropy of the predicted probability distribution to obtain the completed uncertainty.

[0025] Further, step 500 includes the following steps: Step 501: Summarize the completion uncertainty of the required features to be completed for candidate learning materials.

[0026] Step 502: Combining the completion uncertainty, the degree of overlap between the observed features of the user task and the input features required by the candidate learning device model, and the learning device quality information, calculate the sample-level learning device reliability score using a weighted summation method.

[0027] Step 503: Select reliable learning devices to participate in the prediction based on the sample-level learning device reliability score.

[0028] Step 504: Call the selected reliable learning software to output the prediction results, and perform weighted integration of multiple prediction results.

[0029] Step 505: In the optional path, input the joint context and user-labeled samples into the table pre-trained model and perform end-to-end prediction.

[0030] Step 506: Output the user task prediction results, or combine the selective integration results and the end-to-end prediction results.

[0031] As a second aspect of this application, the present invention also discloses an electronic device, including at least one processor and a memory communicatively connected to the at least one processor; the memory stores instructions executable by the at least one processor, which, when executed, enable the at least one processor to perform the above-described method steps.

[0032] As a third aspect of this application, the present invention also discloses a computer storage medium having a computer program stored thereon, wherein the computer program, when executed by a processor, implements the above-described method steps.

[0033] Beneficial effects: This invention replaces the alignment method that strongly depends on the reconstruction of a unified subspace with a global feature space expansion method, reducing the dependence of heterogeneous feature space learning component reuse on manual feature partitioning and strict feature consistency; it preserves the local variable relationships of multiple learning components through a reduction-based joint context, providing external distribution information to user tasks without sharing the developer's original data; it completes missing feature completion through a table pre-trained model, which can capture complex nonlinear feature dependencies; it quantifies the uncertainty of completion through normalized entropy, and selects reliable learning components at the sample level accordingly, reducing the interference of low-quality completion and low-matching learning components on the final prediction; it also provides two reuse paths: calling the original learning component model and end-to-end prediction based on the joint context, improving the adaptability of the method under different task conditions such as general tables and medical tables. Attached Figure Description

[0034] Figure 1 This is a flowchart illustrating the overall process of the heterogeneous feature space table learning component reuse method based on local variable relationship integration in this embodiment of the invention. Figure 2 This is a schematic diagram of the missing feature completion and uncertainty quantification process based on table context learning in step 300 of this embodiment of the invention. Figure 3 This is a schematic diagram of the uncertainty-based selective learning component integration process in step 400 of this embodiment of the invention; Figure 4 This is a schematic diagram illustrating the complementary reuse process of the original learning object calling path and the joint context end-to-end prediction path in step 500 of this embodiment of the invention.

[0035] It should be noted that this specification adopts a phased numbering method. Steps 100 to 500 represent the main stages of the method of the present invention; steps 301 to 305, steps 401 to 406, and steps 501 to 506 respectively represent further elaboration of steps 300, 400, and 500. Detailed Implementation

[0036] The present invention will be further illustrated below with reference to specific embodiments. It should be understood that these embodiments are for illustrative purposes only and are not intended to limit the scope of the invention. After reading the present invention, any modifications of the present invention in various equivalent forms by those skilled in the art will fall within the scope defined by the appended claims.

[0037] A method for reusing heterogeneous table learning materials based on the integration of local variable relationships is proposed. This method does not require different learning materials to share the exact same feature space. Instead, it constructs a global feature space based on user task features and candidate learning material features. The specifications of each learning material are then filled into the global space according to the feature index alignment relationship to form a joint context. A pre-trained table pre-trained model is then used to learn the local variable relationships from the joint context to complete the missing features of the user task for specific learning materials, while also providing the completion uncertainty. Finally, based on the sample-level uncertainty, a more reliable learning material is selected for prediction integration, thereby improving the reuse effect of heterogeneous table learning materials in scenarios with small samples and inconsistent features.

[0038] It should be noted that this invention emphasizes the universality of the methodological framework. The specification is not limited to a specific mathematical form; as long as it can express the data distribution, model capabilities, model behavior, or task applicability of the learning materials while protecting the original data, it can be used for the global feature space construction and joint context generation of this invention. The table pre-training model is also not limited to a specific model name; as long as it can perform table feature completion, probabilistic prediction, few-shot inference, or end-to-end table prediction based on context samples, it can be used as an alternative. Feature alignment, uncertainty measurement, learning material selection, and ensemble fusion can all be implemented by equivalent rule-based methods, statistical methods, vector retrieval methods, or learning-based methods.

[0039] The method disclosed in this invention mainly includes a submission phase, an organization phase, and a deployment phase. In the submission phase, developers generate and upload the learning material model and specification; in the organization phase, the learning material base system establishes a global feature space, joint context, and completion mechanism based on user tasks and candidate learning materials; in the deployment phase, the system performs completion, uncertainty assessment, selective ensemble, or end-to-end prediction on user samples and outputs the prediction results.

[0040] To better understand this invention, the following is combined with... Figures 1 to 4 Taking a medical form task as an example, one implementation of the method of the present invention will be described. The training of learning materials, submission of learning materials, generation of specifications, indexing of specifications, and preliminary search of candidate learning materials can all be implemented using existing technologies. This embodiment does not limit the specific implementation of the above processes.

[0041] Figure 1 This paper illustrates the phased overall flow of the heterogeneous feature space table component reuse method in an embodiment of the present invention. The flow includes the following phases: Step 100: The user terminal generates a user task specification based on the local table task data and submits it to the learning material base system; the learning material base system determines candidate learning materials from the existing learning materials in the system based on the specification information.

[0042] Step 200: The learning material base system constructs a global feature space based on user task characteristics and candidate learning material local features, and establishes an index alignment relationship between the local feature space and the global feature space.

[0043] Step 300: The learning material base system fills the global feature space with the candidate learning material specifications according to the index alignment relationship, and generates a joint context.

[0044] Step 400: Based on the joint context, the learning software base system determines the features to be completed that are not observed relative to the user sample in the input features of the candidate learning software model, completes the missing features when the user sample to be completed calls the candidate learning software, and calculates the completion uncertainty.

[0045] Step 500: The learning device base system calculates the sample-level learning device reliability based on the completion uncertainty, selects learning devices that meet the reliability requirements for prediction integration, or performs end-to-end prediction based on the joint context.

[0046] In this embodiment, the learning tool base system can select candidate learning tools through one or more of the following methods: semantic reduction screening, feature overlap ratio screening, task type screening, domain label screening, and causal feature screening. For example, the learning tool base system can prioritize medical form learning tools that are related to the in-hospital mortality risk prediction task and have a set degree of overlap or semantic relevance with the user's task features. The candidate learning tools obtained after screening can correspond to different local feature spaces such as demographic information, vital signs, laboratory test indicators, or neurological indicators.

[0047] In step 200, the learning material base system organizes user task features and candidate learning material local features into the same global feature space and records the index alignment relationship between the local features and the global feature positions. The global feature space is used to uniformly accommodate user task features and candidate learning material local features, and the index alignment relationship is used to record the corresponding position of the local features in the global feature space.

[0048] In this embodiment, if the user task feature "heart rate" and the candidate learning material local feature "heart rate" or "HR" are determined to correspond to the same variable, the learning material base system will assign the user task feature and the candidate learning material local feature to the same global feature position. If the candidate learning material local feature "serum creatinine" fails to align with an existing global feature position, the learning material base system will add the candidate learning material local feature as a new global feature to the global feature space. The learning material base system records the index alignment relationship between the local feature sequence number and the global feature position, and this index alignment relationship is used for subsequent sample reduction filling.

[0049] Figure 2The specific process of step 300 is shown. This process includes steps 301 to 305.

[0050] Step 301: Read the specification of the candidate learning object. In this embodiment, the specification may include specification samples, specification sample weights, labels, model output information, or statistical summary information.

[0051] Step 302: Based on the index alignment relationship from local features to global features, fill the reduced samples into the global feature space. For example, the value of the local feature "HR" of the candidate learning component is filled into the global feature position corresponding to "heart rate" in the global feature space.

[0052] Step 303: Set missing values ​​and missing masks for global feature locations not covered by the current candidate learning sample. The missing mask is used to represent the unobserved location of the current global reduction sample in the global feature space.

[0053] Step 304: Retain the reduction sample weights, source learning component identifiers, labels, or model output information. The source learning component identifier is used to record the candidate learning component to which the reduction sample belongs.

[0054] Step 305: Summarize the reduction samples row by row to obtain a joint context. The joint context includes global feature values, a missing mask, reduction sample weights, and source learning device identifiers. Merge multiple candidate learning device reduction samples to form a joint context. The above concatenation or summarization can be achieved using conventional data table merging or tensor concatenation methods.

[0055] Figure 3 The specific process of step 400 is shown. This process includes steps 401 to 406.

[0056] Step 401: For the sample to be predicted in the user task, compare the observed features of the sample with the input features required by the candidate learning model, and determine the features required by the candidate learning model but not observed in the sample to be predicted, as the set of features to be completed. For example, if a candidate learning model requires "blood glucose" and "creatinine" as input features, but the sample to be predicted has not observed these features, then "blood glucose" and "creatinine" are determined as features in the set of features to be completed.

[0057] Step 402: Filter observed context samples of the features to be completed from the joint context. The completion context sample set is used to provide context information for subsequent table completion tasks.

[0058] Step 403: Construct a table completion task using the features to be completed as temporary prediction targets.

[0059] Step 404: Call TabPFN or an equivalent tabular pre-trained model to complete missing features. The tabular pre-trained model outputs the prediction result of the feature to be completed based on the completion context sample set and the observed features of the sample to be predicted.

[0060] Step 405: Obtain the missing feature completion values ​​and their predicted probability distribution.

[0061] Step 406: Calculate the normalized entropy of the predicted probability distribution to obtain the completion uncertainty. The larger the normalized entropy, the higher the uncertainty of the completed value; the smaller the normalized entropy, the more concentrated the predicted probability distribution of the completed value, and the more reliable the completion result.

[0062] Figure 4 The specific process of step 500 is shown. This process includes steps 501 to 506.

[0063] Step 501: Summarize the completion uncertainty of the required features to be completed for candidate learning materials.

[0064] Step 502: Combining the completion uncertainty, the degree of overlap between the observed features of the user task and the input features required by the candidate learning device model, and the learning device quality information, calculate the sample-level learning device reliability score using a weighted summation method.

[0065] Step 503: Select a set of reliable learning materials to participate in the prediction based on the sample-level learning material reliability scores. In this embodiment, candidate learning materials with scores exceeding a preset threshold can be selected, or candidate learning materials with higher score rankings can be selected.

[0066] Step 504: Call the selected multiple reliable learning materials to output the prediction results, and perform weighted integration of the multiple prediction results.

[0067] Step 505: In the optional path, input the joint context and user-labeled samples into the table pre-trained model and perform end-to-end prediction.

[0068] Step 506: Output the user task prediction results, or combine the selective integration results and the end-to-end prediction results.

[0069] In the above-described medical form task embodiment, for samples to be predicted that have relatively complete vital sign information but many missing laboratory test indicators, candidates related to vital signs can be prioritized for prediction; for samples to be predicted that have relatively complete laboratory test indicators, candidates related to laboratory tests can be further included for prediction. Thus, the software base system can adaptively select candidate software based on the completeness and reliability of sample-level data completion.

[0070] This invention can be executed by a server, workstation, cloud computing node, or local computing device. The learning model, reduction, feature metadata, index alignment relations, joint context, and historical reliability statistics can be stored in a database or object storage; feature semantic vectors and reduction vectors can be stored in a vector database to support fast retrieval; the table-pretrained model can be deployed as an independent inference service.

[0071] The above embodiments disclose a method for reusing tabular learning components in heterogeneous feature spaces. This method integrates local variable relationships in the reduction of multiple candidate learning components through a global feature space and index alignment relationships, completes missing features of the user task and quantifies uncertainty through tabular context learning, and completes the user task through sample-level selective integration or end-to-end prediction with joint context. Techniques not detailed in this invention can be implemented using existing technologies. The above embodiments are merely preferred embodiments. Equivalent substitutions and combinations made by those skilled in the art to the above steps, parameters, model types, scoring functions, or fusion methods without departing from the concept of this invention should all fall within the protection scope of this invention.

[0072] Obviously, those skilled in the art should understand that the steps of the heterogeneous table learning component reuse method based on local variable relationship integration in the above embodiments of the present invention can be implemented using general-purpose computing devices. They can be centralized on a single computing device or distributed across a network of multiple computing devices. Optionally, they can be implemented using computer-executable program code, thereby storing them in a storage device for execution by a computing device. Furthermore, in some cases, the steps shown or described can be performed in a different order than presented here, or they can be fabricated as separate integrated circuit modules, or multiple modules or steps can be fabricated as a single integrated circuit module. Thus, the embodiments of the present invention are not limited to any particular hardware and software combination.

Claims

1. A method for reusing heterogeneous table learning materials based on the integration of local variable relationships, applied to a computing environment including a learning material base system, a developer's end, and a user's end, characterized in that, The method includes the following steps: Step 100: The user client generates a user task specification based on local table task data and submits it to the learning material base system; the learning material base system determines candidate learning materials from the existing learning materials in the system based on the specification information; Step 200: The learning material base system constructs a global feature space based on user task characteristics and local features of candidate learning materials, and establishes an index alignment relationship from the local feature space to the global feature space; Step 300: The learning material base system fills the global feature space with the candidate learning material specifications according to the index alignment relationship, generating a joint context; Step 400: Based on the joint context, the learning software base system determines the features to be completed that are not observed relative to the user samples in the input features of the candidate learning software model, completes the features to be completed, and calculates the completion uncertainty; Step 500: The learning device base system calculates the sample-level learning device reliability based on the completion uncertainty, selects learning devices that meet the reliability requirements for prediction integration, or performs end-to-end prediction based on the joint context.

2. The method for reusing heterogeneous table components based on the integration of local variable relationships according to claim 1, characterized in that, The candidate learning materials in step 100 are selected from all learning materials in the learning material base system through one or more of the following methods: semantic reduction screening, feature overlap ratio screening, task type screening, domain label screening, and causal feature screening; the candidate learning materials have a set degree of overlap or semantic relevance with the feature space of the target task.

3. The method for reusing heterogeneous table components based on the integration of local variable relationships according to claim 1, characterized in that, The core of step 200 is to organize user task features and candidate learning device local features into the same global feature space and record the index alignment relationship between the local features and the global feature positions. For user task features and candidate learning device local features that can be determined to correspond to the same variable, they are assigned to the same global feature position. For candidate learning device local features that fail to align with the existing global feature positions, the candidate learning device local features are added to the global feature space as new global features.

4. The method for reusing heterogeneous table components based on local variable relationship integration according to claim 1, characterized in that, Step 300 includes the following steps: Step 301: Read the specification of the candidate learning materials; Step 302: Fill the reduced samples into the global feature space according to the index alignment relationship from local features to global features; Step 303: Set missing values ​​and missing masks for global feature locations not covered by the current candidate learning materials; Step 304: Retain the reduced sample weights, source file identifiers, labels, or model output information; Step 305: Summarize the reduction samples by row to obtain the joint context, which includes global feature values, missing mask, reduction sample weights and source component identifiers.

5. The method for reusing heterogeneous table components based on local variable relationship integration according to claim 1, characterized in that, Step 400 includes the following steps: Step 401: For the sample to be predicted in the user task, compare the observed features of the sample to be predicted with the input features required by the candidate learning model, and determine the features required by the candidate learning model but not observed by the sample to be predicted, as the set of features to be completed. Step 402: Filter the observed context samples of the features to be completed from the joint context; Step 403: Construct a table completion task using the features to be completed as temporary prediction targets; Step 404: Call TabPFN or equivalent table pre-trained model to complete missing features; Step 405: Obtain the missing feature completion values ​​and their predicted probability distribution; Step 406: Calculate the normalized entropy of the predicted probability distribution to obtain the completed uncertainty.

6. The method for reusing heterogeneous table components based on local variable relationship integration according to claim 1, characterized in that, Step 500 includes the following steps: Step 501: Summarize the completion uncertainties of the required features to be completed for candidate learning materials; Step 502: Combining the completion uncertainty, the degree of overlap between the observed features of the user task and the input features required by the candidate learning device model, and the learning device quality information, calculate the sample-level learning device reliability score using a weighted summation method; Step 503: Select reliable learning devices to participate in the prediction based on the sample-level learning device reliability score; Step 504: Call the selected reliable learning software to output the prediction results, and perform weighted integration of multiple prediction results; Step 505: In the optional path, input the joint context and user-labeled samples into the table pre-trained model and perform end-to-end prediction; Step 506: Output the user task prediction results, or combine the selective integration results and the end-to-end prediction results.

7. A computer device, characterized in that: The computer device includes a memory, a processor, and a computer program stored in the memory and executable on the processor. When the processor executes the computer program, it implements the steps of the heterogeneous table learning software reuse method based on local variable relationship integration as described in any one of claims 1-6.

8. A computer-readable storage medium having a computer program / instructions stored thereon, characterized in that: When the computer program / instruction is executed by the processor, it implements the steps of the heterogeneous table learning component reuse method based on local variable relationship integration as described in any one of claims 1-6.

Images