Data table field filtering method and apparatus using data application result feedback

Abstract

The application relates to a data table field screening method and device using data application result feedback. The method comprises the following steps: taking type definition of a data application and a corresponding multi-dimensional evaluation index system; constructing a multi-objective collaborative optimization function which integrates data application precision optimization, application performance evaluation and field simplification requirements; solving the function through a self-adaptive iterative optimization algorithm, and outputting an optimal parameter solution when parameter convergence conditions and application performance convergence conditions are met simultaneously; determining field importance ranking and screening a key field subset based on L0 norm of the optimal parameter solution. The application deeply couples the field screening process and data application performance optimization in the same mathematical framework, solves the technical problem that the existing screening method is decoupled from the specific application target, realizes significant reduction of the field quantity and the calculation complexity while ensuring optimal application performance, and improves the accuracy and efficiency of the data application.

Description

Technical Field

[0001] This application relates to the field of data processing technology, and in particular to a method and apparatus for filtering data table fields using data application result feedback. Background Technology

[0002] With the development of big data and artificial intelligence technologies, data platforms have become core platforms for enterprises to integrate data resources and empower business applications. Data platforms typically store massive amounts of data containing numerous fields, which only realize their value through specific applications (such as prediction, classification, and regression). However, directly using all fields for data applications can lead to problems such as high computational complexity, high resource consumption, and model overfitting.

[0003] Existing field selection methods mainly include: methods based on relevance measures (such as the Pearson coefficient), methods based on feature space transformation (such as PCA), methods based on information criteria (such as AIC and BIC), and manual selection methods based on business experience. While these methods can reduce the number of fields to some extent, they have significant drawbacks: First, they are usually based on general statistical rules and are not linked to the specific data application goals, so the selected subset of fields may negatively impact application performance; second, methods that conduct experiments by exhaustively listing field combinations are computationally intensive, inefficient, and have poor generalization ability; finally, existing methods struggle to achieve the most concise field selection while ensuring optimal application performance.

[0004] Therefore, there is an urgent need in this field for a new method that can closely integrate with data application goals and achieve efficient field filtering while ensuring application performance. Summary of the Invention

[0005] Therefore, it is necessary to provide a method and apparatus for filtering data table fields based on the feedback of data application results, which can improve the accuracy of data application and reduce computational complexity, in order to address the above-mentioned technical problems.

[0006] A method for filtering data table fields based on data application results feedback, the method comprising: Obtain the type definition of the target data application and the corresponding multi-dimensional evaluation index system; Based on the dataset of fields to be filtered and the multi-dimensional evaluation index system, a multi-objective collaborative optimization function is constructed that integrates data application accuracy optimization, application performance evaluation, and field simplification requirements. The multi-objective collaborative optimization function deeply couples the field filtering process with the data application performance optimization process and solves them synchronously within the same mathematical optimization framework. The multi-objective collaborative optimization function is solved by an adaptive iterative optimization algorithm, and the optimal parameter solution is output when both the parameter convergence condition and the application performance convergence condition are satisfied. Based on the L0 norm of the parameter values ​​corresponding to each field in the optimal parameter solution, the importance ranking of the fields is determined and a subset of key fields is selected.

[0007] In one embodiment, the multi-objective collaborative optimization function is specifically expressed as:

[0008] in, Apply the target value vector to the data. This is a mapping function from field data to the application target. For field data matrix, Let be the parameter vector to be optimized. To maximize the objective function based on the aforementioned multi-dimensional evaluation index system, , For adjustable regularization parameters, It is a norm order and satisfies 0 <k<2。

[0009] In one embodiment, the maximization objective function Using a weighted summation method:

[0010] in, The evaluation index is normalized using the min-max standardization method.

[0011] in, and These are the theoretical maximum and minimum values ​​of the i-th evaluation index, respectively.

[0012] In one embodiment, the parameter convergence condition is defined as:

[0013] The application performance convergence condition is defined as follows:

[0014] in, Let be the parameter vector for the t-th iteration. This is the comprehensive evaluation value for the t-th iteration. This is the best overall assessment value in history. , This is the preset convergence threshold.

[0015] In one embodiment, the filtering of a subset of key fields based on the optimal parameter solution includes: Calculate the absolute value of the corresponding parameter for each field. ; Based on all The L0 norm of the value is used to determine the dynamic filtering threshold; Will Fields with values ​​lower than the dynamic filtering threshold are identified as redundant fields and removed. According to the remaining fields The values ​​are sorted from largest to smallest to form a subset of the key fields.

[0016] In one embodiment, the method for determining the dynamic screening threshold includes any of the following: based on The percentiles of the value distribution are determined; Or, based on The combination of the mean and standard deviation of the values ​​determines the result; Alternatively, it can be determined based on the results of cluster analysis.

[0017] In one embodiment, the method further includes: a validation step for the field filtering results. The data is applied using the aforementioned subset of key fields to calculate actual application performance indicators. The actual application performance indicators are compared and analyzed with the application performance indicators using the full field set; The effectiveness of field filtering is evaluated based on the comparison results, and the function parameters are adjusted and optimized as needed to re-filter the fields.

[0018] A data table field filtering device that utilizes data application result feedback, the device comprising: The application definition module is used to obtain the type definition of the target data application and the corresponding multi-dimensional evaluation index system; The function construction module is used to construct a multi-objective collaborative optimization function that integrates data application accuracy optimization, application performance evaluation, and field simplification requirements based on the data set of fields to be filtered and the multi-dimensional evaluation index system. The multi-objective collaborative optimization function deeply couples the field filtering process and the data application performance optimization process and solves them synchronously in the same mathematical optimization framework. The optimization solution module is used to solve the multi-objective collaborative optimization function through an adaptive iterative optimization algorithm, and output the optimal parameter solution when both the parameter convergence condition and the application performance convergence condition are satisfied. The field filtering module is used to determine the importance ranking of fields and filter out a subset of key fields based on the L0 norm of the parameter values ​​corresponding to each field in the optimal parameter solution.

[0019] A computer device includes a memory and a processor, the memory storing a computer program, and the processor executing the computer program performing the following steps: Obtain the type definition of the target data application and the corresponding multi-dimensional evaluation index system; Based on the dataset of fields to be filtered and the multi-dimensional evaluation index system, a multi-objective collaborative optimization function is constructed that integrates data application accuracy optimization, application performance evaluation, and field simplification requirements. The multi-objective collaborative optimization function deeply couples the field filtering process with the data application performance optimization process and solves them synchronously within the same mathematical optimization framework. The multi-objective collaborative optimization function is solved by an adaptive iterative optimization algorithm, and the optimal parameter solution is output when both the parameter convergence condition and the application performance convergence condition are satisfied. Based on the L0 norm of the parameter values ​​corresponding to each field in the optimal parameter solution, the importance ranking of the fields is determined and a subset of key fields is selected.

[0020] A computer-readable storage medium having a computer program stored thereon, the computer program performing the following steps when executed by a processor: Obtain the type definition of the target data application and the corresponding multi-dimensional evaluation index system; Based on the dataset of fields to be filtered and the multi-dimensional evaluation index system, a multi-objective collaborative optimization function is constructed that integrates data application accuracy optimization, application performance evaluation, and field simplification requirements. The multi-objective collaborative optimization function deeply couples the field filtering process with the data application performance optimization process and solves them synchronously within the same mathematical optimization framework. The multi-objective collaborative optimization function is solved by an adaptive iterative optimization algorithm, and the optimal parameter solution is output when both the parameter convergence condition and the application performance convergence condition are satisfied. Based on the L0 norm of the parameter values ​​corresponding to each field in the optimal parameter solution, the importance ranking of the fields is determined and a subset of key fields is selected.

[0021] The aforementioned data table field filtering method and apparatus that utilizes the feedback from data application results unifies the principles of field filtering efficiency, data application goal achievement, and optimal evaluation indicators into a multi-objective optimization problem. It simultaneously obtains the optimal solution for data application and field filtering results in a single calculation, effectively solving problems such as the decoupling of existing methods from specific applications, low computational efficiency, and poor generalization ability, and significantly improving the accuracy and efficiency of data application. Attached Figure Description

[0022] Figure 1 This is an application scenario diagram of a data table field filtering method that utilizes data application result feedback in one embodiment; Figure 2 This is a flowchart illustrating a data table field filtering method that utilizes data application result feedback in one embodiment. Figure 3This is a structural block diagram of a data table field filtering device that utilizes data application result feedback in one embodiment; Figure 4 This is an internal structural diagram of a computer device in one embodiment. Detailed Implementation

[0023] To make the objectives, technical solutions, and advantages of this application clearer, the following detailed description is provided in conjunction with the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are merely illustrative and not intended to limit the scope of this application.

[0024] The data table field filtering method provided in this application, which utilizes data application result feedback, can be applied to, for example... Figure 1 In the application environment shown, terminal 102 communicates with server 104 via a network. Server 104 has a data platform deployed on it, storing a large number of data tables and fields. Terminal 102 can request specific data application services, and server 104 performs field filtering and data application calculations. Terminal 102 can be, but is not limited to, a personal computer, laptop, smartphone, tablet, etc., and server 104 can be implemented using a standalone server or a server cluster.

[0025] In one embodiment, such as Figure 2 As shown, a method for filtering data table fields using data application results feedback is provided, and this method is applied to... Figure 1 Taking the server in the example, the following steps are included: Step 202: Obtain the type definition of the target data application and the corresponding multi-dimensional evaluation index system.

[0026] The type of target data application is defined by the user or automatically identified based on business needs, such as linear regression, data classification (e.g., logistic regression, decision tree classification), or data prediction (e.g., time series prediction). Based on the type of data application, the application evaluation metrics or set of metrics are specified, and a multi-dimensional evaluation metric system is constructed. For example, for linear regression, the evaluation metric could be mean squared error (MSE); for classification analysis, the evaluation metric could be classification accuracy; and for data prediction, the evaluation metric could be prediction accuracy.

[0027] Step 204: Solve the multi-objective collaborative optimization function using an adaptive iterative optimization algorithm, and output the optimal parameter solution when both the parameter convergence condition and the application performance convergence condition are satisfied.

[0028] This step, based on the dataset of fields to be filtered and the aforementioned multi-dimensional evaluation index system, constructs a multi-objective collaborative optimization function that integrates three optimization objectives: data application accuracy optimization objective, application efficiency evaluation objective, and field simplification requirement objective. These objectives respectively ensure the accuracy of data application results, ensure the optimal overall efficiency of data application, and ensure the minimum number of fields selected.

[0029] After integrating the three optimization objectives, the multi-objective collaborative optimization function deeply couples the field selection process with the data application performance optimization process, solving them synchronously within the same mathematical optimization framework.

[0030] Step 206: Solve the multi-objective collaborative optimization function using an adaptive iterative optimization algorithm, and output the optimal parameter solution when both the parameter convergence condition and the application performance convergence condition are satisfied.

[0031] Step 208: Based on the L0 norm of the parameter values ​​corresponding to each field in the optimal parameter solution, determine the importance ranking of the fields and filter out a subset of key fields.

[0032] The above-mentioned data table field filtering method that utilizes the feedback of data application results unifies the principles of field filtering efficiency, data application goal achievement, and optimal evaluation indicators into a multi-objective optimization problem. It obtains the optimal solution for data application and field filtering results simultaneously in one calculation, effectively solving the problems of existing methods being decoupled from specific applications, having low computational efficiency, and poor generalization ability, and significantly improving the accuracy and efficiency of data application.

[0033] To more clearly illustrate the technical solution of this invention, let's assume the data field is... The data corresponding to each field is , The set of indicators for evaluating the effectiveness of data application is as follows: (Established by the user according to different application needs), the corresponding indicators are not mutually exclusive, and the values ​​are non-negative, and the higher the better.

[0034] The value range of the above indicators was normalized using the following method, and all values ​​are within the range of... Within the range:

[0035] in, This represents the maximum value of the indicator. If the maximum value is infinity, then a larger value can be chosen, such as... replace; This is the minimum value of the indicator, which is usually 0.

[0036] The goal of data application is .in, For target values ​​in data applications, such as data fitting results or data classification results, express arrive The mapping can be linear or nonlinear. For the parameters that need to be solved during data application, and Correspondingly. This represents the error in the mapping process.

[0037] In one embodiment, the multi-objective collaborative optimization function is specifically expressed as:

[0038] in, Apply the target value vector to the data. This is a mapping function from field data to the application target. For field data matrix, Let be the parameter vector to be optimized. To maximize the objective function based on a multi-dimensional evaluation index system, , For adjustable regularization parameters, It is a norm order and satisfies 0 <k<2。

[0039] Specifically, the objective function for data application is:

[0040] The objective function indicates that the smaller the mapping error, the better. Wherein, express The square of the norm is the sum of the squares of the differences between the elements.

[0041] The objective function for maximizing the multi-dimensional evaluation index system is:

[0042] This indicates that the weighted sum of all indicators is maximized. Among them, The weight is , and its value range is . And the sum of all weights is 1.

[0043] While solving for the corresponding field values ​​can yield the best results by minimizing mapping error and maximizing evaluation metrics, the result may include all fields, failing to achieve the desired field filtering effect. Therefore, to minimize the number of fields, a field filtering objective function based on the principle of economy is constructed as follows:

[0044] in, , It can be the sum of the absolute values ​​of the field, or it can be a 0 or 1 value that indicates whether the field exists. express Norm, .when When the value is 0 or very small, and corresponding Need to be deleted, or indicate Not important. Theoretically, it should be adopted. time Norm, Norm refers to the number of non-zero elements in a vector. However... The norm is discontinuous and leads to NP-hard problems. If , Norm refers to the sum of the absolute values ​​of the elements in a vector. The optimal convex approximation of the norm. But here, for Generalization is performed to achieve more flexible and appropriate results.

[0045] In one embodiment, the objective function is maximized. Using a weighted summation method:

[0046] in, The evaluation index is normalized using the min-max standardization method.

[0047] in, and These are the theoretical maximum and minimum values ​​of the i-th evaluation index, respectively.

[0048] In another embodiment, the parameter convergence condition is defined as:

[0049] The convergence condition for application performance is defined as follows:

[0050] in, Let be the parameter vector for the nth iteration. This is the comprehensive evaluation value for the nth iteration. This is the best overall assessment value in history. , This is the preset convergence threshold.

[0051] Specifically, for the above objective function, an iterative method is typically used to solve it. The iterative solution method is a method of gradually approximating the solution; its idea is to obtain the solution by continuously approaching the extrema or zeros of the objective function. To obtain the solution that satisfies the above objective function... The value allows the objective function to be made more sensitive to the objective function. The partial derivatives are 0. Assume the objective function is expressed as... ,Right now

[0052] Or it can be expressed as: ; Thus, we can obtain information about The equation can be transformed into The form is given. To ensure the objective function is a convex optimization problem, Newton's iteration method is used for solving it. Let the initial value be... Substituting into the right side of the above equation, we get...

[0053] in, This represents the result obtained in the first iteration. The estimated value. And so on, the... The result obtained from the next iteration is:

[0054] If it exists

[0055] Then the iteration ends. For very small positive numbers, we can take... .

[0056] Application effect evaluation as an iteration termination condition The overall objective function can be written as:

[0057] in, The regularization parameter is a positive constant. The objective function can be solved using the method described above.

[0058] make The role of data application evaluation metrics is reflected in the iterative optimization process of the solution algorithm. Assume the... sequence The results obtained from the iteration are represented as follows: , , Let represent the maximum value obtained during the iterative solution process. Then: ,

[0059] As a termination condition for iteration. , For very small positive numbers, we can take... .

[0060] In one embodiment, filtering a subset of key fields includes: calculating the absolute value of the parameters corresponding to each field. Based on all The L0 norm of the value is used to determine the dynamic filtering threshold; Fields with values ​​below the dynamic filtering threshold are identified as redundant and removed; the remaining fields are then processed according to... The values ​​are sorted from largest to smallest to form a subset of the key fields.

[0061] In another embodiment, the method for determining the dynamic screening threshold includes: based on The percentiles of the value distribution are determined.

[0062] In one embodiment, the verification steps for the field filtering results are as follows: data application is performed using a subset of key fields to calculate the actual application performance index; the actual application performance index is compared and analyzed with the application performance index using the full set of fields; the field filtering effect is evaluated based on the comparison results, and the field filtering is performed again by adjusting the optimization function parameters as needed.

[0063] Taking linear regression as an example, linear regression is a statistical analysis method that uses regression analysis in mathematical statistics to determine the quantitative relationship of interdependence between two or more variables. Let... ,in, The performance evaluation index is the mean squared error. :

[0064] in for The estimated value can be used as part of the objective or as part of the iteration stopping condition. The objective function and solution process described above can be used directly. If... If it is very small, it indicates that, given the existing data, the field... This is not important and can be excluded from linear regression calculations. This case is a special case of the above model.

[0065] Take classification analysis as an example. Classification analysis refers to an algorithm learning to derive "a certain classification criterion," which can determine the category of a new thing. Assume the set of data classification labels is... The data used for classification are x , z , The initial value is 1. The evaluation metric is classification accuracy, i.e., the predicted value. Compared with the true value Matching consistency, if If the classification is correct, the precision is incremented by 1; otherwise, the precision remains unchanged. If it is very small, it indicates that, given the existing data, the field... It's not important and can be excluded from the classification calculation.

[0066] Take data prediction as an example. Data prediction refers to the process of building a prediction model based on existing data samples, training the model on the samples, and then predicting a specific field of the sample to be processed. Assume the set of true values ​​is... The data used for prediction are x , z , The initial value is 1. The evaluation metric is prediction accuracy, i.e., the predicted value. If the prediction is correct, the accuracy is incremented by 1; otherwise, the accuracy remains unchanged. The threshold can be set to 1 / 20 of the absolute value of the true value. If it is very small, it indicates that, given the existing data, the field... It's not important and can be excluded from prediction calculations.

[0067] In practical applications, the data platform stores data from drone manufacturers. A relational database is used to store drone production process data, including order data, production material procurement data, material in / out data, finished product in / out data, production process data, transportation data, and other types of production cycle data. Each type of data has dozens to hundreds of fields. It is assumed that there is a linear relationship between the longest and shortest drone production and delivery times and these various types of production cycle data, but the specific parameters are uncertain. The evaluation metric is set as MSE (Minimum Search Engine Optimization). Used to indicate whether a field participates, 0 indicates no participation, 1 indicates participation, and the initial value of each is 1.

[0068] Input: A relational data table containing fields and records, specifying the data application type, such as linear fitting.

[0069] Processing procedure: 1. Based on the characteristics of the data application, retrieve the evaluation indicator system corresponding to the data application type. Here is the simplest evaluation indicator, namely MSE; 2. Based on the data application type and the established evaluation index system, construct the target optimization model and define the termination conditions in the iteration process, see Part 4 for detailed steps of the technical solution; 3. Solve the objective function using an iterative optimization method to obtain the optimal solution for data application, and calculate the data for each field. Norm; 4. Based on field data The norm value is used to remove the corresponding fields and retain the remaining fields as the optimal field combination. Output: The optimal combination of fields for data application, used to fit and predict the longest and shortest time for drone production and delivery.

[0070] It should be understood that, although Figure 2 The steps in the flowchart are shown sequentially as indicated by the arrows, but these steps are not necessarily executed in the order indicated by the arrows. Unless otherwise specified herein, there is no strict order in which these steps are executed, and they can be performed in other orders. Figure 2 At least some of the steps in the process may include multiple sub-steps or multiple stages. These sub-steps or stages are not necessarily completed at the same time, but can be executed at different times. The execution order of these sub-steps or stages is not necessarily sequential, but can be executed in turn or alternately with other steps or at least some of the sub-steps or stages of other steps.

[0071] In one embodiment, such as Figure 3 As shown, a data table field filtering device that utilizes data application result feedback is provided, comprising: Application definition module 302 is used to obtain the type definition of the target data application and the corresponding multi-dimensional evaluation index system; The function construction module 304 is used to construct a multi-objective collaborative optimization function that integrates data application accuracy optimization, application performance evaluation, and field simplification requirements based on the data set of fields to be filtered and the multi-dimensional evaluation index system; wherein, the multi-objective collaborative optimization function deeply couples the field filtering process and the data application performance optimization process and solves them synchronously in the same mathematical optimization framework; The optimization solution module 306 is used to solve the multi-objective collaborative optimization function through an adaptive iterative optimization algorithm, and output the optimal parameter solution when both the parameter convergence condition and the application performance convergence condition are satisfied. The field filtering module 308 is used to determine the importance ranking of fields and filter out a subset of key fields based on the L0 norm of the parameter values ​​corresponding to each field in the optimal parameter solution.

[0072] Specific limitations regarding the data table field filtering device utilizing data application result feedback can be found in the limitations of the data table field filtering method utilizing data application result feedback described above, and will not be repeated here. Each module in the aforementioned data table field filtering device utilizing data application result feedback can be implemented entirely or partially through software, hardware, or a combination thereof. These modules can be embedded in or independent of the processor in the computer device in hardware form, or stored in the memory of the computer device in software form, so that the processor can call and execute the corresponding operations of each module.

[0073] In one embodiment, a computer device is provided, which may be a server, and its internal structure diagram may be as follows: Figure 4 As shown, the computer device includes a processor, memory, network interface, and database connected via a system bus. The processor provides computing and control capabilities. The memory includes non-volatile storage media and internal memory. The non-volatile storage media stores the operating system, computer programs, and database. The internal memory provides the environment for the operating system and computer programs in the non-volatile storage media to run. The database stores data such as the type definition of the target data application, the corresponding multi-dimensional evaluation index system, and the set of fields to be filtered. The network interface is used to communicate with external terminals via a network connection. When the computer program is executed by the processor, it implements a data table field filtering method that utilizes feedback from data application results.

[0074] Those skilled in the art will understand that Figure 4 The structure shown is merely a block diagram of a portion of the structure related to the present application and does not constitute a limitation on the computer device to which the present application is applied. Specific computer devices may include more or fewer components than those shown in the figure, or combine certain components, or have different component arrangements.

[0075] In one embodiment, a computer device is provided, including a memory and a processor, the memory storing a computer program, the processor executing the computer program to implement the steps of the method described above.

[0076] In one embodiment, a computer-readable storage medium is provided having a computer program stored thereon, which, when executed by a processor, implements the steps of the method described above.

[0077] Those skilled in the art will understand that all or part of the processes in the methods of the above embodiments can be implemented by a computer program instructing related hardware. The computer program can be stored in a non-volatile computer-readable storage medium. When executed, the computer program can include the processes of the embodiments of the above methods. Any references to memory, storage, databases, or other media used in the embodiments provided in this application can include non-volatile and / or volatile memory. Non-volatile memory may include read-only memory (ROM), programmable ROM (PROM), electrically programmable ROM (EPROM), electrically erasable programmable ROM (EEPROM), or flash memory. Volatile memory may include random access memory (RAM) or external cache memory. By way of illustration and not limitation, RAM is available in a variety of forms, such as static RAM (SRAM), dynamic RAM (DRAM), synchronous DRAM (SDRAM), dual data rate SDRAM (DDRSDRAM), enhanced SDRAM (ESDRAM), synchronous link DRAM (SLDRAM), RAMbus direct RAM (RDRAM), direct memory bus dynamic RAM (DRDRAM), and memory bus dynamic RAM (RDRAM), etc.

[0078] The technical features of the above embodiments can be combined in any way. For the sake of brevity, not all possible combinations of the technical features in the above embodiments are described. However, as long as there is no contradiction in the combination of these technical features, they should be considered to be within the scope of this specification.

[0079] The embodiments described above are merely illustrative of several implementation methods of this application, and while the descriptions are specific and detailed, they should not be construed as limiting the scope of the invention. It should be noted that those skilled in the art can make various modifications and improvements without departing from the concept of this application, and these all fall within the protection scope of this application. Therefore, the protection scope of this application should be determined by the appended claims.

Claims

1. A method for filtering data table fields using feedback from data application results, characterized in that, The method includes: Obtain the type definition of the target data application and the corresponding multi-dimensional evaluation index system; Based on the dataset of fields to be filtered and the multi-dimensional evaluation index system, a multi-objective collaborative optimization function is constructed that integrates data application accuracy optimization, application performance evaluation, and field simplification requirements. The multi-objective collaborative optimization function deeply couples the field filtering process with the data application performance optimization process and solves them synchronously within the same mathematical optimization framework. The multi-objective collaborative optimization function is solved by an adaptive iterative optimization algorithm, and the optimal parameter solution is output when both the parameter convergence condition and the application performance convergence condition are satisfied. Based on the L0 norm of the parameter values ​​corresponding to each field in the optimal parameter solution, the importance ranking of the fields is determined and a subset of key fields is selected.

2. The method according to claim 1, characterized in that, The multi-objective collaborative optimization function is specifically expressed as follows: in, Apply the target value vector to the data. This is a mapping function from field data to the application target. For field data matrix, Let be the parameter vector to be optimized. To maximize the objective function based on the aforementioned multi-dimensional evaluation index system, , For adjustable regularization parameters, It is a norm order and satisfies 0 <k<2。 3. The method according to claim 2, characterized in that, The maximization objective function Using a weighted summation method: in, The evaluation index is normalized using the min-max standardization method. in, and These are the theoretical maximum and minimum values ​​of the i-th evaluation index, respectively.

4. The method according to claim 1, characterized in that, The convergence condition for the parameters is defined as follows: The application performance convergence condition is defined as follows: in, Let be the parameter vector for the nth iteration. This is the comprehensive evaluation value for the nth iteration. This is the best overall assessment value in history. , This is the preset convergence threshold.

5. The method according to claim 1, characterized in that, The subset of key fields selected based on the optimal parameter solution includes: Calculate the absolute value of the corresponding parameter for each field. ; Based on all The L0 norm of the value is used to determine the dynamic filtering threshold; Will Fields with values ​​lower than the dynamic filtering threshold are identified as redundant fields and removed. According to the remaining fields The values ​​are sorted from largest to smallest to form a subset of the key fields.

6. The method according to claim 5, characterized in that, The method for determining the dynamic screening threshold includes: based on The percentiles of the value distribution are determined.

7. The method according to claim 1, characterized in that, The method also includes: a verification step for the field filtering results: The data is applied using the aforementioned subset of key fields to calculate actual application performance indicators. The actual application performance indicators are compared and analyzed with the application performance indicators using the full field set; The effectiveness of field filtering is evaluated based on the comparison results, and the function parameters are adjusted and optimized as needed to re-filter the fields.

8. A data table field filtering device that utilizes data application result feedback, characterized in that, The device includes: The application definition module is used to obtain the type definition of the target data application and the corresponding multi-dimensional evaluation index system; The function construction module is used to construct a multi-objective collaborative optimization function that integrates data application accuracy optimization, application performance evaluation, and field simplification requirements based on the data set of fields to be filtered and the multi-dimensional evaluation index system. The multi-objective collaborative optimization function deeply couples the field filtering process and the data application performance optimization process and solves them synchronously in the same mathematical optimization framework. The optimization solution module is used to solve the multi-objective collaborative optimization function through an adaptive iterative optimization algorithm, and output the optimal parameter solution when both the parameter convergence condition and the application performance convergence condition are satisfied. The field filtering module is used to determine the importance ranking of fields and filter out a subset of key fields based on the L0 norm of the parameter values ​​corresponding to each field in the optimal parameter solution.

Images