Data processing method and device, computer equipment and storage medium

Abstract

The invention relates to a data processing method and device, computer equipment and a storage medium. The method comprises the following steps: acquiring an original data set, and acquiring a first constraint related to resource conversion processing; determining a precision scaling factor based on the data precision of the original data set and the current data application scene; obtaining a to-be-processed data set according to the original data set, and converting the to-be-processed data set into a preprocessed data set through the precision scaling factor; acquiring and analyzing a data application rule corresponding to the original data set to obtain a second constraint; based on the first constraint and the second constraint, selecting data in the preprocessed data set for combination and constraint verification to obtain a plurality of candidate solutions, and constructing a candidate solution pool; and selecting a target candidate solution from the candidate solution pool based on a preset screening rule, and generating a corresponding data processing result. By adopting the method, various constraint conditions can be fused and adapted, and the calculation precision and efficiency are considered, so that an accurate data processing result is efficiently generated.

Description

Technical Field

[0001] This application relates to the field of data processing technology, and in particular to a data processing method, apparatus, computer equipment, and storage medium. Background Technology

[0002] In scenarios involving multi-entity collaboration or multi-system interaction, numerous bidirectional or multidirectional resource flows often exist between nodes, such as computational task scheduling, energy exchange, data interaction, material allocation, or task load balancing. In such scenarios, processing each resource flow independently not only leads to redundant transmission and allocation but also results in high resource occupancy, slow system response, and increased overall processing costs. To improve system efficiency, existing technologies typically perform netting and combination processing on multiple resource flows, while meeting resource application scenario constraints and security control requirements. This reduces the number of transactions processed and the actual resource allocation, thereby improving the overall system throughput and availability.

[0003] However, existing resource conversion and processing technologies still suffer from the contradiction between computational accuracy and processing efficiency, as well as difficulties in integrating multiple constraints. Specifically, on the one hand, in large-scale systems, resource values ​​between nodes are usually represented by high-precision data. Traditional dynamic programming algorithms experience a sharp increase in computational load and memory consumption under high-precision conditions, while greedy algorithms, although able to solve problems quickly, cannot guarantee the global optimality of the result. On the other hand, the resource conversion process needs to simultaneously satisfy multi-dimensional constraints such as the upper limit of object capacity, resource retention ratio, task dependencies, and execution order rules. Existing solutions generally use static configuration or fixed parameter loading, lacking the ability to dynamically integrate and adaptively adjust for different scenarios, making it difficult to balance computational efficiency and constraint compliance. Summary of the Invention

[0004] Therefore, it is necessary to provide a data processing method, apparatus, computer equipment, and storage medium that can integrate and adapt to multiple constraints, balance computational accuracy and efficiency, and thus efficiently generate accurate data processing results, in order to address the above-mentioned technical problems.

[0005] On the one hand, a data processing method is provided, the method comprising: Obtain the original dataset and the first constraint related to resource transformation processing; Based on the data precision of the original dataset and the current data application scenario, determine the precision scaling factor; The dataset to be processed is obtained from the original dataset, and the dataset to be processed is converted into a preprocessed dataset using the precision scaling factor. Obtain and parse the data application rules corresponding to the original dataset to obtain the second constraint; Based on the first constraint and the second constraint, data from the preprocessed dataset are selected for combination and constraint verification to obtain multiple candidate solutions and construct a candidate solution pool; Based on preset filtering rules, target candidate solutions are selected from the candidate solution pool, and corresponding data processing results are generated based on the target candidate solutions.

[0006] In one embodiment, obtaining the original dataset and obtaining the first constraint related to the resource transformation process includes: Read the target database to obtain multiple data within the target period to construct the original dataset, and determine multiple data objects and their corresponding input and output resources; Request resource data for multiple data objects from the resource management library; in response to receiving resource data from the resource management library, parse the resource data to obtain the available resources for the multiple data objects. Based on the available resources, the input resources, and the output resources, the predicted actual available resource values ​​of multiple data objects are obtained; Acquire and parse preset data application scenario parameters to obtain the resource retention ratio and the upper limit of a single resource conversion; Based on the predicted actual available resource value and the resource retention ratio, generate resource conversion upper limits for multiple data objects; Based on the resource conversion upper limit and the resource single conversion upper limit, corresponding resource conversion constraint parameters are generated, and the resource conversion constraint parameters are used as the first constraint.

[0007] In one embodiment, determining the precision scaling factor based on the data precision of the original dataset and the current data application scenario includes: Iterate through the resource value fields in the original dataset to obtain the maximum absolute value of the resource value; Obtain the maximum absolute value of the resource conversion limit and compare it with the maximum absolute value of the resource value, and select the larger value as the benchmark value; Identify the current data application scenario based on the preset scenario configuration table, determine the corresponding resource conversion accuracy level, and generate an initial accuracy scaling factor. Based on the initial precision scaling factor, the reference value is integerized to obtain the reference integer value; Compare the baseline integer value with the upper limit of integer values ​​that the processor can process; If the reference integer value is greater than or equal to the upper limit of the integer value that the processor can process, the initial precision scaling factor is reduced based on the reduction coefficient to obtain the corrected precision scaling factor, until the reference integer value is less than the upper limit of the integer value that the processor can process. If the reference integer value is less than the upper limit of the integer value that the processor can process, then the current precision scaling factor is used as the precision scaling factor.

[0008] In one embodiment, obtaining the dataset to be processed based on the original dataset and converting the dataset to be processed into a preprocessed dataset using the precision scaling factor includes: Extract the valid data from the original dataset and perform field mapping processing to obtain the dataset to be processed; The resource fields in the dataset to be processed are standardized to obtain a standard dataset; The resource fields in the standard dataset are scaled using the precision scaling factor to obtain the preprocessed dataset consisting of multiple integer resource values.

[0009] In one embodiment, obtaining and parsing the data application rules corresponding to the original dataset to obtain the second constraint includes: Based on the current data application scenario, query historical scenario records and obtain the historical constraint set corresponding to the current data application scenario. The historical constraint set includes multiple historical constraint entries generated and verified under the same or similar scenarios. Validity checks and logical conflict checks are performed sequentially on multiple historical constraint entries to remove invalid and logically conflicting constraints. The remaining historical constraint entries are then merged and their boundaries are tightened to obtain a set of historical constraints. In response to receiving resource application rules input by the user, the resource application rules are formatted and mapped to obtain a structured user constraint set; The historical constraint set and the structured user constraint set are merged to obtain a fused constraint set. The constraint entries in the fused constraint set are then prioritized and conflict-resolved to obtain a usable fused constraint set. The second constraint is obtained based on the constraint entries in the available fusion constraint set.

[0010] In one embodiment, the step of selecting data from the preprocessed dataset based on the first constraint and the second constraint, combining and verifying the constraints to obtain multiple candidate solutions, and constructing a candidate solution pool includes: Multiple data sets to be combined are extracted from the preprocessed dataset, and based on the calculated data similarity between the multiple data sets to be combined, the multiple data sets to be combined are grouped to generate multiple initial data combinations; Based on the first constraint and the second constraint, multiple combinations of the initial data are verified. If all the initial data combinations satisfy the first constraint and the second constraint, then the initial data combinations are determined to be valid combinations. If the initial data combination does not satisfy one or more of the first constraint or the second constraint, the initial data combination is determined to be an illegal combination. The initial data combination is then broken down into multiple data to be combined and recombined with at least one data to be combined that is not in the initial data combination to obtain a recombined data combination, until the recombined data combination is a legal combination. Multiple legal combinations are used as multiple candidate solutions. A corresponding combination score value is calculated for each of the multiple candidate solutions. The multiple candidate solutions are sorted in descending order based on the combination score values ​​to obtain a candidate solution sequence. The combination score value is obtained by weighting the resource conversion efficiency, resource occupancy rate, exposure risk and resource conversion coverage of the current data combination. Based on the total number of data, resource value distribution, and resource dispersion of the dataset to be processed, set the upper limit of the candidate solution pool; Based on the upper limit of the candidate solution pool, multiple candidate solutions with a number less than or equal to the upper limit of the candidate solution pool are selected sequentially from the candidate solution sequence and stored as the candidate solution pool.

[0011] In one embodiment, the step of selecting a target candidate solution from the candidate solution pool based on a preset filtering rule, and generating a corresponding data processing result based on the target candidate solution, includes: Based on the current number of available threads and the average processing latency of candidate solutions, determine the concurrency level of the current round, and set the current number of solutions based on the current round concurrency level; According to the preset filtering rules, a number of candidate solutions not exceeding the current number of solutions are selected sequentially from the front end of the candidate solution pool to form the current target candidate solution set; If there is one or more of the following issues among the candidate solutions in the current target candidate solution set: data object overlap, resource conflict, or execution order mutual exclusion, then the current target candidate solution set is determined to satisfy intra-batch consistency. Based on one or more of the preset priority strategy, alternative solution strategy, or local rearrangement strategy, conflict resolution is performed on the current target candidate solution set until the current target candidate solution set satisfies intra-batch consistency. Take a number of candidate solutions from the current target candidate solution set as the target candidate solution, and determine the corresponding target data combination based on the target candidate solution; The resource values ​​in the target data combination are reverse-converted according to the scaling factor to obtain the target resource conversion value, and the corresponding data processing result is generated.

[0012] On the other hand, a data processing apparatus is provided, the apparatus comprising: The data acquisition module is used to acquire the raw dataset and obtain the first constraint related to resource transformation processing; The scaling determination module is used to determine the precision scaling factor based on the data precision of the original dataset and the current data application scenario; The data processing module is used to obtain the dataset to be processed based on the original dataset, and to convert the dataset to be processed into a preprocessed dataset using the precision scaling factor. The data constraint module is used to acquire and parse the data application rules corresponding to the original dataset to obtain the second constraint; The candidate solution pool construction module is used to perform combined search and constraint verification on the preprocessed dataset based on the first constraint and the second constraint to obtain multiple candidate solutions and construct a candidate solution pool. The result generation module is used to select target candidate solutions from the candidate solution pool based on preset filtering rules, and generate corresponding data processing results based on the target candidate solutions.

[0013] In another aspect, a computer device is provided, including a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor executes the computer program to perform the following steps: Obtain the original dataset and the first constraint related to resource transformation processing; Based on the data precision of the original dataset and the current data application scenario, determine the precision scaling factor; The dataset to be processed is obtained from the original dataset, and the dataset to be processed is converted into a preprocessed dataset using the precision scaling factor. Obtain and parse the data application rules corresponding to the original dataset to obtain the second constraint; Based on the first constraint and the second constraint, data from the preprocessed dataset are selected for combination and constraint verification to obtain multiple candidate solutions and construct a candidate solution pool; Based on preset filtering rules, target candidate solutions are selected from the candidate solution pool, and corresponding data processing results are generated based on the target candidate solutions.

[0014] In another aspect, a computer-readable storage medium is provided having a computer program stored thereon, which, when executed by a processor, performs the following steps: Obtain the original dataset and the first constraint related to resource transformation processing; Based on the data precision of the original dataset and the current data application scenario, determine the precision scaling factor; The dataset to be processed is obtained from the original dataset, and the dataset to be processed is converted into a preprocessed dataset using the precision scaling factor. Obtain and parse the data application rules corresponding to the original dataset to obtain the second constraint; Based on the first constraint and the second constraint, data from the preprocessed dataset are selected for combination and constraint verification to obtain multiple candidate solutions and construct a candidate solution pool; Based on preset filtering rules, target candidate solutions are selected from the candidate solution pool, and corresponding data processing results are generated based on the target candidate solutions.

[0015] The aforementioned data processing method, apparatus, computer equipment, and storage medium, by acquiring the original dataset and extracting the first constraint related to resource conversion processing, achieve dynamic modeling of the resource status and allocation constraints of data objects, enabling resource conversion calculations to possess high real-time performance and adaptability. Secondly, by determining the precision scaling factor based on data precision and application scenarios, and performing integer preprocessing on the original data, the memory consumption and computational complexity caused by high-precision floating-point calculations are effectively reduced, significantly improving processing efficiency while ensuring controllable result accuracy. Furthermore, by parsing the corresponding data application rules during resource conversion and generating the second constraint, the system can simultaneously consider task priority, allocation rules, and risk control requirements during the solution process, achieving dynamic constraint fusion for different resource application scenarios. Simultaneously, based on… The first and second constraints perform combined operations and constraint verification on the preprocessed dataset, generating candidate solutions that satisfy multi-dimensional constraints. This ensures that the subsequently generated resource conversion schemes are executable in terms of logical consistency and constraint compliance. Furthermore, by dynamically filtering candidate solutions based on preset filtering rules and outputting the corresponding data processing results, the system can generate optimal or near-optimal resource conversion results while ensuring legality and consistency. Thus, this application, by constructing an integrated data processing flow of "constraint extraction—precision scaling—rule fusion—consistency verification—dynamic filtering," not only solves the contradiction between accuracy and efficiency in resource conversion calculations in the prior art, but also achieves adaptive fusion of multiple constraints and generation of optimal results, significantly improving the computational efficiency, stability, and system scalability of resource conversion processing. Attached Figure Description

[0016] To more clearly illustrate the embodiments of this application, the accompanying drawings used in the embodiments will be briefly introduced below. Obviously, the drawings described below are only some embodiments of this application. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0017] Figure 1This is a diagram illustrating the application environment of a data processing method in one embodiment. Figure 2 This is a flowchart illustrating a data processing method in one embodiment; Figure 3 This is a structural block diagram of a data processing device in one embodiment; Figure 4 This is an internal structural diagram of a computer device in one embodiment. Detailed Implementation

[0018] To make the objectives, technical solutions, and advantages of this application clearer, the following detailed description is provided in conjunction with the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are merely illustrative and not intended to limit the scope of this application.

[0019] The data processing method provided in this application can be applied to, for example... Figure 1 In the application environment shown, terminal 102 communicates with server 104 via a network to receive and display information such as resource conversion calculation results, data application rule configuration, and processing results. Terminal 102 may include, but is not limited to, personal computers, laptops, smartphones, tablets, or wearable devices; server 104 may be a single server or a server cluster composed of multiple servers to achieve high-concurrency computing and distributed data storage. In this application environment, server 104 can execute the data processing method of claim 1 of this application. Its overall process includes: obtaining the original dataset within the target processing cycle and the first constraints related to data objects and resource allocation; determining the precision scaling factor based on data precision and scenario parameters, and performing integer or discretization preprocessing on the original data; parsing the resource application rules and historical constraint information configured by the user, and generating the corresponding second constraints; performing resource combination calculation and constraint verification on the preprocessed data under the combined conditions of the first and second constraints, and constructing a candidate solution set; then, according to preset screening rules, selecting the optimal or near-optimal solution from the candidate solution set, and generating resource conversion results that can be directly used for execution by the business system. Terminal 102 can interact with server 104 through a graphical interface or application programming interface (API) to submit resource application rule configurations, trigger resource conversion calculation tasks, query data processing results, and confirm, archive, or distribute the generated results as needed. Server 104 is responsible for specific algorithm execution and data processing, including data parsing, constraint fusion, combined solution, concurrent filtering, and result output. Through the above methods, the data processing method of this application can achieve efficient and compliant generation of resource conversion results in high-precision, large-scale resource interaction scenarios, and supports distributed business deployment with multiple terminals and multiple users.

[0020] In one embodiment, such as Figure 2 As shown, a data processing method is provided, which can be applied to... Figure 1 Taking server 104 as an example, the following steps are included: Step 201: Obtain the original dataset and the first constraint related to resource transformation processing; Step 202: Determine the precision scaling factor based on the data precision of the original dataset and the current data application scenario; Step 203: Obtain the dataset to be processed from the original dataset, and convert the dataset to be processed into a preprocessed dataset by using a precision scaling factor; Step 204: Obtain and parse the data application rules corresponding to the original dataset to obtain the second constraint; Step 205: Based on the first and second constraints, select data from the preprocessed dataset for combination and constraint verification to obtain multiple candidate solutions and construct a candidate solution pool; Step 206: Select a target candidate solution from the candidate solution pool based on the preset filtering rules, and generate the corresponding data processing results based on the target candidate solution.

[0021] The aforementioned data processing method, apparatus, computer equipment, and storage medium, by acquiring the original dataset and extracting the first constraint related to resource conversion processing, achieve dynamic modeling of the resource status and allocation constraints of data objects, enabling resource conversion calculations to possess high real-time performance and adaptability. Secondly, by determining the precision scaling factor based on data precision and application scenarios, and performing integer preprocessing on the original data, the memory consumption and computational complexity caused by high-precision floating-point calculations are effectively reduced, significantly improving processing efficiency while ensuring controllable result accuracy. Furthermore, by parsing the corresponding data application rules during resource conversion and generating the second constraint, the system can simultaneously consider task priority, allocation rules, and risk control requirements during the solution process, achieving dynamic constraint fusion for different resource application scenarios. Simultaneously, based on… The first and second constraints perform combined operations and constraint verification on the preprocessed dataset, generating candidate solutions that satisfy multi-dimensional constraints. This ensures that the subsequently generated resource conversion schemes are executable in terms of logical consistency and constraint compliance. Furthermore, by dynamically filtering candidate solutions based on preset filtering rules and outputting the corresponding data processing results, the system can generate optimal or near-optimal resource conversion results while ensuring legality and consistency. Thus, this application, by constructing an integrated data processing flow of "constraint extraction—precision scaling—rule fusion—consistency verification—dynamic filtering," not only solves the contradiction between accuracy and efficiency in resource conversion calculations in the prior art, but also achieves adaptive fusion of multiple constraints and generation of optimal results, significantly improving the computational efficiency, stability, and system scalability of resource conversion processing.

[0022] In one embodiment, the original dataset is obtained, and a first constraint related to the resource transformation process is obtained, including: Read the target database to obtain multiple data points within the target period to construct the original dataset, and determine multiple data objects and their corresponding input and output resources; Request resource data for multiple data objects from the resource management library. In response to receiving the resource data from the resource management library, parse the resource data to obtain the available resources for the multiple data objects. Based on available resources, required input resources, and required output resources, the predicted actual available resource values ​​for multiple data objects are obtained; Obtain and parse preset data application scenario parameters to obtain the resource retention ratio and the upper limit of a single resource conversion. The resource retention ratio can be set to 5%~20% according to the data application scenario, and the upper limit of a single resource conversion can be dynamically adjusted according to the system resource limit. Based on the predicted actual available resource value and resource retention ratio, generate resource conversion limits for multiple data objects; Based on the upper limit of resource conversion and the upper limit of single resource conversion, corresponding resource conversion constraint parameters are generated, and the resource conversion constraint parameters are used as the first constraint.

[0023] In one specific embodiment, the calculation of the predicted actual available resource value is expressed as follows: Among them, P i Let R represent the predicted actual available resource value of the i-th data object, where Ai represents the available resource of the i-th data object, and R represents the available resource of the i-th data object. i This represents the resource to be applied to the i-th data object. This represents the resource to be provided by the i-th data object; The resource conversion cap is generated as follows: Among them, L i The resource conversion limit for the i-th data object is represented by M, the resource retention ratio is represented by C. max This indicates the maximum number of resource conversions per transaction.

[0024] Specifically, in this embodiment, multiple data points within the target period are obtained by reading the target database, and combined with the resource data of each data object in the resource management library, the actual available resource value and the upper limit of resource conversion are calculated and predicted, thereby realizing dynamic evaluation of the resource status of data objects. At the same time, the resource retention ratio and the upper limit of single resource conversion are obtained by parsing the data application scenario parameters, and resource conversion constraint parameters are generated as the first constraint, enabling the system to automatically adjust the resource conversion limit according to the real-time resource situation and risk parameters. Thus, the problem of constraint lag or failure under the traditional static configuration method is effectively avoided, thereby improving the accuracy and adaptability of resource conversion constraints and providing a dynamically feasible constraint basis for subsequent combinatorial solutions.

[0025] In one embodiment, the precision scaling factor is determined based on the data precision of the original dataset and the current data application scenario, including: Iterate through the resource value fields in the original dataset to obtain the maximum absolute value of the resource value; Get the maximum absolute value of the resource conversion limit and compare it with the maximum absolute value of the resource value, and select the larger value as the benchmark value; The system identifies the current data application scenario based on the preset scenario configuration table, determines the corresponding resource conversion accuracy level, and generates an initial accuracy scaling factor. The preset scenario configuration table includes parameters such as scenario identifier, resource level, and system processing threshold, which are used to automatically match the calculation accuracy under different resource application scenarios, i.e., automatically determine the initial accuracy scaling factor. Compare the base integer value with the upper limit of integer values ​​that the processor can process; If the base integer value is greater than or equal to the upper limit of the integer value that the processor can process, the initial precision scaling factor is reduced based on the reduction factor to obtain the corrected precision scaling factor until the base integer value is less than the upper limit of the integer value that the processor can process. The reduction factor is preferably -1, that is, when the base integer value exceeds the limit, the initial precision scaling factor is reduced by a step of 1 to reduce the magnification of the resource value. If the base integer value is less than the upper limit of the integer values ​​that the processor can process, then the current precision scaling factor will be used as the precision scaling factor.

[0026] It is worth noting that the scenarios corresponding to the scenario identifier can include, but are not limited to, high-precision energy scheduling scenarios, resource allocation scenarios, data synchronization scenarios, or task assignment scenarios. For example, when the scenario identifier corresponds to a high-precision energy scheduling scenario, the system needs to accurately track changes in energy units. In this case, the initial precision scaling factor is preferentially set to 4, which means a magnification of 104 times, to meet the high-precision computing requirements. When the scenario identifier corresponds to a large-scale task assignment or resource allocation scenario, the precision scaling factor can be dynamically adjusted according to the system processing threshold and real-time load to balance computing accuracy and processing efficiency.

[0027] Specifically, in this embodiment, the precision scaling factor is determined based on the data precision of the original dataset and the current data application scenario, thereby significantly reducing the data size while ensuring resource precision. At the same time, a baseline value is determined by traversing resource values ​​and resource conversion upper limits, and the precision scaling factor is progressively corrected according to the integer range that the processor can process, so that resource conversion calculations can be executed efficiently within the integer space. Thus, while ensuring controllable accuracy of the calculation results, the problem of excessive memory consumption and excessive computation time that dynamic programming algorithms in the prior art are prone to in high-precision resource scenarios is effectively solved, thereby achieving a balanced optimization of calculation precision and processing efficiency.

[0028] In one embodiment, the process of obtaining a dataset to be processed from the original dataset and converting the dataset to a preprocessed dataset using a precision scaling factor includes: Extract valid data from the original dataset and perform field mapping to obtain the dataset to be processed. Field mapping is preferably implemented through a mapping table to ensure data format compatibility. The resource fields in the dataset to be processed are standardized to obtain a standard dataset. The standardization process includes, but is not limited to, unifying resource units, rounding decimal places, and unifying signs. By using a precision scaling factor, scaling operations are performed on the resource fields in the standard dataset to obtain a preprocessed dataset consisting of multiple integer resource values.

[0029] Specifically, in this embodiment, by effectively extracting records, mapping fields, and standardizing resources from the original dataset, and converting resource fields into a preprocessed dataset composed of integer values ​​based on a precision scaling factor, the structured and integerized data input is achieved. Thus, the computational complexity of the data can be simplified without losing resource precision, the problem of floating-point error accumulation can be avoided, and the execution efficiency of subsequent combination verification and solution processes can be improved, thereby providing a lightweight and high-precision data foundation for high-concurrency resource conversion calculations.

[0030] In one embodiment, the data application rules corresponding to the original dataset are obtained and parsed to obtain the second constraint, including: Based on the current data application scenario, query historical scenario records and obtain the historical constraint set corresponding to the current data application scenario. The historical constraint set includes multiple historical constraint entries generated and verified under the same or similar scenarios. Validity checks and logical conflict checks are performed sequentially on multiple historical constraint entries to remove invalid and logically conflicting constraints. The remaining historical constraint entries are then merged and their boundaries are tightened to obtain a set of historical constraints. Logical conflict checks are preferentially performed using a Boolean expression detection algorithm. In response to receiving resource application rules input by the user, the resource application rules are standardized in format and mapped in fields to generate a structured set of user constraints. The resource application rules may include, but are not limited to, the following types: the maximum resource allocation ratio of a single data object, resources from different resource domains or system modules must not be mixed, and the resource allocation direction of the same associated object must be consistent. The historical constraint set and the structured user constraint set are merged to obtain the fused constraint set. The constraint entries in the fused constraint set are then prioritized and conflict-resolved to obtain the usable fused constraint set. The second constraint is obtained based on the constraint entries in the available fusion constraint set.

[0031] Specifically, in this embodiment, by parsing the resource application rules corresponding to the original dataset and fusing them with historical constraints and user-defined rules, a second constraint matching the current data processing scenario can be dynamically generated. Simultaneously, the system performs validity checks, logical conflict checks, and priority sorting to ensure that the final fused constraint set retains the empirical validity of historical rules while also considering the business customization of user configurations. Therefore, the system can adapt to different types of resource application scenarios, system modules, and processing strategy requirements during resource conversion, effectively solving the problems of difficulty in fusing multiple constraints and insufficient rule adaptability in existing technologies.

[0032] In a further embodiment, the historical constraint set and the structured user constraint set are merged to obtain a fused constraint set. The constraint entries in the fused constraint set are then prioritized and conflict-resolved to obtain a usable fused constraint set, including: Each constraint item is prioritized based on its source, generation time, and weight level to avoid rule conflicts and ensure the consistency of constraint logic. Among them, regulatory and liquidation rules have the highest priority, followed by system history rules, and resource application rules can cover similar items within the same level. The entries in the fusion constraint set are arranged from high to low priority, and conflict resolution is performed when a logical conflict is detected. Specifically, conflict resolution methods include: First, the master-slave overriding method, where when there are two constraints on the same field (e.g., "maximum resource conversion ratio ≤ 0.8" and "maximum resource conversion ratio ≤ 0.9"), the system retains the higher priority entry (0.8) and discards the lower priority entry; Second, the boundary compromise method, where when all conflicting constraints are valid values, the system can determine the executable boundary by median compromise or by using a default threshold; Third, the conditional partitioning method, where when constraints in different resource application scenarios have cross-conflicts, the system stores and applies constraint entries according to scenario partitions. In this embodiment, the fusion constraint set is used to uniformly manage the historical constraints pre-stored by the system and the personalized resource application rules input by the user in real time. The available fusion constraint set formed by the constraint entries after priority sorting and conflict resolution is used to guide the subsequent data combination and constraint verification process. This maintains the stability of historical rules while taking into account the flexibility of user customization, thereby improving the system's adaptability and executability.

[0033] In one embodiment, based on the first and second constraints, data from the preprocessed dataset are selected, combined, and constraint-verified to obtain multiple candidate solutions, and a candidate solution pool is constructed, including: Multiple data sets to be combined are extracted from the preprocessed dataset. Based on the calculated data similarity between the multiple data sets to be combined, the multiple data sets to be combined are grouped to generate multiple initial data combinations. The data similarity is preferably calculated by vector cosine similarity. That is, the fields such as resource value, data object type, and resource conversion direction in the data are converted into transaction vector inputs. The vector cosine similarity between at least two data sets is calculated by the vector cosine similarity formula and used as the data similarity. Based on the first and second constraints, multiple combinations of initial data are validated; If all initial data combinations satisfy the first and second constraints, then the initial data combination is determined to be a valid combination. If the initial data combination does not satisfy one or more of the first or second constraints, the initial data combination is determined to be an illegal combination. The initial data combination is then broken down into multiple data to be combined and recombined with the data to be combined in at least one non-initial data combination to obtain a recombined data combination, until the recombined data combination becomes a legal combination. Multiple legal combinations are used as multiple candidate solutions. The corresponding combination score value is calculated for each candidate solution. The candidate solutions are then sorted in descending order based on the combination score value to obtain a candidate solution sequence. The combination score value is obtained by weighting the resource conversion efficiency, resource utilization rate, exposure risk and resource conversion coverage of the current data combination. Set the upper limit of the candidate solution pool based on the total number of data, resource value distribution, and resource dispersion of the dataset to be processed; Based on the upper limit of the candidate solution pool, select multiple candidate solutions from the candidate solution sequence in order of their number from front to back, which are less than or equal to the upper limit of the candidate solution pool, and store them as the candidate solution pool.

[0034] Specifically, in this embodiment, data from the preprocessed dataset is selected and combined based on the first and second constraints, and constraint verification is performed to automatically generate candidate data combinations that meet multiple constraints. At the same time, the comprehensiveness and effectiveness of candidate combinations are ensured through data similarity grouping, legality verification, and reorganization mechanisms. Furthermore, the candidate solutions are evaluated from multiple dimensions and a candidate solution pool is constructed through a combination scoring mechanism, which can achieve a balance between ensuring resource conversion efficiency, risk control, and resource utilization. Thus, the intelligence and controllability of resource conversion solutions are effectively improved, avoiding the problems of candidate solution omission or insufficient legality in traditional methods.

[0035] In a further embodiment, an upper limit for the candidate solution pool is set based on the total number of data points, resource value distribution, and resource dispersion of the dataset to be processed, including: The mean and standard deviation of resource values ​​in the dataset to be processed are calculated, and the resource dispersion is obtained based on the mean and standard deviation. The resource dispersion can be expressed as: ; where D A Represented as resource dispersion, Represented as the standard deviation of resource values, A avg Represented as the average resource value; The upper limit of the candidate solution pool is determined based on the resource dispersion and the total number of data in the dataset to be processed, as follows: Among them, K max α represents the upper limit of the candidate solution pool, α represents the basic size coefficient, which is used to control the overall pool size, and the preferred value range is 50 to 200; β represents the resource dispersion adjustment coefficient, which ranges from 0.1 to 1.0 and is used to reflect the impact of resource distribution on the diversity of candidate combinations.

[0036] Specifically, in this embodiment, an upper limit threshold is determined based on the size characteristics of the dataset to be processed. This threshold limits the number of candidate solutions that can be retained during the combinatorial solution process, thereby controlling computational complexity and avoiding excessive memory resource consumption. In actual implementation, the system can also adjust the upper limit of the candidate solution pool based on available memory capacity and CPU load. When computational resources are sufficient, the upper limit is allowed to increase slightly, while the pool capacity is dynamically reduced when the system is under high load. This ensures both the diversity of candidate solutions and the balance between computational efficiency and system stability. Through this adaptive setting method, the upper limit of the candidate solution pool can be automatically adjusted under different resource application scenarios, avoiding the problems of insufficient combinatorial coverage or excessive resource consumption caused by a fixed upper limit, further improving the scalability and real-time response performance of resource conversion computation.

[0037] In one embodiment, a target candidate solution is selected from the candidate solution pool based on a preset filtering rule, and a corresponding data processing result is generated based on the target candidate solution, including: Based on the current number of available threads and the average processing latency of candidate solutions, determine the concurrency level of the current round, and set the current number of solutions based on the current concurrency level. According to the preset filtering rules, a number of candidate solutions not exceeding the current number of solutions are selected from the front end of the candidate solution pool to form the current target candidate solution set; If there is one or more of the following issues among several candidate solutions in the current target candidate solution set: overlapping data objects, resource conflicts, or mutually exclusive execution order, then the current target candidate solution set is determined to meet the intra-batch consistency. Based on one or more of the preset priority strategy, alternative solution strategy, or local rearrangement strategy, conflict resolution is performed on the current target candidate solution set until the current target candidate solution set meets the intra-batch consistency. Take several candidate solutions from the current set of target candidate solutions as target candidate solutions, and determine the corresponding target data combination based on the target candidate solutions; The resource values ​​in the target data combination are reverse-transformed according to the scaling factor to obtain the target resource transformation value, and the corresponding data processing result is generated.

[0038] Specifically, in this embodiment, the concurrency is dynamically determined based on the current number of available threads in the system and the average processing latency of candidate solutions. This enables adaptive resource scheduling during the candidate solution selection process. Furthermore, by combining batch consistency detection and conflict resolution strategies, the final selected target candidate solutions maintain logical consistency in terms of data objects, resources, and execution order, thereby ensuring that the generated data processing results can be directly executed. As a result, not only is the computational concurrency efficiency of the result generation stage improved, but execution errors caused by data object conflicts and resource overlap are also avoided. Ultimately, this achieves high-efficiency, low-risk, and highly executable data processing result output.

[0039] In one embodiment, the concurrency level of the current round is determined based on the current number of available threads and the average processing latency of candidate solutions, and the number of solutions to be solved is set according to the current round concurrency level, including: ; Among them, Z d T represents the concurrency level of the current round, i.e., the upper limit of the number of solutions at the current round. The number of solutions at the current round should be less than or equal to the concurrency level of the current round. f T represents the number of available threads on the current compute node. a T represents the average processing time for candidate solutions. ref Z represents the system parameter processing delay. max This indicates the maximum number of concurrent users allowed by the system, which can be preset based on the number of CPU cores or computing resource limitations.

[0040] It is worth noting that, in one specific embodiment, the preset screening rules are used to quickly select candidate solutions that meet the predetermined optimization objectives and computational constraints from the candidate solution pool, so as to improve the solution efficiency and result quality. The filtering rules can be configured according to different resource application scenarios, including but not limited to: First, efficiency-first rules, which are suitable for scenarios with high requirements for processing speed or system response time, such as cross-system resource synchronization or high-frequency task scheduling scenarios. They prioritize candidate solutions with the highest resource conversion efficiency weight in the combined score to quickly generate data processing results covering the main resource objects. Second, risk control rules, which are suitable for scenarios with high requirements for resource conflict, task dependency, or load risk control, such as large-scale resource allocation or multi-node parallel processing scenarios. They prioritize candidate solutions with lower risk exposure and higher stability to ensure the safety and controllability of the resource conversion process. Third, coverage-first rules, which are suitable for scenarios with a large number of data objects, wide resource distribution, or batch tasks. They prioritize candidate solutions with higher resource coverage to minimize unallocated or unprocessed resource objects. Fourth, weighted comprehensive rules, which set multiple weights such as resource conversion efficiency, risk exposure, and resource coverage to weight and sort the candidate solution scores, and select the candidate solution with the highest comprehensive score. They are suitable for general resource optimization and comprehensive allocation scenarios.

[0041] In a specific implementation, the system can perform conflict resolution processing on the current target candidate solution based on one or more of the preset priority strategy, alternative solution strategy, or local reordering strategy. Specifically, this includes, but is not limited to: Priority strategy: Sort the candidate solutions with conflicts according to the combination score of the candidate solutions or the preset business priority, retain the one with the highest priority, and remove the rest; Alternative solution strategy: Find alternative solutions similar to the removed candidate solutions in the candidate solution pool (e.g., combinations containing the same data object group, similar resource types, and no conflicts) to fill gaps and maintain resource coverage integrity; Local reordering strategy: Make local adjustments to the task or resource allocation order within the target candidate solution to avoid resource locking, task blocking, or data conflict problems caused by the order of execution, such as prioritizing the execution of tasks with less resource consumption and shorter time windows.

[0042] In this embodiment, through the above-described screening and conflict resolution process, the system can achieve a dynamic balance between efficiency, risk, and coverage while ensuring the legality and stability of the computation, thereby making full use of system resources to complete efficient parallel data computation and optimization processing.

[0043] In this embodiment, through the above screening and conflict resolution process, the final target candidate solution can not only make full use of system resources to complete parallel computing, but also achieve a dynamic balance between efficiency, risk and coverage.

[0044] It should be understood that, although Figure 2 The steps in the flowchart are shown sequentially as indicated by the arrows, but these steps are not necessarily executed in the order indicated by the arrows. Unless otherwise specified herein, there is no strict order in which these steps are executed, and they can be performed in other orders. Figure 2 At least some of the steps in the process may include multiple sub-steps or multiple stages. These sub-steps or stages are not necessarily completed at the same time, but can be executed at different times. The execution order of these sub-steps or stages is not necessarily sequential, but can be executed in turn or alternately with other steps or at least some of the sub-steps or stages of other steps.

[0045] In one embodiment, such as Figure 3 As shown, a data processing device is provided, including: a data acquisition module, a scaling determination module, a data processing module, a data constraint module, a candidate solution pool construction module, and a result generation module, wherein: The data acquisition module is used to acquire the raw dataset and obtain the first constraint related to resource transformation processing; The scaling determination module is used to determine the precision scaling factor based on the data precision of the original dataset and the current data application scenario. The data processing module is used to obtain the dataset to be processed from the original dataset and convert the dataset to be processed into a preprocessed dataset by using a precision scaling factor. The data constraint module is used to obtain and parse the data application rules corresponding to the original dataset to obtain the second constraint; The candidate solution pool construction module is used to perform combined search and constraint verification on the preprocessed dataset based on the first and second constraints to obtain multiple candidate solutions and construct a candidate solution pool. The result generation module is used to select target candidate solutions from the candidate solution pool based on preset filtering rules, and generate corresponding data processing results based on the target candidate solutions.

[0046] Specific limitations regarding the data processing device can be found in the limitations regarding the data processing method described above, and will not be repeated here. Each module in the aforementioned data processing device can be implemented entirely or partially through software, hardware, or a combination thereof. These modules can be embedded in or independent of the processor in the computer device in hardware form, or stored in the memory of the computer device in software form, so that the processor can call and execute the operations corresponding to each module.

[0047] In one embodiment, a computer device is provided, which may be a server, and its internal structure diagram may be as follows: Figure 4As shown, the computer device includes a processor, memory, network interface, and database connected via a system bus. The processor provides computing and control capabilities. The memory includes non-volatile storage media and internal memory. The non-volatile storage media stores the operating system, computer programs, and database. The internal memory provides an environment for the operation of the operating system and computer programs stored in the non-volatile storage media. The database stores data for data processing. The network interface communicates with external terminals via a network connection. When the computer program is executed by the processor, it implements a data processing method.

[0048] Those skilled in the art will understand that Figure 4 The structure shown is merely a block diagram of a portion of the structure related to the present application and does not constitute a limitation on the computer device to which the present application is applied. Specific computer devices may include more or fewer components than those shown in the figure, or combine certain components, or have different component arrangements.

[0049] In one embodiment, a computer device is provided, including a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor executes the computer program to perform the following steps: Obtain the original dataset and the first constraint related to resource transformation processing; Determine the precision scaling factor based on the data precision of the original dataset and the current data application scenario; The dataset to be processed is obtained from the original dataset, and then converted into a preprocessed dataset by using a precision scaling factor. Obtain and parse the data application rules corresponding to the original dataset to obtain the second constraint; Based on the first and second constraints, data from the preprocessed dataset are selected for combination and constraint verification to obtain multiple candidate solutions and construct a candidate solution pool. Based on preset filtering rules, target candidate solutions are selected from the candidate solution pool, and corresponding data processing results are generated based on the target candidate solutions.

[0050] In one embodiment, a computer-readable storage medium is provided having a computer program stored thereon, the computer program performing the following steps when executed by a processor: Obtain the original dataset and the first constraint related to resource transformation processing; Determine the precision scaling factor based on the data precision of the original dataset and the current data application scenario; The dataset to be processed is obtained from the original dataset, and then converted into a preprocessed dataset by using a precision scaling factor. Obtain and parse the data application rules corresponding to the original dataset to obtain the second constraint; Based on the first and second constraints, data from the preprocessed dataset are selected for combination and constraint verification to obtain multiple candidate solutions and construct a candidate solution pool. Based on preset filtering rules, target candidate solutions are selected from the candidate solution pool, and corresponding data processing results are generated based on the target candidate solutions.

[0051] Those skilled in the art will understand that implementing all or part of the processes in the methods of the above embodiments can be accomplished by a computer program using related hardware. The computer program can be stored in a non-volatile computer-readable storage medium. When executed, the computer program can include the processes of the embodiments of the methods described above. Any references to memory, storage, databases, or other media used in the embodiments provided in this application can include non-volatile and / or volatile memory. Non-volatile memory may include read-only memory (ROM), programmable ROM (PROM), electrically programmable ROM (EPROM), electrically erasable programmable ROM (EEPROM), or flash memory. Volatile memory may include random access memory (RAM) or external cache memory. By way of illustration and not limitation, RAM is available in a variety of forms, such as static RAM (SRAM), dynamic RAM (DRAM), synchronous DRAM (SDRAM), dual data rate SDRAM (DDRSDRAM), enhanced SDRAM (ESDRAM), synchronous link DRAM (SLDRAM), RAMbus direct RAM (RDRAM), direct memory bus dynamic RAM (DRDRAM), and memory bus dynamic RAM (RDRAM), etc.

[0052] The technical features of the above embodiments can be combined in any way. For the sake of brevity, not all possible combinations of the technical features in the above embodiments are described. However, as long as there is no contradiction in the combination of these technical features, they should be considered to be within the scope of this specification.

[0053] The above embodiments are merely illustrative of several implementation methods of this application, and their descriptions are relatively specific and detailed. However, they should not be construed as limiting the scope of the invention patent. It should be noted that those skilled in the art can make various modifications and improvements without departing from the concept of this application, and these all fall within the protection scope of this application.

Claims

1. A data processing method, characterized in that, include: Obtain the original dataset and the first constraint related to resource transformation processing; Based on the data precision of the original dataset and the current data application scenario, determine the precision scaling factor; The dataset to be processed is obtained from the original dataset, and the dataset to be processed is converted into a preprocessed dataset using the precision scaling factor. Obtain and parse the data application rules corresponding to the original dataset to obtain the second constraint; Based on the first constraint and the second constraint, data from the preprocessed dataset are selected for combination and constraint verification to obtain multiple candidate solutions and construct a candidate solution pool; Based on preset filtering rules, target candidate solutions are selected from the candidate solution pool, and corresponding data processing results are generated based on the target candidate solutions.

2. The data processing method according to claim 1, characterized in that, The process of obtaining the original dataset and obtaining the first constraint includes: Read the target database to obtain multiple data within the target period to construct the original dataset, and determine multiple data objects and their corresponding input and output resources; Request resource data for multiple data objects from the resource management library; in response to receiving resource data from the resource management library, parse the resource data to obtain the available resources for the multiple data objects. Based on the available resources, the input resources, and the output resources, the predicted actual available resource values ​​of multiple data objects are obtained; Acquire and parse preset data application scenario parameters to obtain the resource retention ratio and the upper limit of a single resource conversion; Based on the predicted actual available resource value and the resource retention ratio, generate resource conversion upper limits for multiple data objects; Based on the resource conversion upper limit and the resource single conversion upper limit, corresponding resource conversion constraint parameters are generated, and the resource conversion constraint parameters are used as the first constraint.

3. The data processing method according to claim 2, characterized in that, The determination of the precision scaling factor based on the data precision of the original dataset and the current data application scenario includes: Iterate through the resource value fields in the original dataset to obtain the maximum absolute value of the resource value; Obtain the maximum absolute value of the resource conversion limit and compare it with the maximum absolute value of the resource value, and select the larger value as the benchmark value; Identify the current data application scenario based on the preset scenario configuration table, determine the corresponding resource conversion accuracy level, and generate an initial accuracy scaling factor. Based on the initial precision scaling factor, the reference value is integerized to obtain the reference integer value; Compare the baseline integer value with the upper limit of integer values ​​that the processor can process; If the reference integer value is greater than or equal to the upper limit of the integer value that the processor can process, the initial precision scaling factor is reduced based on the reduction coefficient to obtain the corrected precision scaling factor, until the reference integer value is less than the upper limit of the integer value that the processor can process. If the reference integer value is less than the upper limit of the integer value that the processor can process, then the current precision scaling factor is used as the precision scaling factor.

4. The data processing method according to claim 1, characterized in that, The step of obtaining the dataset to be processed based on the original dataset and converting the dataset to be processed into a preprocessed dataset using the precision scaling factor includes: Extract the valid data from the original dataset and perform field mapping processing to obtain the dataset to be processed; The resource fields in the dataset to be processed are standardized to obtain a standard dataset; The resource fields in the standard dataset are scaled using the precision scaling factor to obtain the preprocessed dataset consisting of multiple integer resource values.

5. The data processing method according to claim 1, characterized in that, The process of obtaining and parsing the data application rules corresponding to the original dataset to obtain the second constraint includes: Based on the current data application scenario, query historical scenario records and obtain the historical constraint set corresponding to the current data application scenario. The historical constraint set includes multiple historical constraint entries generated and verified under the same or similar scenarios. Validity checks and logical conflict checks are performed sequentially on multiple historical constraint entries to remove invalid and logically conflicting constraints. The remaining historical constraint entries are then merged and their boundaries are tightened to obtain a set of historical constraints. In response to receiving resource application rules input by the user, the resource application rules are formatted and mapped to obtain a structured user constraint set; The historical constraint set and the structured user constraint set are merged to obtain a fused constraint set. The constraint entries in the fused constraint set are then prioritized and conflict-resolved to obtain a usable fused constraint set. The second constraint is obtained based on the constraint entries in the available fusion constraint set.

6. The data processing method according to claim 1, characterized in that, Based on the first constraint and the second constraint, data from the preprocessed dataset is selected, combined, and constraint verified to obtain multiple candidate solutions, and a candidate solution pool is constructed, including: Multiple data sets to be combined are extracted from the preprocessed dataset, and based on the calculated data similarity between the multiple data sets to be combined, the multiple data sets to be combined are grouped to generate multiple initial data combinations; Based on the first constraint and the second constraint, multiple combinations of the initial data are verified. If all the initial data combinations satisfy the first constraint and the second constraint, then the initial data combinations are determined to be valid combinations. If the initial data combination does not satisfy one or more of the first constraint or the second constraint, the initial data combination is determined to be an illegal combination. The initial data combination is then broken down into multiple data to be combined and recombined with at least one data to be combined that is not in the initial data combination to obtain a recombined data combination, until the recombined data combination is a legal combination. Multiple legal combinations are used as multiple candidate solutions. A corresponding combination score value is calculated for each of the multiple candidate solutions. The multiple candidate solutions are sorted in descending order based on the combination score values ​​to obtain a candidate solution sequence. The combination score value is obtained by weighting the resource conversion efficiency, resource occupancy rate, exposure risk and resource conversion coverage of the current data combination. Based on the total number of data, resource value distribution, and resource dispersion of the dataset to be processed, set the upper limit of the candidate solution pool; Based on the upper limit of the candidate solution pool, multiple candidate solutions with a number less than or equal to the upper limit of the candidate solution pool are selected sequentially from the candidate solution sequence and stored as the candidate solution pool.

7. The data processing method according to claim 1, characterized in that, The step of selecting a target candidate solution from the candidate solution pool based on a preset filtering rule, and generating a corresponding data processing result based on the target candidate solution, includes: Based on the current number of available threads and the average processing latency of candidate solutions, determine the concurrency level of the current round, and set the current number of solutions based on the current round concurrency level; According to the preset filtering rules, a number of candidate solutions not exceeding the current number of solutions are selected sequentially from the front end of the candidate solution pool to form the current target candidate solution set; If there is one or more of the following issues among the candidate solutions in the current target candidate solution set: data object overlap, resource conflict, or execution order mutual exclusion, then the current target candidate solution set is determined to satisfy intra-batch consistency. Based on one or more of the preset priority strategy, alternative solution strategy, or local rearrangement strategy, conflict resolution is performed on the current target candidate solution set until the current target candidate solution set satisfies intra-batch consistency. Take a number of candidate solutions from the current target candidate solution set as the target candidate solution, and determine the corresponding target data combination based on the target candidate solution; The resource values ​​in the target data combination are reverse-converted according to the scaling factor to obtain the target resource conversion value, and the corresponding data processing result is generated.

8. A data processing apparatus, characterized in that, The device includes: The data acquisition module is used to acquire the raw dataset and obtain the first constraint related to resource transformation processing; The scaling determination module is used to determine the precision scaling factor based on the data precision of the original dataset and the current data application scenario; The data processing module is used to obtain the dataset to be processed based on the original dataset, and to convert the dataset to be processed into a preprocessed dataset using the precision scaling factor. The data constraint module is used to acquire and parse the data application rules corresponding to the original dataset to obtain the second constraint; The candidate solution pool construction module is used to perform combined search and constraint verification on the preprocessed dataset based on the first constraint and the second constraint to obtain multiple candidate solutions and construct a candidate solution pool. The result generation module is used to select target candidate solutions from the candidate solution pool based on preset filtering rules, and generate corresponding data processing results based on the target candidate solutions.

9. A computer device, comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that, When the processor executes the computer program, it implements the steps of the method according to any one of claims 1 to 7.

10. A computer-readable storage medium having a computer program stored thereon, characterized in that, When the computer program is executed by a processor, it implements the steps of the method according to any one of claims 1 to 7.

Images