Data processing method and apparatus, computing device cluster, medium, and program product

Abstract

Embodiments of the present application relate to the technical field of data processing. Disclosed are a data processing method and apparatus, a computing device cluster, a medium, and a program product, which improve the efficiency of mathematical model construction. The method comprises: obtaining a mathematical model expression and an instance data file, wherein each sub-expression is a target function or a constraint condition, each sub-expression comprises a variable and a coefficient, and instance data is used for indicating the value of the coefficient in each sub-expression of the mathematical model expression; converting the mathematical model expression and the instance data into an original data block, i.e., a matrix used for indicating the value of the coefficient in each sub-expression of the mathematical model expression; segmenting the original data block into a plurality of sub-data blocks, wherein data features of the plurality of sub-data blocks meet a similarity threshold, the data features are used for indicating the sparsity or uniformity of the data blocks; and allocating the plurality of sub-data blocks to a plurality of threads for parallel processing to generate solving files. The processing efficiency of a solver is improved.

Description

Data processing method, device, computing device cluster, medium and program product Technical Field

[0001] The present application relates to the field of data processing technology, and in particular to data processing methods, devices, computing device clusters, media, and program products. Background Art

[0002] Mathematical programming is a technology that models complex practical problems into mathematical problems. It can be widely used in the business of solving practical problems. Different mathematical algorithms can be used to establish mathematical models of corresponding mathematical problems. By converting the mathematical model expressions used to solve mathematical programming problems into solution files that can be processed by computing devices, the optimal solution of the mathematical programming model can be found through the mathematical programming solver.

[0003] Currently, users can input a mathematical model expression based on a practical problem into a computing device. The computing device then generates the objective function expression and the expressions of each constraint condition in the mathematical model expression, converting each into a partial solution file and generating a complete solution file to complete the construction of the mathematical programming model. If the constraints in the mathematical model expression are complex and the data volume is large, the process of sequentially generating each partial solution file can be time-consuming, resulting in poor efficiency in building the mathematical programming model.

[0004] Summary of the Invention

[0005] The embodiments of the present application provide a data processing method, apparatus, computing device cluster, medium, and program product, which improve the efficiency of mathematical programming model construction.

[0006] In a first aspect, the present application provides a data processing method, which is applied to a mathematical programming solver, and the mathematical programming solver is deployed on a computing device, and the computing device has multiple threads. The method includes: obtaining a mathematical model expression and an instance data file, the mathematical model expression includes at least one sub-expression, the sub-expression is an objective function or a constraint condition, the sub-expression includes variables and coefficients, and the instance data file includes instance data, and the instance data is used to indicate the values ​​of the coefficients in each sub-expression of the mathematical model expression; converting the mathematical model expression and the instance data into an original data block, the original data block is a matrix used to indicate the values ​​of the coefficients in each sub-expression of the mathematical model expression; dividing the original data block into multiple sub-data blocks, and the data features of the multiple sub-data blocks meet a similarity threshold, the data features are used to indicate the sparsity or uniformity of the data blocks, and the similarity threshold is used to indicate the degree of similarity between the data features of the multiple sub-data blocks; allocating the multiple sub-data blocks to multiple threads for parallel processing to generate a solution file, and the solution file is used to perform solution operations on the sub-expressions corresponding to the multiple sub-data blocks to obtain the solution results of the mathematical model expression.

[0007] It is understandable that the computing device can convert the mathematical model expression and instance data into an original data block by obtaining the mathematical model expression and the instance data file, and can divide the original data block into multiple sub-data blocks according to the similarity threshold, wherein the data features of the multiple sub-data blocks can be guaranteed to meet the similarity threshold, and the multiple sub-data blocks obtained by the division can be assigned to multiple threads for parallel processing, thereby generating a solution file and completing the construction process of the mathematical model. The solution file can be used to solve the sub-expressions corresponding to the multiple sub-data blocks, thereby obtaining the solution result of the mathematical model expression. Since the similarity between the sparsity or uniformity values ​​of each sub-data block after the division meets the similarity threshold, when each sub-data block is assigned to multiple threads for parallel processing, on the basis of ensuring the parallel generation of the solution file by multiple threads, it is further ensured that the workload assigned to each thread for processing is relatively uniform, thereby improving the efficiency of generating the solution file of the mathematical programming model, and thereby improving the efficiency of building the mathematical programming model.

[0008] In one possible implementation, converting a mathematical model expression and instance data into an original data block includes: converting the mathematical model expression into the form of a mathematical model expression tree; determining that the mathematical model expression tree includes at least one first structure to determine that the mathematical model expression has a nested structure, the first structure being used to indicate the structure of the mathematical model expression corresponding to the nested structure; performing expression expansion processing on the mathematical model expression to obtain an expanded model expression; and substituting the instance data into the expanded mathematical model expression to obtain an original data block of the mathematical model expression.

[0009] It can be understood that the computing device can convert the mathematical model expression into the form of a mathematical model expression tree, and determine whether there is a nested structure in the mathematical model expression by judging whether the mathematical model expression tree contains the first structure. When it is determined that the mathematical model expression includes a nested structure, the expression is expanded to obtain an expanded model expression. The original data block can be obtained by substituting the instance data into the expanded mathematical model expression, so that the computing device can accurately judge whether there is a nested structure in the mathematical model expression, thereby solving the problem of expanding the expression when there is a nested structure, which facilitates the subsequent splitting of the original data block.

[0010] In one possible implementation, expression expansion processing is performed on a mathematical model expression, including: transforming a first structure in a mathematical model expression tree according to a specified rule to obtain a transformed model expression tree, where the transformed model expression tree is a model expression tree corresponding to the expanded mathematical model expression; wherein the specified rule is to replace the first structure with a second structure, and the second structure is used to indicate the structure of the mathematical model expression corresponding to the expanded structure of the nested structure.

[0011] It can be understood that by specifying rules, the first structure can be replaced with the second structure, thereby structurally transforming the model expression tree. The model expression corresponding to the transformed model expression tree is the expanded model expression. Through the above-mentioned transformation calculation device, the nested structure in the model expression can be accurately and quickly expanded, thereby ensuring that the original data block can be successfully segmented subsequently.

[0012] In one possible implementation, the data feature includes sparsity, and the method further includes: determining a similarity threshold, the similarity threshold being used to indicate the degree of similarity between the sparsities of multiple sub-data blocks; dividing the original data block into multiple sub-data blocks, including: according to the similarity threshold, dividing the original data block into multiple sub-data blocks, and the degree of similarity between the sparsities of the multiple sub-data blocks meets the similarity threshold.

[0013] It can be understood that when the data features include sparsity, the computing device can determine a similarity threshold for indicating the degree of similarity between the sparsity of multiple sub-data blocks, and then divide the original data block into multiple sub-data blocks according to the similarity threshold, so that the sparsity of the multiple sub-data blocks after division can be relatively similar, thereby ensuring that the amount of computation of each thread is relatively balanced when subsequently assigned to multiple threads for processing, thereby ensuring the balance of the computing load of the computing device.

[0014] In a possible implementation, determining the similarity threshold includes: inputting data features into an optimal threshold selection model to obtain a similarity threshold output by the optimal threshold selection model; or receiving the similarity threshold input by a user.

[0015] It can be understood that, in one case, the computing device can input data features into the optimal threshold selection model to obtain the similarity threshold output by the optimal threshold selection model, or in another case, the computing device can determine the similarity threshold received from the user input as the similarity threshold, thereby obtaining a suitable similarity threshold to meet the requirement that the computational complexity of the sub-data blocks processed by multiple threads is relatively balanced.

[0016] In one possible implementation, the optimal threshold selection model is a neural network model trained using data features of sample data blocks and similarity threshold labels. The data features of the sample data blocks include the sparsity of the sample data blocks, and the similarity threshold labels are similarity thresholds manually labeled for the sample data blocks.

[0017] It is understandable that the optimal threshold selection model can be a neural network model that has been pre-trained, thereby ensuring that the obtained similarity threshold can be adapted according to the data characteristics to ensure that the computational load of the sub-data blocks processed by subsequent multiple threads is relatively balanced.

[0018] In a possible implementation, dividing the original data block into a plurality of sub-data blocks includes dividing the original data block into a plurality of sub-data blocks according to the number of threads.

[0019] It is understandable that since the divided sub-data blocks need to be assigned to multiple threads, the original data block needs to be divided into multiple sub-data blocks according to the number of threads in the computing device to ensure that the computing power of each thread is relatively balanced when the multiple sub-data blocks are subsequently assigned to each thread for processing, thereby improving the efficiency of model construction.

[0020] In a possible implementation, the original data block is divided into multiple sub-data blocks, including: converting the original data block into a sparse matrix format COO to obtain a format-converted original data block; and dividing the format-converted original data block into multiple sub-data blocks.

[0021] It is understandable that when splitting the original data block, the original data block can be first converted into an original data block in COO format. Since the values ​​of the first column in the original data block in COO format change according to a regular pattern, it is convenient to split according to the change regularity of the first column value.

[0022] In one possible implementation, converting an original data block into a sparse matrix format COO to obtain a format-converted original data block includes: obtaining the number of rows and columns of the original data block; if the ratio between the number of columns and the number of rows is greater than a specified threshold, swapping the coordinates of the matrix elements in the original data block to obtain a coordinate-swapped original data block; and converting the coordinate-swapped original data block into COO to obtain a format-converted original data block.

[0023] It is understandable that by performing coordinate processing on the original data blocks in the COO format, the subsequent segmentation of the original data blocks can be facilitated, thereby improving the efficiency of model construction.

[0024] In one possible implementation, multiple sub-data blocks are assigned to multiple threads for parallel processing to generate a solution file, including: assigning multiple sub-data blocks to multiple threads for parallel processing to generate sub-constraint blocks, the sub-constraint blocks are used to indicate the mapping relationship between the sub-data blocks and the sub-expressions that are constraint conditions in the mathematical model expression and the mapping relationship between the sub-data blocks and / or the sub-expressions that are objective functions in the digital model expression; splicing the sub-constraint blocks to generate a solution file.

[0025] It can be understood that since the sub-data blocks after segmentation contain the coefficients in the sub-expressions, each thread needs to generate a sub-constraint block indicating the mapping relationship between the sub-data blocks and the sub-expressions and / or the mapping relationship between the sub-expressions of the objective function in the digital model expression, so that the solution file can splice the sub-constraint blocks to generate a complete solution file, thereby completing the construction process of the mathematical programming model.

[0026] In a possible implementation, the data feature includes one of sparsity, non-zero element ratio, number of non-zero elements, number of zero elements, variance, and word-embedding mapping value.

[0027] In one possible implementation, a mathematical model expression is used to solve one of a factory scheduling problem, a supply and demand simulation problem, and an energy scheduling problem; the instance data is instance data of one of the factory scheduling problem, the supply and demand simulation problem, and the energy scheduling problem; and the solution result is used to guide the solution of one of the factory scheduling problem, the supply and demand simulation problem, and the energy scheduling problem.

[0028] In a second aspect, an embodiment of the present application provides a data processing device, which is used to execute any one of the data processing methods provided in the first aspect above.

[0029] In one possible implementation, the embodiment of the present application can divide the data processing device into functional modules according to the method provided in the first aspect above. For example, each functional module can be divided according to each function, or two or more functions can be integrated into one processing module. Exemplarily, the embodiment of the present application can divide the data processing device into an acquisition module, a processing module, a segmentation module, and a generation module, etc. according to the function. The description of the possible technical solutions and beneficial effects executed by each of the functional modules divided above can refer to the technical solutions provided by the first aspect above or its corresponding possible implementation method, and will not be repeated here.

[0030] In a third aspect, an embodiment of the present application provides a computing device, which includes a processor and a memory, wherein the processor is coupled to the memory; the memory is used to store computer instructions, which are loaded and executed by the processor to enable the computing device to implement the data processing method described in the above aspects.

[0031] In a fourth aspect, an embodiment of the present application provides a computing device cluster, which includes at least one computing device, each computing device including: a processor and a memory, the processor of at least one computing device being used to execute instructions stored in the memory of at least one computing device, so that the computing device cluster executes the data processing method provided in various optional implementations of the above-mentioned first aspect.

[0032] In a fifth aspect, an embodiment of the present application provides a computer-readable storage medium, which stores at least one computer program instruction, and the computer program instruction is loaded and executed by a processor to implement the method for generating actions of a virtual object as described in the above aspects.

[0033] In a sixth aspect, embodiments of the present application provide a computer program product comprising computer instructions stored in a computer-readable storage medium. A processor of a computing device reads the computer instructions from the computer-readable storage medium and executes the computer instructions, causing the computing device cluster to perform the data processing method provided in various optional implementations of the first aspect.

[0034] For the specific descriptions of the second to sixth aspects and their various implementations in this application, reference can be made to the detailed descriptions in the first aspect and its various implementations; and for the beneficial effects of the second to sixth aspects and their various implementations, reference can be made to the analysis of the beneficial effects in the first aspect and its various implementations, which will not be repeated here.

[0035] These and other aspects of the present application will become more readily apparent from the following description. BRIEF DESCRIPTION OF THE DRAWINGS

[0036] FIG1 is a schematic diagram of an application scenario according to an exemplary embodiment;

[0037] FIG2 is a schematic diagram showing a hardware structure of a computing device according to an exemplary embodiment;

[0038] FIG3 is a flow chart showing a data processing method according to an exemplary embodiment;

[0039] FIG4 is a schematic diagram of the structure of an expression tree involved in the embodiment shown in FIG3 ;

[0040] FIG5 is a schematic diagram of a process of a balanced model expression tree involved in the embodiment shown in FIG3 ;

[0041] FIG6 is a schematic diagram of a data processing process involved in the embodiment shown in FIG3 ;

[0042] FIG7 is a schematic structural diagram of a data processing device according to an exemplary embodiment;

[0043] FIG8 is a schematic diagram of a computing device according to an exemplary embodiment;

[0044] FIG9 is a schematic diagram showing a computing device cluster according to an exemplary embodiment;

[0045] FIG10 is a schematic diagram showing a connection mode between computing device clusters according to an exemplary embodiment. DETAILED DESCRIPTION

[0046] In order to make the objectives, technical solutions and advantages of this application clearer, the implementation methods of this application will be further described in detail below with reference to the accompanying drawings.

[0047] In this document, "plurality" refers to two or more. "And / or" describes a relationship between associated objects, indicating that three possible relationships exist. For example, "A and / or B" can mean: A exists alone, A and B exist simultaneously, or B exists alone. The character " / " generally indicates an "or" relationship between the associated objects.

[0048] Furthermore, in the description of this application, unless otherwise specified, "plurality" means two or more than two. "At least one of the following" or similar expressions refers to any combination of these items, including any combination of single or plural items. For example, at least one of a, b, or c can mean: a, b, c, ab, ac, bc, or abc, where a, b, and c can be single or plural.

[0049] In addition, in order to facilitate the clear description of the technical solutions of the embodiments of the present application, in the embodiments of the present application, words such as "first" and "second" are used to distinguish between identical or similar items with substantially the same functions and effects. Those skilled in the art will understand that words such as "first" and "second" do not limit the quantity and execution order, and words such as "first" and "second" do not necessarily limit differences. At the same time, in the embodiments of the present application, words such as "exemplary" or "for example" are used to indicate examples, illustrations or explanations. Any embodiment or design described as "exemplary" or "for example" in the embodiments of the present application should not be interpreted as being more preferred or more advantageous than other embodiments or design schemes. Specifically, the use of words such as "exemplary" or "for example" is intended to present related concepts in a concrete way for easy understanding.

[0050] First, the application scenarios of the embodiments of the present application are exemplarily introduced.

[0051] Mathematical programming can be used to model complex real-world problems into mathematical problems, enabling them to be solved using computing devices. Different mathematical algorithms can be used to model different real-world problems, resulting in their own mathematical models.

[0052] Among them, practical problems solved through mathematical modeling and solving through mathematical programming can be widely used in the industrial sector to solve practical problems including warehouse transfers, port scheduling, and factory emissions.

[0053] Specifically, the process of mathematical programming modeling and solving practical problems can include at least multiple steps of collecting data from the business side, compiling mathematical models, solving the mathematical models, and returning the solutions to the business side. This process can be implemented by an algebraic modeling system (AMS).

[0054] For example, FIG1 is an architectural diagram of an application scenario provided by an exemplary embodiment of the present application. As shown in FIG1 , the architecture may include two parts: a business side and an AMS. The business side proposes business demands and provides a model expression based on the business demands. The model expression is input into the AMS in the form of a mathematical model expression. The AMS may run on a computing device. In addition, the business side may also preprocess the business data related to the business demands (for example, data review, data filtering, etc.), and then input the preprocessed model data into the AMS. The AMS uses an algebraic modeling language (AML) to construct a model based on the obtained preprocessed model data and model expression, that is, converting the mathematical expression into a statement understandable to the computing device to generate a solution file for the mathematical programming model, and then calling the mathematical programming solver to run the solution file to solve the constructed mathematical programming model, that is, performing numerical calculations on the input mathematical programming model based on the mathematical programming solver to obtain the optimal solution. The mathematical programming solver outputs the final numerical result, which is the solution. Then, through data post-processing, the solution is restored to the actual decision and returned to the business side. Among them, data post-processing can be to restore the solution obtained by the mathematical programming solver into an actual decision plan that the business side can understand, and analyze the feasibility, robustness and other characteristics of the solution, so that the business side can execute the corresponding decision according to the solution instructions to solve business problems and realize business needs.

[0055] Currently, due to the rapid increase in the complexity and scale of business scenarios of actual problems, mathematical programming problems have become larger and more complex, which requires AMS to be optimized to adapt to the mathematical modeling process of larger-scale mathematical programming problems.

[0056] In related technologies, AMS builds models based on mathematical model expressions and related instance data. This means that the solution file generation process is handled by a single CPU thread, meaning that AMS sequentially builds the complete mathematical programming model using that single CPU thread. This reliance on a single thread for model building results in low mathematical model construction efficiency, making it incapable of solving increasingly large and complex mathematical programming problems.

[0057] The software used for model construction may include a mathematical programming language (AMPL) or modeling software (OPTVOM).

[0058] In view of this, in order to realize the process of constructing a mathematical programming model through multi-threaded parallelism and generating a corresponding solution file, the embodiment of the present application can obtain a mathematical model expression and instance data, the mathematical model expression includes a sub-expression of the objective function and a sub-expression of the constraint condition, each sub-expression can include variables and the coefficients of the variables, and convert the mathematical model expression and the instance data into an original data block indicating the value of the coefficient of the variable in each sub-expression, and then determine the similarity threshold of the degree of similarity between the data features of multiple sub-data blocks, divide the original data block into multiple sub-data blocks according to the similarity threshold, assign the multiple sub-data blocks to multiple threads, and have multiple threads process in parallel to generate a solution file, so that the subsequent computing device can parse the solution file through the solver to obtain the solution result of the mathematical model expression. This allows the computing device to synchronously generate the solution file parts corresponding to the sub-expressions in multiple mathematical model expressions through multiple threads in parallel, thereby saving the time for generating the solution file. At the same time, since data characteristics need to be considered when splitting the original data blocks, it is ensured that the similarity between the data characteristics of the sub-data blocks after segmentation meets the similarity threshold, thereby ensuring that the amount of calculation of each thread subsequently assigned when processing the sub-data blocks is relatively balanced, thereby improving the efficiency of generating the solution file, that is, improving the efficiency of building the mathematical programming model.

[0059] FIG2 is a schematic diagram of the hardware structure of a computing device 10 provided in an embodiment of the present application. The computing device 10 may be a server, and the server may be an X86 architecture server, specifically a blade server, a high-density server, a rack server, or a high-performance server. The computing device 10 may run an AMS. The server includes a processor 201, a memory 202, and a bus 203.

[0060] The processor 201 may include one or more processing units, for example, the processor 201 may include an application processor (AP), a modem processor, and / or a neural-network processing unit (NPU), etc. Different processing units may be independent devices or integrated into one or more processors. The processor 201 may run a mathematical programming solver, which may be used to operate on the solution file of the mathematical programming model to obtain the solution result of the mathematical programming problem corresponding to the mathematical programming model. The processor 201 may have multiple threads including thread 1, thread 2, etc., and the solution file of the mathematical programming model supported by the mathematical programming solver may be generated in parallel through multiple threads.

[0061] The memory 202 is used to store data, including but not limited to random access memory (RAM), read-only memory (ROM), flash memory, optical storage, database, etc. The memory 202 is used to interact with the processor 201 .

[0062] The bus 203 , the processor 201 and the memory 202 are usually connected to each other via the bus 203 , or are connected to each other in other ways.

[0063] It should be noted that the application scenarios and system architectures described in the embodiments of the present application are intended to more clearly illustrate the technical solutions of the embodiments of the present application, and do not constitute a limitation on the technical solutions provided in the embodiments of the present application. Ordinary technicians in this field can know that with the evolution of the system architecture and the emergence of new business scenarios, the technical solutions provided in the embodiments of the present application are also applicable to similar technical problems.

[0064] For ease of understanding, the data processing method provided in this application is exemplarily introduced below in conjunction with the accompanying drawings. The data processing method is applicable to the computing device shown in FIG1 .

[0065] FIG3 shows a flow chart of a data processing method provided by an exemplary embodiment of the present application. The data processing method can be executed by a computing device and includes the following steps:

[0066] S101, a computing device obtains a mathematical model expression.

[0067] In an embodiment of the present application, a computing device may receive a mathematical model expression, which may be a mathematical model expression for solving a linear programming problem.

[0068] In a possible implementation, the computing device obtains the mathematical model expression uploaded by the user through mathematical programming modeling software (eg, Grassland software or OPTVOM software).

[0069] That is, after editing the mathematical model expression, the user can input it into the mathematical programming modeling software, and the computing device receives the mathematical model expression through the mathematical programming modeling software.

[0070] The mathematical model expression may include sub-expressions of the objective function and sub-expressions of the constraints. The mathematical model expression may be used to solve for the values ​​of the variables that produce the optimal solution to the objective function while satisfying the constraints. The mathematical model expression may include variables and coefficients. The variables may be variables to be solved, and the coefficients may be prefix coefficients of the variables in the mathematical model expression or constant coefficients. The coefficients in the mathematical model expression may be obtained by substituting instance data into the mathematical model expression.

[0071] Exemplarily, the mathematical model expression obtained by the computing device may be as follows.

[0072] Among them, the sub-expression of the objective function is The subexpression of the constraint is The variable is x i , the coefficients can include b i and v i The coefficient is b i and v i Each has corresponding instance data. i and v i Substituting the corresponding instance data into the above mathematical model expression, we can solve the problem of satisfying the constraints. In the case of The variable x at the minimum i The value of .

[0073] S102: The computing device obtains an instance data file.

[0074] In the embodiment of the present application, the computing device may obtain the instance data file, and the triggering time for the computing device to obtain the instance data file and the computing device to obtain the mathematical model expression is not limited here.

[0075] The instance data file includes instance data, and the instance data can be used to indicate the values ​​of coefficients in each sub-expression of the mathematical model expression.

[0076] In a possible implementation, the computing device may also obtain the instance data file uploaded by the user through mathematical programming modeling software (eg, Grassland software or OPTVOM software).

[0077] That is, after editing the instance data file, the user may input the edited instance data file into the mathematical programming modeling software, and the computing device may receive the instance data file through the mathematical programming modeling software.

[0078] The instance data can be input into the computing device in the form of a two-dimensional data table such as an Excel table or a dataframe table.

[0079] For example, if the coefficients include b i and v i , the instance data received by the computing device may be recorded in the form of Table 1.

[0080] Table 1

[0081] That is to say, the computing device can obtain the b recorded in the form of two-dimensional data tables such as the above-mentioned Excel table or dataframe table. i and v i The corresponding instance data.

[0082] S103: The computing device converts the mathematical model expression and the instance data into original data blocks.

[0083] The original data block may be a matrix for indicating the values ​​of coefficients in each sub-expression of the mathematical model expression.

[0084] In a possible implementation, the computing device may convert the original data block according to the following steps S11 to S13.

[0085] S11, the computing device determines whether the mathematical model expression has a nested structure.

[0086] In an embodiment of the present application, after obtaining the mathematical model expression, the computing device can determine whether the mathematical model expression contains a nested structure.

[0087] The mathematical model expression with a nested structure means that there is an expandable structure in the mathematical model expression.

[0088] In one possible implementation, the computing device transforms the mathematical model expression into the form of a mathematical model expression tree, determines whether the mathematical model expression tree includes a first structure, and if so, determines that the mathematical model expression has a nested structure; if not, determines that the mathematical model expression does not have a nested structure.

[0089] Among them, the first structure can include at least five nodes, namely, a node representing multiplication is connected to a node representing addition, the node representing multiplication is also connected to a node representing a variable or a constant, and the node representing addition is respectively connected to at least two nodes representing variables or constants.

[0090] For example, FIG4 is a schematic diagram of the structure of an expression tree involved in an embodiment of the present application. i +v i (x i +b i ) is transformed into an expression tree as shown in Figure 4, where, due to vi (x i +b i ) can be expanded and transformed into v i x i +v i b i , so the expression is an expression with a single-layer nested structure, but the computer cannot directly identify whether the expression has a nested structure, and the expression needs to be converted into the form of an expression tree. The computing device determines whether the expression tree includes the first structure 21 to determine whether the corresponding mathematical model expression has a nested structure. Since the expression tree includes the first structure 21, it can be determined that the mathematical model expression has a nested structure and the expression needs to be expanded.

[0091] S12: If it is determined that the mathematical model expression has a nested structure, the computing device performs expression expansion processing on the mathematical model expression to obtain an expanded model expression.

[0092] In an embodiment of the present application, the computing device can determine whether the mathematical model expression has a nested structure by executing S11. If it is determined that it has a nested structure, the nested structure can be decoupled to obtain a fully expanded model expression after the mathematical model expression is expanded.

[0093] In one possible implementation, the computing device transforms the first structure in the model expression tree corresponding to the mathematical model expression according to a specified rule to obtain a transformed model expression tree, where the transformed model expression tree is an expression tree of the mathematical model expression after the mathematical model expression is decoupled and expanded.

[0094] The specified rule may be to replace the first structure with a second structure, wherein the second structure may include at least seven nodes, wherein a node representing addition is connected to at least two nodes representing multiplication, and each node representing multiplication may be connected to two nodes representing variables or coefficients.

[0095] That is to say, the process of performing expression processing on the mathematical model expression is the process of balancing the model expression tree.

[0096] For example, FIG5 is a schematic diagram of a process for balancing a model expression tree according to an embodiment of the present application. As shown in FIG5 , when a model expression tree includes a first structure 21, the model expression tree needs to be balanced, that is, the first structure 21 needs to be transformed according to a specified rule. The specified rule is to transform the first structure 21 into a second structure 22. The second structure 22 can represent the model expression after the nested structure in the model expression tree is expanded.

[0097] S13, the computing device substitutes the instance data into the expanded mathematical model expression to obtain the original data block of the mathematical model expression.

[0098] In an embodiment of the present application, after the computing device expands and processes the model expression, it can substitute the instance data according to the corresponding coefficients to obtain a mathematical model expression after substituting the instance data, and the original data block can be generated based on the mathematical model expression after substituting the instance data.

[0099] The original data block may be a matrix for indicating the values ​​of coefficients in each sub-expression of the mathematical model expression.

[0100] Specifically, since the mathematical model expression may include multiple sub-expressions of constraints, after each sub-expression of the constraint is expanded separately, each expanded sub-expression of the constraint may include multiple meta-expressions, each meta-expression is connected by addition operations, and each meta-expression includes variables and coefficients (that is, constant values), and the original data block can be a matrix used to record the constant values ​​of the coefficients of each meta-expression.

[0101] For example, if the mathematical model expression includes the constraint condition sub-expression including:

[0102] By executing steps S11 and S12, the computing device can expand the sub-expressions of each constraint condition to obtain the sub-expression trees of each constraint condition after balancing. The sub-expressions of the constraint condition after expansion are shown below.

[0103] Among them, each meta-expression is connected by addition. After substituting the instance data, the prefix coefficients of the variable x corresponding to the same i value can be combined and calculated. Therefore, each meta-expression in each expanded constraint condition sub-expression is the variable x with each i value with the prefix coefficient. For example, if i is 0, 1, 2, and the instance data includes b i =(2,3,6), v i =(5,7,2), substitute the instance data into the expanded mathematical model expression above, and after merging similar terms, the sub-expression of the constraint condition after substituting the instance data is as follows.

[0104] After the above processing, the coefficients of each element expression of each constraint condition (that is, constant values) can be obtained, and the corresponding original data blocks can be obtained as shown below.

[0105] S104: The computing device divides the original data block into a plurality of sub-data blocks.

[0106] The data features of the multiple sub-data blocks meet the similarity threshold, the data features can be used to indicate the sparsity or uniformity of the data block, and the similarity threshold can be used to indicate the degree of similarity between the data features of the multiple sub-data blocks.

[0107] In a possible implementation, the computing device determines a similarity threshold, and then divides the original data block into a plurality of sub-data blocks according to the similarity threshold.

[0108] In an embodiment of the present application, after determining the similarity threshold, the computing device may evenly divide the original data block into a plurality of sub-data blocks according to the similarity threshold.

[0109] The similarity threshold can be used to indicate the granularity of segmenting the original data block.

[0110] In a possible implementation, the computing device may determine the similarity threshold through a similarity threshold selection process, which may specifically include determining the similarity threshold through at least the following two methods.

[0111] 1) The similarity threshold selection process may include a user directly inputting a desired similarity threshold into a computing device.

[0112] In other words, the user can input the degree to which the converted original data block should be segmented as required. For example, the similarity threshold can be used to input the number of sub-data blocks to be segmented into, or the number of rows each data block should contain.

[0113] For example, if the user needs to divide the original data block into 5 sub-data blocks, the user can input a similarity threshold of 5. When the computing device determines that the similarity threshold is 5, if the original data block is formatted into the above-mentioned 3*9 matrix, the original data block after format conversion can be divided into a sub-data block with every two rows, thereby obtaining 5 sub-data blocks.

[0114] 2) The similarity threshold selection process may include a computing device inputting data features into an optimal threshold selection model to obtain a similarity threshold output by the optimal threshold selection model.

[0115] Among them, the data characteristics of the original data block may include at least one of the uniformity or sparsity of the original data block. The uniformity can be used to indicate the uniformity of the data in the data block, and the sparsity can be used to indicate the proportion of null values ​​in the data block.

[0116] In a possible implementation, the data feature may include one of sparsity, non-zero element ratio, number of non-zero elements, number of zero elements, variance, and word-embedding mapping value.

[0117] Optionally, the computing device may determine the similarity threshold based on data features of the original data block and the number of threads of the computing device.

[0118] Specifically, the computing device may take the data features of the original data block and the number of threads of the computing device as input data and input them into the optimal threshold selection model, and the computing device may obtain the similarity threshold output by the optimal threshold selection model.

[0119] The optimal threshold selection model may be a neural network model, for example, a multilayer perceptron (MLP) model.

[0120] In one possible implementation, a training device inputs data feature samples of the original data block and a sample of the number of threads of the computing device into an optimal threshold selection model, determines a loss function based on the similarity threshold and similarity threshold label output by the optimal threshold selection model, trains the optimal threshold selection model according to the loss function, updates the model parameters, and finally obtains a trained optimal threshold selection model. The data features of the sample data block may include the sparsity of the sample data block, and the similarity threshold label is the similarity threshold manually labeled for the sample data block.

[0121] Exemplarily, the computing device can extract data features of the original data block. If the data features of the original data block include that the dimension of the original data block is 2, the sparsity is 0, and the number of threads of the computing device is 4, the above data features and the number of threads are input into the trained optimal threshold selection model, and the trained optimal threshold selection model outputs the calculated similarity threshold. For example, the similarity threshold can indicate that the original data block after format conversion is divided into a sub-data block with every two rows.

[0122] If the variables corresponding to the same-sized data blocks have different dimensions, the amount of computation required by threads to calculate the sub-expressions corresponding to the same-sized data blocks will be different. The larger the variable dimension, the greater the amount of computation required by the threads to calculate the sub-expressions corresponding to the same-sized data blocks. Therefore, to ensure that the computational complexity of the sub-expressions corresponding to the sub-data blocks assigned to each thread is similar, it is necessary to ensure that each thread calculates the sub-expressions corresponding to the sub-data blocks in parallel as much as possible, thereby improving the efficiency of the subsequent solution of the model expression.

[0123] For example, if the variable is x i , each corresponding sub-data block after segmentation is a 2*3 matrix. The subsequent computing device evenly distributes the computing tasks corresponding to each sub-data block to each thread. Similarly, if the variable is x ij, each corresponding data block after segmentation is a 2*3 matrix. The subsequent computing device evenly distributes the computing tasks corresponding to each data block to each thread. The sizes of the data blocks in the above two cases are the same, but the dimensions of the variables in the corresponding computing tasks are different, resulting in the threads actually processing x i The amount of computation required for the corresponding computation task is less than that required for processing x ij The amount of computation required for the corresponding computational task, so the dimension of the variable can be used as one of the sample labels for training the neural network model.

[0124] Similarly, if the sparsity in data blocks of the same size is different, the amount of computation performed by threads on sub-expressions corresponding to data blocks of the same size will also be different. Sparsity can indicate the number of null values ​​in a data block. In other words, when two sub-data blocks of the same size need to be assigned to threads to calculate their respective corresponding sub-expressions, the thread with the larger sparsity will have a smaller amount of computation performed on the sub-expression corresponding to the sub-data block. Therefore, in order to ensure that the amount of computation of the sub-expressions corresponding to the sub-data blocks assigned to each thread is not much different, it is necessary to ensure that each thread calculates the sub-expressions corresponding to each sub-data block in parallel as much as possible, so as to achieve the purpose of improving the efficiency of subsequent solution of the model expression.

[0125] For example, if each data sub-block after segmentation is a 2*3 matrix, the sparsity corresponding to the first data sub-block is greater than the sparsity corresponding to the second data sub-block, that is, the number of null values ​​in the first data sub-block is greater than the number of null values ​​in the second data sub-block. In the above case, if the first and second data sub-blocks are the same size, but the corresponding sparsity is different, the actual amount of computation performed by the thread when processing the calculation task corresponding to the first data sub-block is less than the amount of computation performed when processing the calculation task corresponding to the second data sub-block. Therefore, the sparsity of the data block can also be used as one of the sample labels for training the neural network model.

[0126] Furthermore, to distribute the sub-data blocks more evenly across the various threads of the computing device, the computing device can subsequently perform computations corresponding to the sub-data blocks in parallel through the threads, thereby improving computational efficiency. Therefore, the number of threads in the computing device can also be used as one of the sample labels for training the neural network model.

[0127] After determining the similarity threshold according to the similarity threshold selection process, the computing device may perform format conversion on the original data block to generate a format-converted original data block.

[0128] In a possible implementation, after obtaining the original data block of the constraint condition, the original data block may be converted into a sparse matrix format (coordinate list, COO) to obtain the original data block after format conversion.

[0129] For example, if the original data block is as follows.

[0130] The original data block after format conversion can be the following matrix.

[0131] Among them, each row of the original data block after format conversion indicates the position of each matrix element in the original data block in the matrix and the value of each matrix element, the value of the first column is used to indicate the row to which the matrix element belongs in the original data block, the value of the second column is used to indicate the column to which the corresponding matrix element belongs in the original data block, and the value of the third column is the corresponding matrix element value in the original data block.

[0132] In one possible implementation, the computing device obtains the number of rows and columns of the original data block, and if the ratio between the number of columns and the number of rows is greater than a specified threshold, the coordinates of the matrix elements in the matrix of the original data block are swapped to obtain the original data block with swapped coordinates, and the original data block with swapped coordinates is converted into the COO format to obtain the original data block with format conversion.

[0133] Since the original data block may have a situation where the number of rows is much smaller than the number of columns, this results in the ratio between the number of rows and the number of columns being greater than a specified threshold. In this case, if the original data block is directly converted into the COO format, the first column of the matrix in the COO format will have a large number of rows with the same value. This will result in each sub-data block being larger if the computing device divides the data block according to the numerical jump position of the first column, thereby affecting the effect of subsequently allocating the sub-data blocks to each thread to perform computing tasks.

[0134] For example, if the original data block is as follows.

[0135] The ratio of the number of columns to the number of rows is 2. If the specified threshold is 1, it can be determined that the ratio of the number of columns to the number of rows is greater than the specified threshold. Therefore, it is necessary to swap the coordinates of each matrix element in the original data block. The original data block after the coordinate swap is shown below.

[0136] The computing device can then convert the original data block after the coordinate swap into the COO format, and the obtained original data block after the format conversion is as shown below.

[0137] Among them, if the similarity threshold is 3, it can be divided according to the digital transition position in the first column, and divided into 6 sub-data blocks of 3*3. If the computing device has 3 threads, each subsequent thread can be evenly divided into two sub-data blocks to perform parallel calculations on the corresponding computing tasks.

[0138] S105: The computing device assigns the multiple sub-data blocks to multiple threads for parallel processing to generate a solution file.

[0139] The solution file can be used to perform solution operations on the sub-expressions corresponding to the multiple sub-data blocks to obtain the solution results of the mathematical model expression.

[0140] In one possible implementation, the computing device may assign multiple sub-data blocks to multiple threads for parallel processing to generate sub-constraint blocks, and then splice the sub-constraint blocks to generate a solution file.

[0141] In an embodiment of the present application, a computing device can generate corresponding sub-constraint blocks based on sub-data blocks in parallel through multiple threads. The sub-constraint blocks can be used to indicate the mapping relationship between the sub-data blocks and the sub-expressions of the constraint conditions in the mathematical model expression, as well as the mapping relationship between the sub-data blocks and / or the sub-expressions of the objective function in the mathematical model expression, and the sub-constraint blocks are spliced ​​together to generate a complete solution file.

[0142] The solution file may be a file that is recognized by a mathematical programming solver of a computing device and is used to solve the optimal solution of variables in a mathematical model expression. The solution file may be an LP file or an MPS file.

[0143] In one possible implementation, the mathematical model expression can be used to solve one of the factory scheduling problems, supply and demand simulation problems, and energy scheduling problems. The instance data can be instance data of one of the factory scheduling problems, supply and demand simulation problems, and energy scheduling problems. The solution result can be used to guide the solution of one of the factory scheduling problems, supply and demand simulation problems, and energy scheduling problems.

[0144] That is to say, this application can be applied in the scenario of solving one of the factory scheduling problems, supply and demand simulation problems, and energy scheduling problems, in the process of constructing mathematical programming models for the above practical problems.

[0145] Exemplarily, FIG6 is a flow chart of a mathematical programming model construction process involved in an embodiment of the present application. As shown in FIG6, on each CPU, a data block can be converted into a constraint block through a two-stage process of balancing the model expression tree and adaptively segmenting the data. The process can include three core devices: a symbol distributor, a data segmenter, and an iterator. The symbol distributor located at the front end of the OPTVOM software can transform the mathematical model expression into a model expression tree, and then balance the model expression tree to obtain a model expression tree with several independent subtrees. The original data block can be generated by combining the instance data with the model expression tree. The data segmenter located at the back end can adaptively segment the original data block and divide the original data block into data block 1, data block 2, and data block 3, etc. The segmented data blocks are assigned to each thread, and each thread can traverse the data block through the iterator and establish a mapping relationship between the variables involved in the data block and the constraints. For example, thread 1, assigned data block 1, can traverse data block 1 and establish mapping relationship 1 between the variables and constraints involved in data block 1; thread 2, assigned data block 2, can traverse data block 2 and establish mapping relationship 2 between the variables and constraints involved in data block 2; thread 3, assigned data block 3, can traverse data block 3 and establish mapping relationship 3 between the variables and constraints involved in data block 3, and so on. The generated constraint blocks 1, 2, and 3 are then concatenated to obtain the constraint matrix, and thus the solution file.

[0146] In summary, the computing device can convert the mathematical model expression and instance data into an original data block by obtaining the mathematical model expression and the instance data file, and can divide the original data block into multiple sub-data blocks according to the similarity threshold, wherein the data features of the multiple sub-data blocks can be guaranteed to meet the similarity threshold, and the multiple sub-data blocks obtained by the division can be assigned to multiple threads for parallel processing, thereby generating a solution file and completing the construction process of the mathematical model. The solution file can be used to solve the sub-expressions corresponding to the multiple sub-data blocks, thereby obtaining the solution result of the mathematical model expression. Since the similarity between the sparsity or uniformity values ​​of each sub-data block after the division meets the similarity threshold, when each sub-data block is assigned to multiple threads for parallel processing, on the basis of ensuring the parallel generation of the solution file by multiple threads, it further ensures that the workload assigned to each thread for processing is relatively uniform, thereby improving the efficiency of generating the solution file of the mathematical programming model, and further improving the efficiency of building the mathematical programming model.

[0147] The above mainly introduces the scheme of the embodiment of the present application from the perspective of method. It is understandable that, in order to realize the above functions, the data processing device includes at least one of the hardware structure and software modules corresponding to the execution of each function. Those skilled in the art should easily realize that, in combination with the units and algorithm steps of each example described in the embodiments disclosed herein, the present application can be implemented in the form of hardware or a combination of hardware and computer software. Whether a function is executed in the form of hardware or computer software driving hardware depends on the specific application and design constraints of the technical solution. Professional and technical personnel can use different methods to implement the described functions for each specific application, but such implementation should not be considered to be beyond the scope of this application.

[0148] The embodiment of the present application can divide the data processing device into functional units according to the above method example. For example, each functional unit can be divided according to each function, or two or more functions can be integrated into one processing unit. The above-mentioned integrated unit can be implemented in the form of hardware or in the form of software functional units. It should be noted that the division of units in the embodiment of the present application is schematic and is only a logical functional division. There may be other division methods in actual implementation.

[0149] For example, FIG7 shows a schematic diagram of the structure of a data processing device 500 provided by an exemplary embodiment of the present application. The data processing device 500 is applied to a computing device, or the data processing device 500 can be a computing device. The data processing device 500 includes:

[0150] An acquisition module 510 is configured to acquire a mathematical model expression and an instance data file, wherein the mathematical model expression includes at least one sub-expression, the sub-expression being an objective function or a constraint condition, the sub-expression including variables and coefficients, and the instance data file including instance data indicating values ​​of the coefficients in each of the sub-expressions of the mathematical model expression;

[0151] A processing module 520 is configured to convert the mathematical model expression and the instance data into an original data block, where the original data block is a matrix indicating the values ​​of the coefficients in each of the sub-expressions of the mathematical model expression;

[0152] a segmentation module 530 configured to segment the original data block into a plurality of sub-data blocks, wherein data features of the plurality of sub-data blocks meet a similarity threshold, wherein the data features are used to indicate the sparsity or uniformity of the data block, and the similarity threshold is used to indicate the degree of similarity between the data features of the plurality of sub-data blocks;

[0153] The generation module 540 is used to assign the multiple sub-data blocks to multiple threads for parallel processing to generate a solution file, which is used to solve the sub-expressions corresponding to the multiple sub-data blocks to obtain the solution result of the mathematical model expression.

[0154] For example, in combination with Figure 3, the acquisition module 510 can be used to execute S101 and S102 as shown in Figure 3, the processing module 520 can be used to execute S103 as shown in Figure 3, the segmentation module 530 can be used to execute S104 as shown in Figure 3, and the generation module 540 can be used to execute S105 as shown in Figure 3.

[0155] In one possible implementation, the processing module 520 is further used to convert the mathematical model expression into the form of a mathematical model expression tree; determine that the mathematical model expression tree includes at least one first structure to determine that the mathematical model expression has a nested structure, and the first structure is used to indicate the structure of the mathematical model expression corresponding to the nested structure; perform expression expansion processing on the mathematical model expression to obtain an expanded model expression; substitute the instance data into the expanded mathematical model expression to obtain the original data block of the mathematical model expression.

[0156] In one possible implementation, the processing module 520 is also used to transform the first structure in the mathematical model expression tree according to a specified rule to obtain a transformed model expression tree, and the transformed model expression tree is the model expression tree corresponding to the expanded mathematical model expression; wherein the specified rule is to replace the first structure with a second structure, and the second structure is used to indicate the structure of the mathematical model expression corresponding to the expanded structure of the nested structure.

[0157] In a possible implementation, the data feature includes sparsity, and the apparatus further includes:

[0158] a determination module, configured to determine the similarity threshold, wherein the similarity threshold is used to indicate a degree of similarity between the sparsities of the plurality of sub-data blocks;

[0159] The processing module 520 is further configured to divide the original data block into the multiple sub-data blocks according to the similarity threshold, wherein the similarity between the sparsities of the multiple sub-data blocks meets the similarity threshold.

[0160] In a possible implementation, the determination module is further configured to input the data feature into an optimal threshold selection model to obtain the similarity threshold output by the optimal threshold selection model; or receive the similarity threshold input by a user.

[0161] In one possible implementation, the optimal threshold selection model is a neural network model trained using data features of sample data blocks and similarity threshold labels, where the data features of the sample data blocks include the sparsity of the sample data blocks, and the similarity threshold labels are similarity thresholds manually labeled for the sample data blocks.

[0162] In a possible implementation, the processing module 520 is further configured to convert the original data block into a sparse matrix format COO to obtain a format-converted original data block; and divide the format-converted original data block into the multiple sub-data blocks.

[0163] In one possible implementation, the generation module 540 is also used to assign the multiple sub-data blocks to the multiple threads for parallel processing to generate sub-constraint blocks, which are used to indicate the mapping relationship between the sub-data blocks and the sub-expressions that are constraint conditions in the mathematical model expression and the mapping relationship between the sub-data blocks and / or the sub-expressions that are objective functions in the digital model expression; and splice the sub-constraint blocks to generate the solution file.

[0164] In a possible implementation, the data feature includes one of sparsity, non-zero element ratio, number of non-zero elements, number of zero elements, variance, and word-embedding mapping value.

[0165] In one possible implementation, the mathematical model expression is used to solve one of the factory scheduling problem, the supply and demand simulation problem, and the energy scheduling problem; the instance data is instance data of one of the factory scheduling problem, the supply and demand simulation problem, and the energy scheduling problem; and the solution result is used to guide the solution of one of the factory scheduling problem, the supply and demand simulation problem, and the energy scheduling problem.

[0166] For the detailed description of the above optional methods, please refer to the above method embodiments, which will not be repeated here. In addition, the explanation of any of the above data processing devices and the description of the beneficial effects can refer to the above corresponding method embodiments, which will not be repeated here.

[0167] Among them, the acquisition module 510, the processing module 520, the segmentation module 530, and the generation module 540 can all be implemented by software or by hardware. For example, the implementation of the acquisition module 510 is described below using the acquisition module 510 as an example. Similarly, the implementation of the processing module 520, the segmentation module 530, and the generation module 540 can refer to the implementation of the acquisition module 510.

[0168] As an example of a software functional unit, the acquisition module 510 may include code running on a computing instance. The computing instance may include at least one of a physical host (computing device), a virtual machine, and a container. Furthermore, the computing instance may be one or more. For example, the acquisition module 510 may include code running on multiple hosts / virtual machines / containers. It should be noted that the multiple hosts / virtual machines / containers used to run the code may be distributed in the same region or in different regions. Furthermore, the multiple hosts / virtual machines / containers used to run the code may be distributed in the same availability zone (AZ) or in different AZs, each AZ including one data center or multiple geographically close data centers. Typically, a region may include multiple AZs.

[0169] Similarly, multiple hosts / virtual machines / containers running the code can be distributed within the same virtual private cloud (VPC) or across multiple VPCs. Typically, a VPC is set up within a region. Cross-region communication between two VPCs within the same region, or between VPCs in different regions, requires a communication gateway within each VPC to interconnect the VPCs.

[0170] As an example of a hardware functional unit, the acquisition module 510 may include at least one computing device, such as a server. Alternatively, the acquisition module 510 may be implemented using an application-specific integrated circuit (ASIC) or a programmable logic device (PLD). The PLD may be a complex programmable logical device (CPLD), a field-programmable gate array (FPGA), a generic array logic (GAL), or any combination thereof.

[0171] The multiple computing devices included in acquisition module 510 can be distributed in the same region or in different regions. The multiple computing devices included in acquisition module 510 can be distributed in the same AZ or in different AZs. Similarly, the multiple computing devices included in acquisition module 510 can be distributed in the same VPC or in multiple VPCs. The multiple computing devices can be any combination of servers, ASICs, PLDs, CPLDs, FPGAs, GALs, and other computing devices.

[0172] It should be noted that in other embodiments, the acquisition module 510 can be used to execute any step in the data processing method, the processing module 520 can be used to execute any step in the data processing method, the segmentation module 530 can be used to execute any step in the data processing method, and the generation module 540 can be used to execute any step in the data processing method. The steps that the acquisition module 510, processing module 520, segmentation module 530, and generation module 540 are responsible for implementing can be specified as needed. By having the acquisition module 510, processing module 520, segmentation module 530, and generation module 540 respectively implement different steps in the data processing method, the full functionality of the data processing device can be achieved. This application also provides a computing device 100. As shown in Figure 8, the computing device 100 includes: a bus 102, a processor 104, a memory 106, and a communication interface 108. The processor 104, memory 106, and communication interface 108 communicate with each other via the bus 102. The computing device 100 can be a server or a terminal device. It should be understood that this application does not limit the number of processors and memories in the computing device 100.

[0173] Bus 102 may be a Peripheral Component Interconnect (PCI) bus or an Extended Industry Standard Architecture (EISA) bus, among others. Buses may be classified as address buses, data buses, control buses, and the like. For ease of illustration, FIG8 shows a single bus line, but this does not imply a single bus or type of bus. Bus 104 may include a path for transmitting information between various components of computing device 100 (e.g., memory 106, processor 104, and communication interface 108).

[0174] The processor 104 may include any one or more processors such as a central processing unit (CPU), a graphics processing unit (GPU), a microprocessor (MP), or a digital signal processor (DSP).

[0175] The memory 106 may include volatile memory, such as random access memory (RAM). The processor 104 may also include non-volatile memory, such as read-only memory (ROM), flash memory, hard disk drive (HDD), or solid state drive (SSD).

[0176] The memory 106 stores executable program code, which the processor 104 executes to implement the functions of the acquisition module 510, processing module 520, segmentation module 530, and generation module 540, thereby implementing the data processing method. In other words, the memory 106 stores instructions for executing the data processing method.

[0177] Alternatively, the memory 106 stores executable codes, and the processor 104 executes the executable codes to respectively implement the functions of the aforementioned data processing apparatus, thereby implementing the data processing method. That is, the memory 106 stores instructions for executing the data processing method.

[0178] The communication interface 108 uses a transceiver module such as, but not limited to, a network interface card or a transceiver to implement communication between the computing device 100 and other devices or a communication network.

[0179] Embodiments of the present application also provide a computing device cluster. The computing device cluster includes at least one computing device. The computing device can be a server, such as a central server, an edge server, or a local server in a local data center. In some embodiments, the computing device can also be a terminal device such as a desktop computer, a laptop computer, or a smartphone.

[0180] As shown in Figure 9, the computing device cluster includes at least one computing device 100. The memory 106 in one or more computing devices 100 in the computing device cluster may store the same instructions for executing the data processing method.

[0181] In some possible implementations, the memory 106 of one or more computing devices 100 in the computing device cluster may also store some instructions for executing the data processing method. In other words, the combination of one or more computing devices 100 can jointly execute the instructions for executing the data processing method.

[0182] It should be noted that the memory 106 in different computing devices 100 in the computing device cluster can store different instructions, each for executing a portion of the functions of the data processing apparatus. That is, the instructions stored in the memory 106 in different computing devices 100 can implement the functions of one or more of the acquisition module 510, processing module 520, segmentation module 530, and generation module 540.

[0183] In some possible implementations, one or more computing devices in a computing device cluster may be connected via a network. The network may be a wide area network or a local area network, etc. FIG10 shows a possible implementation. As shown in FIG10 , two computing devices 100A and 100B are connected via a network. Specifically, the network is connected via a communication interface in each computing device. In this type of possible implementation, the memory 106 in the computing device 100A stores instructions for executing the functions of the acquisition module 510. At the same time, the memory 106 in the computing device 100B stores instructions for executing the functions of the processing module 520, the segmentation module 530, and the generation module 540.

[0184] The connection method between the computing device clusters shown in Figure 10 can be considered to be that the data processing method provided in this application requires a large amount of data storage and calculation data, so it is considered to entrust the functions implemented by the processing module 520, the segmentation module 530 and the generation module 540 to the computing device 100B for execution.

[0185] It should be understood that the functions of the computing device 100A shown in FIG10 may also be completed by multiple computing devices 100. Similarly, the functions of the computing device 100B may also be completed by multiple computing devices 100.

[0186] The present application also provides another computing device cluster. The connection relationship between the computing devices in this computing device cluster can be similar to the connection method of the computing device cluster described in Figures 8 and 9. However, the memory 106 of one or more computing devices 100 in this computing device cluster can store the same instructions for executing the data processing method.

[0187] In some possible implementations, the memory 106 of one or more computing devices 100 in the computing device cluster may also store some instructions for executing the data processing method. In other words, the combination of one or more computing devices 100 can jointly execute the instructions for executing the data processing method.

[0188] It should be noted that the memory 106 in different computing devices 100 in the computing device cluster can store different instructions for executing partial functions of the data processing system. In other words, the instructions stored in the memory 106 in different computing devices 100 can implement the functions of one or more devices in the data processing device.

[0189] The present application also provides a computer program product comprising instructions. The computer program product may be software or a program product comprising instructions that can be run on a computing device or stored in any available medium. When the computer program product is run on at least one computing device, the at least one computing device executes the data processing method.

[0190] The embodiment of the present application also provides a computer-readable storage medium. The computer-readable storage medium can be any available medium that can be stored by a computing device or a data storage device such as a data center that contains one or more available media. The available medium can be a magnetic medium (e.g., a floppy disk, a hard disk, a magnetic tape), an optical medium (e.g., a DVD), or a semiconductor medium (e.g., a solid-state drive). The computer-readable storage medium includes instructions that instruct a computing device to execute a data processing method, or instructs a computing device to execute a data processing method.

[0191] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention, rather than to limit it. Although the present invention has been described in detail with reference to the aforementioned embodiments, those skilled in the art should understand that they can still modify the technical solutions described in the aforementioned embodiments, or make equivalent replacements for some of the technical features therein. However, these modifications or replacements do not deviate the essence of the corresponding technical solutions from the protection scope of the technical solutions of the various embodiments of the present invention.

Claims

1. A data processing method, characterized in that: Applied to a mathematical programming solver, the mathematical programming solver is deployed on a computing device, the method comprising: Acquire a mathematical model expression and an instance data file, wherein the mathematical model expression includes at least one sub-expression, the sub-expression is an objective function or a constraint condition, the sub-expression includes variables and coefficients, and the instance data file includes instance data, and the instance data is used to indicate the values ​​of the coefficients in each of the sub-expressions of the mathematical model expression; Converting the mathematical model expression and the instance data into an original data block, wherein the original data block is a matrix for indicating the values ​​of the coefficients in each of the sub-expressions of the mathematical model expression; The original data block is divided into a plurality of sub-data blocks, wherein data features of the plurality of sub-data blocks meet a similarity threshold, the data features are used to indicate the sparsity or uniformity of the data block, and the similarity threshold is used to indicate the degree of similarity between the data features of the plurality of sub-data blocks; The multiple sub-data blocks are assigned to multiple threads for parallel processing to generate a solution file, and the solution file is used to perform solution operations on the sub-expressions corresponding to the multiple sub-data blocks to obtain a solution result of the mathematical model expression.

2. The method according to claim 1, characterized in that The converting the mathematical model expression and the instance data into an original data block comprises: Converting the mathematical model expression into a mathematical model expression tree; Determining that the mathematical model expression tree includes at least one first structure to determine that the mathematical model expression has a nested structure, wherein the first structure is used to indicate the structure of the mathematical model expression corresponding to the nested structure; Performing expression expansion processing on the mathematical model expression to obtain an expanded model expression; Substitute the instance data into the expanded mathematical model expression to obtain an original data block of the mathematical model expression.

3. The method according to claim 2, characterized in that The step of performing expression expansion processing on the mathematical model expression includes: Transforming the first structure in the mathematical model expression tree according to a specified rule to obtain a transformed model expression tree, wherein the transformed model expression tree is a model expression tree corresponding to the expanded mathematical model expression; The specified rule is to replace the first structure with a second structure, and the second structure is used to indicate the structure of the mathematical model expression corresponding to the expanded structure of the nested structure.

4. The method according to any one of claims 1 to 3, characterized in that: The data feature includes sparsity, and the method further includes: Determining the similarity threshold, where the similarity threshold is used to indicate the similarity between the sparsities of the plurality of sub-data blocks; The step of dividing the original data block into a plurality of sub-data blocks comprises: According to the similarity threshold, the original data block is equally divided into the multiple sub-data blocks, and the similarity between the sparsity of the multiple sub-data blocks meets the similarity threshold.

5. The method according to claim 4, characterized in that The determining the similarity threshold comprises: Input the data feature into an optimal threshold selection model to obtain the similarity threshold output by the optimal threshold selection model; or, The similarity threshold input by a user is received.

6. The method according to claim 5, characterized in that The optimal threshold selection model is a neural network model trained by data features of sample data blocks and similarity threshold labels, wherein the data features of the sample data blocks include the sparsity of the sample data blocks, and the similarity threshold labels are similarity thresholds manually labeled for the sample data blocks.

7. The method according to any one of claims 1 to 6, characterized in that: The step of dividing the original data block into a plurality of sub-data blocks comprises: Convert the original data block into a sparse matrix format COO to obtain an original data block after format conversion; The original data block after the format conversion is divided into the multiple sub-data blocks.

8. The method according to any one of claims 1 to 7, characterized in that: The step of allocating the plurality of sub-data blocks to a plurality of threads for parallel processing to generate a solution file comprises: Allocating the multiple sub-data blocks to the multiple threads for parallel processing to generate sub-constraint blocks, wherein the sub-constraint blocks are used to indicate a mapping relationship between the sub-data blocks and the sub-expressions that are constraint conditions in the mathematical model expression and a mapping relationship between the sub-data blocks and / or the sub-expressions that are objective functions in the mathematical model expression; The sub-constraint blocks are spliced ​​to generate the solution file.

9. The method according to any one of claims 1 to 8, characterized in that: The data feature includes one of sparsity, non-zero element ratio, number of non-zero elements, number of zero elements, matrix variance, and mapping value of word-embedding.

10. The method according to any one of claims 1 to 9, characterized in that: The mathematical model expression is used to solve one of the factory production scheduling problems, supply and demand simulation problems, and energy scheduling problems. The instance data is instance data of one of the factory production scheduling problems, supply and demand simulation problems, and energy scheduling problems. The solution result is used to guide the solution of one of the factory production scheduling problems, supply and demand simulation problems, and energy scheduling problems.

11. A data processing device, characterized in that: Applied to a mathematical programming solver, the mathematical programming solver is deployed on a computing device, the computing device has multiple threads, and the apparatus comprises: An acquisition module, used for acquiring a mathematical model expression and an instance data file, wherein the mathematical model expression includes at least one sub-expression, the sub-expression is an objective function or a constraint condition, the sub-expression includes variables and coefficients, and the instance data file includes instance data, and the instance data is used for indicating the values ​​of the coefficients in each of the sub-expressions of the mathematical model expression; A processing module, used for converting the mathematical model expression and the instance data into an original data block, wherein the original data block is a matrix for indicating the values ​​of the coefficients in each of the sub-expressions of the mathematical model expression; A segmentation module, used for segmenting the original data block into a plurality of sub-data blocks, wherein data features of the plurality of sub-data blocks meet a similarity threshold, the data features are used to indicate the sparsity or uniformity of the data block, and the similarity threshold is used to indicate the degree of similarity between the data features of the plurality of sub-data blocks; A generation module is used to assign the multiple sub-data blocks to multiple threads for parallel processing to generate a solution file, and the solution file is used to solve the sub-expressions corresponding to the multiple sub-data blocks to obtain the solution result of the mathematical model expression.

12. The device according to claim 11, characterized in that The processing module is further used to convert the mathematical model expression into a mathematical model expression tree; determine that the mathematical model expression tree includes at least one first structure to determine that the mathematical model expression has a nested structure, and the first structure is used to indicate the structure of the mathematical model expression corresponding to the nested structure; Performing expression expansion processing on the mathematical model expression to obtain an expanded model expression; Substitute the instance data into the expanded mathematical model expression to obtain an original data block of the mathematical model expression.

13. The device according to claim 12, characterized in that The processing module is also used to transform the first structure in the mathematical model expression tree according to a specified rule to obtain a transformed model expression tree, and the transformed model expression tree is the model expression tree corresponding to the expanded mathematical model expression; wherein the specified rule is to replace the first structure with a second structure, and the second structure is used to indicate the structure of the mathematical model expression corresponding to the expanded structure of the nested structure.

14. The device according to any one of claims 11 to 13, characterized in that The data feature includes sparsity, and the device further includes: A determination module, used to determine the similarity threshold, where the similarity threshold is used to indicate the similarity between the sparsities of the multiple sub-data blocks; The processing module is further configured to equally divide the original data block into the multiple sub-data blocks according to the similarity threshold, and the similarity between the sparsities of the multiple sub-data blocks meets the similarity threshold.

15. The device according to claim 14, characterized in that The determination module is further used to input the data feature into an optimal threshold selection model to obtain the similarity threshold output by the optimal threshold selection model; or to receive the similarity threshold input by a user.

16. The device according to claim 15, characterized in that The optimal threshold selection model is a neural network model trained by data features of sample data blocks and similarity threshold labels, wherein the data features of the sample data blocks include the sparsity of the sample data blocks, and the similarity threshold labels are similarity thresholds manually labeled for the sample data blocks.

17. The device according to any one of claims 11 to 16, characterized in that The processing module is further used to convert the original data block into a sparse matrix format COO to obtain an original data block after format conversion; and divide the original data block after format conversion into the multiple sub-data blocks.

18. The device according to any one of claims 11 to 17, characterized in that The generating module is further used to assign the multiple sub-data blocks to the multiple threads for parallel processing, and generate sub-constraint blocks, wherein the sub-constraint blocks are used to indicate the mapping relationship between the sub-data blocks and the sub-expressions that are constraint conditions in the mathematical model expression, and the mapping relationship between the sub-data blocks and / or the sub-expressions that are objective functions in the digital model expression; The sub-constraint blocks are spliced ​​to generate the solution file.

19. The device according to any one of claims 11 to 18, characterized in that The data feature includes one of sparsity, non-zero element ratio, number of non-zero elements, number of zero elements, matrix variance, and mapping value of word-embedding.

20. The device according to any one of claims 11 to 19, characterized in that The mathematical model expression is used to solve one of the factory production scheduling problems, supply and demand simulation problems, and energy scheduling problems. The instance data is instance data of one of the factory production scheduling problems, supply and demand simulation problems, and energy scheduling problems. The solution result is used to guide the solution of one of the factory production scheduling problems, supply and demand simulation problems, and energy scheduling problems.

21. A computing device cluster, characterized in that: It includes at least one computing device, each computing device includes: a processor and a memory, the processor of the at least one computing device is used to execute instructions stored in the memory of the at least one computing device, so that the computing device cluster executes the data processing method as described in any one of claims 1 to 10.

22. A computer-readable storage medium, characterized in that: The method comprises computer program instructions. When the computer program instructions are executed by a computing device cluster, the computing device cluster performs the data processing method according to any one of claims 1 to 10.

23. A computer program product, characterized in that The computer program product comprises instructions, and when the instructions are executed by a computing device cluster, the computing device cluster executes the data processing method according to any one of claims 1 to 10.

Images