Dynamic adjustment strategy method and device based on batch import data
By determining the data influencing factor in the database and using the batch of the batch model to dynamically adjust the batch size, the problem that the existing technology cannot accurately grasp the data import batch size, and improves the efficiency of data introduction.
Patent Information
- Application Number
- CN202411482106.X
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2024-10-23
- Publication Date
- 2025-01-28
AI Technical Summary
In the prior art, the batch size during the data import process cannot be accurately grasped, resulting in waste of database resources and inefficient import.
By determining the data to the data to be inserted, and entering these factors to the batch prediction model, the batch size of the data submission of the data is dynamically adjusted to optimize the data import efficiency.
Under the limited database and server resources, the batch size is dynamically adjusted, thereby improving the efficiency of data introduction and avoiding waste of resources.
Smart Images
Figure CN119377222A_ABST
Abstract
Description
Technical Field
[0001] The present invention relates to the field of database technology, and in particular to a method and device for dynamically adjusting a strategy based on batch import of data. Background Art
[0002] With the development of banking business, the amount of data and tables generated by the business is increasing, and developers need to insert this data into the database as quickly as possible. The current data import methods are: 1. Single-line reading: read the content of the text file in a single line, assemble the content into an insert statement through the programming language and insert it into the database. This method consumes too many database connections and causes a waste of database resources. 2. Load into memory import: read all the data of the entire text file and load it into the server's memory, and insert the data in the memory into the database at one time through tools such as the big data batch processing framework. This method has high requirements on the server's memory size. When the size of the text file is too large, the consumption of the server's memory is too large. If the server's memory limit is exceeded, some data will not be inserted into the database. 3. Batch insertion: read the content of the text file in batches. First, read multiple lines of files at a time, assemble batch insert statements through the programming language and insert them into the database, and then read the next batch of file contents and insert them into the database until the content of the text file is read. The number of text lines submitted at one time by this method is difficult to determine. If the batch size is set too large, database insertion may take a long time or even cause errors such as memory overflow. If it is set too small, the database and server resources cannot be fully utilized, resulting in low import efficiency. Summary of the invention
[0003] In order to solve the problem in the prior art that the batch size cannot be accurately grasped during the data import process, the embodiments of this specification provide a method and device for a dynamic adjustment strategy based on batch import of data.
[0004] An embodiment of the present specification provides a dynamic adjustment strategy method based on batch import of data, the method comprising: determining data influencing factors of a table to be inserted, the data influencing factors comprising the number of indexes, index size, number of table fields and row data size of the table to be inserted; inputting the number of indexes, index size, number of table fields and row data size of the table to be inserted into a batch prediction model to obtain a prediction result output by the batch prediction model, wherein the batch prediction model is obtained by training a sample table of a batch prediction initial model using data influencing factors.
[0005] According to one aspect of an embodiment of the present specification, a batch prediction initial model is trained in the following manner: a training sample data set is obtained, wherein the training sample data set includes data influencing factors of a sample table and an optimal batch size of the sample table; the data influencing factors of the sample table are input into the prediction initial model to obtain a predicted batch output by the batch prediction initial model and inserted into a database; a loss function is used to calculate a loss value of the predicted batch output by the batch prediction initial model and the optimal batch, and the batch prediction initial model is iterated based on the loss value to obtain a batch prediction model.
[0006] According to one aspect of an embodiment of the present specification, the optimal batch size of the sample table is determined in the following manner: multiple batch sizes are preset for each sample table; the time consumed for inserting each sample table into the database based on each batch size is counted; the batch size corresponding to the shortest time consumption is determined, and the batch size is used as the optimal batch size of the sample table.
[0007] According to one aspect of an embodiment of the present specification, the optimal batch size of the sample table is determined in the following manner: multiple batch sizes are preset for each sample table; the time consumed for inserting each sample table into the database based on each batch size is counted; the batch size corresponding to the shortest time consumption is determined, and the batch size is used as the optimal batch size of the sample table.
[0008] According to one aspect of the embodiments of this specification, the loss function is expressed as follows: in, Represents the return value of the batch prediction initial model, that is, the predicted batch output by the batch prediction initial model, y i is the true value of the batch corresponding to the i-th training sample.
[0009] According to one aspect of an embodiment of the present specification, using a loss function to calculate the loss value of the predicted batch output by the batch prediction initial model and the optimal batch includes: storing the data influencing factors of each sample table, the preset batch size, and the time consumed to insert the sample table into the database based on each batch size in a configuration table; and determining the batch size corresponding to the shortest time from the configuration table.
[0010] According to one aspect of an embodiment of the present specification, the method further includes: storing the predicted batches output by the batch prediction initial model in a configuration table respectively; calculating the loss value using a loss function according to the batch size corresponding to the shortest time recorded in the configuration table and the predicted batch.
[0011] The embodiment of the present specification also provides a dynamic adjustment strategy device based on batch import of data, the device comprising: a determination unit, used to determine the data influencing factors of the table to be inserted, the data influencing factors including the number of indexes, index size, number of table fields and row data size of the table to be inserted; a prediction unit, used to input the number of indexes, index size, number of table fields and row data size of the table to be inserted into a batch prediction model to obtain a prediction result output by the batch prediction model, wherein the batch prediction model is obtained by training the sample table of the batch prediction initial model with the data influencing factors.
[0012] The embodiment of the present specification also provides a computer device, which includes a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor implements the dynamic adjustment strategy method based on batch imported data when executing the computer program.
[0013] The embodiment of the present specification also provides a computer-readable storage medium, wherein the computer-readable storage medium stores a computer program, and when the computer program is executed by a processor, the dynamic adjustment strategy method based on batch imported data is implemented.
[0014] The embodiments of the present specification also provide a computer program product, which includes a computer program. When the computer program is executed by a processor, the dynamic adjustment strategy method based on batch imported data is implemented.
[0015] This solution uses machine learning to monitor the time it takes to insert data of different batch sizes into the database, and dynamically adjusts the batch size of data submission to maximize the efficiency of data import under the premise of limited database and server resources. BRIEF DESCRIPTION OF THE DRAWINGS
[0016] In order to more clearly illustrate the embodiments of this specification or the technical solutions in the prior art, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of this specification. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative work.
[0017] Figure 1 The flowchart of a method for dynamically adjusting a strategy based on batch importing data according to an embodiment of the present specification is shown;
[0018] Figure 2 The flowchart of a method for training batch prediction initial model according to an embodiment of this specification is shown;
[0019] Figure 3Shown is a flow chart of a method for determining an optimal batch size according to an embodiment of this specification;
[0020] Figure 4 The figure shows a flow chart of a method for recording an optimal batch according to an embodiment of the present specification;
[0021] Figure 5 The figure shows a schematic diagram of the structure of a dynamic adjustment strategy device based on batch import data according to an embodiment of the present specification;
[0022] Figure 6 The figure shows a schematic diagram of the structure of a computer device according to an embodiment of the present specification.
[0023] Description of the accompanying symbols:
[0024] 501. Determine the unit;
[0025] 502. prediction unit;
[0026] 602. Computer equipment;
[0027] 604, processor;
[0028] 606. Memory;
[0029] 608, driving mechanism;
[0030] 610, input / output module;
[0031] 612. Input devices;
[0032] 614. Output devices;
[0033] 616. Presentation equipment;
[0034] 618. Graphical user interface;
[0035] 620, network interface;
[0036] 622, communication link;
[0037] 624. Communication bus. DETAILED DESCRIPTION
[0038] In order to enable those skilled in the art to better understand the technical solutions in this specification, the technical solutions in the embodiments of this specification will be clearly and completely described below in conjunction with the drawings in the embodiments of this specification. Obviously, the described embodiments are only part of the embodiments of this specification, not all of the embodiments. Based on the embodiments in this specification, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of this specification.
[0039] It should be noted that the terms "first", "second", etc. in the description and claims of this specification and the above-mentioned drawings are used to distinguish similar objects, and are not necessarily used to describe a specific order or sequence. It should be understood that the data used in this way can be interchanged where appropriate, so that the embodiments of this specification described herein can be implemented in an order other than those illustrated or described herein. In addition, the terms "including" and "having" and any variations thereof are intended to cover non-exclusive inclusions, for example, a process, method, device, product or equipment that includes a series of steps or units is not necessarily limited to those steps or units clearly listed, but may include other steps or units that are not clearly listed or inherent to these processes, methods, products or equipment.
[0040] This specification provides method operation steps as described in the embodiments or flow charts, but more or fewer operation steps may be included based on routine or non-creative work. The order of steps listed in the embodiments is only one way of executing the order of many steps and does not represent the only execution order. When the system or device product is executed in practice, it can be executed in the order of the method shown in the embodiments or the drawings or in parallel.
[0041] It should be noted that the dynamic adjustment strategy method based on batch import data in this specification can be used in the field of software development and in the field of computer technology. This specification does not limit the application field of the dynamic adjustment strategy method based on batch import data.
[0042] Figure 1 The flowchart of a method for dynamically adjusting a strategy based on batch importing data according to an embodiment of the present specification is shown, which specifically includes the following steps:
[0043] Step 101, determine the data influencing factors of the table to be inserted, the data influencing factors include the number of indexes, index size, number of table fields and row data size of the table to be inserted. In the embodiment of this specification, each table has a corresponding data influencing factor, and the data influencing factor can be used to respectively represent the size of the table to be inserted, the situation of the index established in the table, etc. In the process of inserting a piece of data or a data table into a database, the size of the data table, the size and number of fields contained in the table, the size of a row of data, and the size and field of the index established in the table will all affect the time spent on the insertion operation.
[0044] In the embodiments of this specification, the process of inserting a piece of data or a data table into a database includes: establishing a connection between the database and the client, sending an insert data statement to the database, querying and analyzing data (including checking the SQL syntax), inserting the data into the data table, updating the index, and shutting down the server and the client. Among them, for batch data insertion operations, inserting records and updating indexes are the most time-consuming links. Therefore, it is necessary for this specification to count the time spent inserting data into the database, and to study the correlation between different data influencing factors and the time spent inserting data into the database.
[0045] Step 102, input the number of indexes, index size, number of table fields and row data size of the table to be inserted into the batch prediction model to obtain the prediction result output by the batch prediction model, wherein the batch prediction model is obtained through the data influencing factors of the batch prediction initial model training sample table.
[0046] In this step, in order to obtain the time consumption of inserting the table to be inserted into the database, the data influencing factors of the table to be inserted are input into the pre-trained batch prediction model, and the batch prediction model outputs the prediction results to obtain the prediction model and the measured time consumption of inserting the table to be inserted into the database. Among them, the batch prediction model is obtained by training the data influencing factors of the sample table by the batch prediction initial model. In the embodiment of this specification, all tables that need to be imported in batches are divided into training sets and test sets in a ratio of 6:3:1. Therefore, the sample table and the table to be inserted are prepared together in advance.
[0047] Figure 2 The flowchart of a method for training batch prediction initial model according to an embodiment of the present specification is shown, which specifically includes the following steps:
[0048] Step 201, obtain a training sample data set, the training sample data set includes data influencing factors of the sample table and the optimal batch size of the sample table. In the embodiment of this specification, in order to count the time required for inserting the sample table into the database according to different batch sizes, different batch sizes are set for each sample table, and the sample table is inserted into the database according to different batch sizes to obtain the time consumed for inserting into the database according to each batch size. The batch size corresponding to the shortest time is called the optimal batch size. And the optimal batch size is used as the label of the initial model for training batch prediction in the embodiment of this specification.
[0049] Step 202, input the data influencing factors of the sample table into the batch prediction initial model, and obtain the predicted batch inserted into the database output by the batch prediction initial model. In this step, the data influencing factors of the sample table are the input of the batch prediction initial model, and the batch prediction initial model outputs the predicted batch corresponding to the sample table. Among them, for example, the predicted batch output by the batch prediction initial model is 9, which means that when the sample table is imported into the database, the sample table is imported at a scale of 9 rows at a time.
[0050] Step 203, using the loss function to calculate the loss value of the predicted batch output by the batch prediction initial model and the optimal batch, iterating the batch prediction initial model based on the loss value to obtain a batch prediction model.
[0051] In this specification, the predicted batch output by the batch prediction initial model during the training process may not be an accurate value, or the expected import efficiency may not be achieved according to the predicted batch. Therefore, this step needs to analyze the predicted batch output by the batch prediction initial model during the model training process with the optimal batch, so as to iterate and adjust the batch prediction initial model until the training effect of the batch prediction initial model reaches the expected level, and the batch prediction training model training is determined to be completed, and the batch prediction model is constructed.
[0052] The specific description of the loss function in the embodiment of this specification is described below, and this step is not repeated here.
[0053] Figure 3 The flowchart of a method for determining an optimal batch size according to an embodiment of the present specification is shown, which specifically includes the following steps:
[0054] Step 301, preset multiple batch sizes for each sample table. In the embodiment of this specification, in order to determine the optimal batch corresponding to the shortest time consumption for importing each sample table into the database. This step sets different batch sizes for each sample table, wherein only one batch size can be set for a sample table at a time, or the sample table can be split and different batch sizes can be set for different parts of the sample table.
[0055] For example, different batch sizes are set for the sample table A as a whole, namely 10, 15, and 20. When the batch size is 10, all data rows in the sample table A are imported into the database at a rate of 10 rows / time; when the batch size is 20, all data rows in the sample table A are imported into the database at a rate of 20 rows / time; when the batch size is 30, all data rows in the sample table A are imported into the database at a rate of 30 rows / time.
[0056] For another example, different batch sizes are set for parts of sample table A, which are 10, 15, and 20 respectively. Then one way to import the sample table into the database is: divide the data rows in sample table A into three parts, wherein the first part of the data is imported into the database at a size of 10 rows / time; the second part of the data in sample table A is imported into the database at a size of 20 rows / time; the third part of the data in sample table A is imported into the database at a size of 30 rows / time, thereby realizing the batch import of sample table A into the database according to different batch sizes. In the embodiments of this specification, sample table A is not limited to being divided into three parts, it can be divided into more parts, or two parts, and correspondingly, the batch size set for the part of sample table A is not limited to 10, 15 or 20, and the batch size can be set to other values, which are not limited in this specification.
[0057] Step 302, respectively count the time it takes for each sample table to be inserted into the database based on each batch size. In this step, each sample table is imported into the database based on different batch sizes, and the time it takes for each sample table to be imported into the database is counted. For example, the sample table set includes sample table A, sample table B, sample table C, and sample table D. Each sample table corresponds to 5 batch sizes, and these 4 sample tables are respectively imported into the database according to the set batch sizes, and the time it takes for these 4 sample tables to be imported into the database is counted.
[0058] Step 303, determine the batch size corresponding to the shortest time consumption, and use the batch size as the optimal batch size of the sample table. This step determines the minimum time consumption for importing multiple sample tables into the database according to step 302, determines the shortest time consumption, and determines the batch size corresponding to the shortest time consumption. Among them, the time consumption corresponding to multiple different batch sizes is counted for each sample table, so a shortest time consumption can be selected for each sample table, so that each sample table has a corresponding optimal batch size, and the optimal batch size is used as the true value or label of the sample table in the batch prediction initial model training process.
[0059] In one aspect of the embodiments of this specification, Among them, Loss(y,y p ) represents the loss function, Indicates the return value of the batch prediction initial model when the data influencing factors of the i-th sample table are input, that is, the predicted batch output by the batch prediction initial model, y i is the true value of the batch size corresponding to the i-th training sample, Represents the logarithm of the hyperbolic sine inverse of the error between the return value output by the batch prediction model and the true value of the batch size for the ith sample table. In the embodiments of this specification, the initial batch prediction model is continuously optimized with the goal of minimizing the value of the loss function or reaching a preset number of model iterations, and finally a trained machine learning model is obtained and saved to obtain a trained batch prediction model.
[0060] Figure 4 The flowchart of a method for recording an optimal batch according to an embodiment of the present specification is shown, which specifically includes the following steps:
[0061] Step 401, the data influencing factors of each sample table, the preset batch size, and the time taken to insert the sample table into the database based on each batch size are stored in the configuration table. In this step, before the model training starts, the data influencing factors used to characterize the size and content of the sample table are stored in the configuration table. At the same time, the time taken to insert the sample table into the database under different batch sizes is also recorded in the configuration table.
[0062] Step 402, determine the optimal batch size corresponding to the shortest time from the configuration table. In this step, the shortest time of each sample table is determined according to the time consumption of each sample table under different batch sizes recorded in the configuration table, so as to determine the optimal batch size of each sample table, which is further used as the training sample data set in the initial model of training batch prediction.
[0063] The embodiments of this specification also include: storing the predicted batches output by the batch prediction initial model in the configuration table respectively; using the loss function to calculate the loss value according to the optimal batch and the predicted batch recorded in the configuration table.
[0064] Figure 5 The figure shows a schematic diagram of the structure of a dynamic adjustment strategy device based on batch imported data according to an embodiment of the present specification. The basic structure of the dynamic adjustment strategy device based on batch imported data is described in this figure, wherein the functional units and modules can be implemented in software, or a general chip or a specific chip can be used to implement the dynamic adjustment strategy based on batch imported data. The device specifically includes:
[0065] A determination unit 501 is used to determine factors affecting the data to be inserted into the table, wherein the factors affecting the data include the number of indexes, the index size, the number of table fields and the row data size of the table to be inserted;
[0066] The prediction unit 502 is used to input the number of indexes, index size, number of table fields and row data size of the table to be inserted into the batch prediction model to obtain the prediction results output by the batch prediction model, wherein the batch prediction model is obtained by the data influencing factors of the batch prediction initial model training sample table. This solution uses machine learning to monitor the time consumed by inserting data of different batch sizes into the database to dynamically adjust the batch size of data submission, so as to maximize the efficiency of data import under the premise of limited database and server resources.
[0067] like Figure 6 The diagram of a computer device provided by an embodiment of the specification is shown. The dynamic adjustment strategy method based on batch import data described in the present application can be applied to the computer device. The computer device 602 may include one or more processors 604, such as one or more central processing units (CPUs), each of which may implement one or more hardware threads. The computer device 602 may also include any memory 606, which is used to store any kind of information such as code, settings, data, etc. Non-restrictive, for example, the memory 606 may include any one or more combinations of the following: any type of RAM, any type of ROM, flash memory device, hard disk, optical disk, etc. More generally, any memory may use any technology to store information. Further, any memory may provide volatile or non-volatile retention of information. Further, any memory may represent a fixed or removable component of the computer device 602. In one case, when the processor 604 executes an associated instruction stored in any memory or a combination of memories, the computer device 602 may perform any operation of the associated instruction. The computer device 602 also includes one or more drive mechanisms 608 for interacting with any memory, such as a hard disk drive mechanism, an optical disk drive mechanism, etc.
[0068] The computer device 602 may also include an input / output module 610 (I / O) for receiving various inputs (via input devices 612) and for providing various outputs (via output devices 614). A specific output mechanism may include a presentation device 616 and an associated graphical user interface (GUI) 618. In other embodiments, the input / output module 610 (I / O), input device 612, and output device 614 may not be included, and the computer device 602 may be used as a computer device in a network. The computer device 602 may also include one or more network interfaces 620 for exchanging data with other devices via one or more communication links 622. One or more communication buses 624 couple the components described above together.
[0069] The communication link 622 may be implemented in any manner, for example, through a local area network, a wide area network (e.g., the Internet), a point-to-point connection, etc., or any combination thereof. The communication link 622 may include any combination of hardwired links, wireless links, routers, gateway functions, name servers, etc. governed by any protocol or combination of protocols.
[0070] Corresponds to Figures 1 to 4 The method in the embodiment of the present specification also provides a computer-readable storage medium, on which a computer program is stored, and when the computer program is executed by a processor, the steps of the above method are executed.
[0071] The embodiment of the present specification also provides a computer-readable instruction, wherein when the processor executes the instruction, the program therein causes the processor to execute the following Figures 1 to 4 The method shown.
[0072] The present specification also provides a computer program product, wherein the computer program product includes a computer program, wherein the computer program is executed by a processor as follows: Figures 1 to 4 The method shown.
[0073] It should be understood that in the various embodiments of this specification, the size of the serial numbers of the above-mentioned processes does not mean the order of execution. The execution order of each process should be determined by its function and internal logic, and should not constitute any limitation on the implementation process of the embodiments of this specification.
[0074] It should also be understood that in the embodiments of this specification, the term "and / or" is only a description of the association relationship of the associated objects, indicating that three relationships can exist. For example, A and / or B can represent: A exists alone, A and B exist at the same time, and B exists alone. In addition, the character " / " in this specification generally indicates that the associated objects before and after are in an "or" relationship.
[0075] Those of ordinary skill in the art will appreciate that the units and algorithm steps of each example described in conjunction with the embodiments disclosed in this specification can be implemented in electronic hardware, computer software, or a combination of the two. In order to clearly illustrate the interchangeability of hardware and software, the composition and steps of each example have been generally described in the above description according to function. Whether these functions are performed in hardware or software depends on the specific application and design constraints of the technical solution. Professional and technical personnel can use different methods to implement the described functions for each specific application, but such implementation should not be considered to be beyond the scope of this specification.
[0076] Those skilled in the art can clearly understand that, for the convenience and brevity of description, the specific working processes of the systems, devices and units described above can refer to the corresponding processes in the aforementioned method embodiments and will not be repeated here.
[0077] In the several embodiments provided in this specification, it should be understood that the disclosed systems, devices and methods can be implemented in other ways. For example, the device embodiments described above are only schematic. For example, the division of the units is only a logical function division. There may be other division methods in actual implementation, such as multiple units or components can be combined or integrated into another system, or some features can be ignored or not executed. In addition, the mutual coupling or direct coupling or communication connection shown or discussed can be an indirect coupling or communication connection through some interfaces, devices or units, or it can be an electrical, mechanical or other form of connection.
[0078] The units described as separate components may or may not be physically separated, and the components shown as units may or may not be physical units, that is, they may be located in one place or distributed on multiple network units. Some or all of the units may be selected according to actual needs to achieve the purpose of the embodiments of this specification.
[0079] In addition, each functional unit in each embodiment of this specification may be integrated into one processing unit, or each unit may exist physically separately, or two or more units may be integrated into one unit. The above integrated unit may be implemented in the form of hardware or in the form of software functional units.
[0080] If the integrated unit is implemented in the form of a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of this specification is essentially or the part that contributes to the prior art, or all or part of the technical solution can be embodied in the form of a software product, and the computer software product is stored in a storage medium, including a number of instructions to enable a computer device (which can be a personal computer, a server, or a network device, etc.) to perform all or part of the steps of the method described in each embodiment of this specification. The aforementioned storage medium includes: U disk, mobile hard disk, read-only memory (ROM, Read-Only Memory), random access memory (RAM, Random Access Memory), disk or optical disk and other media that can store program codes.
[0081] Specific embodiments are used in this specification to illustrate the principles and implementation methods of this specification. The description of the above embodiments is only used to help understand the methods and core ideas of this specification. At the same time, for those skilled in the art, according to the ideas of this specification, there will be changes in the specific implementation methods and application scope. In summary, the content of this specification should not be understood as a limitation on this specification.
Claims
1. A dynamic adjustment strategy method based on batch import data, characterized in that: The method comprises: Determine the data influencing factors of the table to be inserted, wherein the data influencing factors include the number of indexes, index size, number of table fields and row data size of the table to be inserted; The number of indexes, index size, number of table fields and row data size of the table to be inserted are input into the batch prediction model to obtain the prediction result output by the batch prediction model, wherein the batch prediction model is obtained through the data influencing factors of the batch prediction initial model training sample table.
2. The method according to claim 1, characterized in that The batch prediction initial model is trained as follows: Acquire a training sample data set, wherein the training sample data set includes data influencing factors of a sample table and an optimal batch size of the sample table; Inputting the data influencing factors of the sample table into the batch prediction initial model to obtain the predicted batch output by the batch prediction initial model and inserted into the database; The loss function is used to calculate the loss value of the predicted batch output by the batch prediction initial model and the optimal batch, and the batch prediction initial model is iterated based on the loss value to obtain a batch prediction model.
3. The method according to claim 2, characterized in that The optimal batch size for the sample table is determined as follows: Preset multiple batch sizes for each sample table; Count the time it takes to insert each sample table into the database based on each batch size; The batch size corresponding to the shortest time consumption is determined, and the batch size is used as the optimal batch size of the sample table.
4. The method according to claim 3, characterized in that The loss function is expressed as follows: in, Represents the return value of the batch prediction initial model, that is, the predicted batch output by the batch prediction initial model, y i is the true value of the batch corresponding to the i-th training sample.
5. The method according to claim 2, characterized in that: The method further comprises: The data influencing factors of each sample table, the preset batch size, and the time consumed to insert the sample table into the database based on each batch size are stored in the configuration table; The optimal batch size corresponding to the shortest time consumption is determined from the configuration table.
6. The method according to claim 5, characterized in that The loss function is used to calculate the loss value of the predicted batch output by the batch prediction initial model and the optimal batch, including: The prediction batches output by the batch prediction initial model are stored in the configuration table respectively; The loss function is used to calculate the loss value according to the optimal batch recorded in the configuration table and the predicted batch.
7. A dynamic adjustment strategy device based on batch import data, characterized in that: The device comprises: A determination unit, used to determine factors affecting the data to be inserted into the table, wherein the factors affecting the data include the number of indexes, the index size, the number of table fields and the row data size of the table to be inserted; The prediction unit is used to input the number of indexes, index size, number of table fields and row data size of the table to be inserted into the batch prediction model to obtain the prediction result output by the batch prediction model, wherein the batch prediction model is obtained by the data influencing factors of the batch prediction initial model training sample table.
8. A computer device comprising a memory, a processor and a computer program stored in the memory and executable on the processor, characterized in that: When the processor executes the computer program, the method according to any one of claims 1 to 6 is implemented.
9. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores a computer program, and when the computer program is executed by a processor, the method according to any one of claims 1 to 6 is implemented.
10. A computer program product, characterized in that The computer program product comprises a computer program, and when the computer program is executed by a processor, the method according to any one of claims 1 to 6 is implemented.