Data reading and writing batch processing method, server, and computer-readable storage medium

Abstract

The present invention relates to a data processing technology, and discloses a method for batch processing of data reading and writing. The method includes: when receiving a batch data reading and writing request for a single database or multiple databases from a calling party, respectively allocating a unique request ID for each reading and writing request; creating a storage structure for each of the reading and writing requests in a hash memory, including a key and a value structure, writing the request ID into the key, and writing the connection identifier of the calling party, the reading and writing SQL statements, and the reading and writing SQL numbers into the value structure; sending each reading and writing task of the reading and writing requests to the corresponding database for parallel processing according to the database identifier in each of the reading and writing requests; receiving the processing results returned by the database and updating them into the value structure corresponding to the reading and writing requests. The present invention also provides a server and a computer-readable storage medium. The present invention can perform parallel processing on batch reading and writing tasks and reduce the access time consumption.

Description

Technical Field

[0001] The present invention relates to the technical field of data processing, and particularly to a method for batch processing of data reading and writing, a server, and a computer-readable storage medium. Background Art

[0002] Currently, when dealing with batch data reading and writing in a single database or multiple databases, perfect processing cannot be achieved. For batch reading and writing in a single database, most current practices are that the processing module first executes an SQL task, and after the database processing returns, it executes the next SQL task. If the read or write processing fails, it returns to the calling party. For batch reading and writing in multiple databases, most current practices are that the processing module first determines which database the data table corresponding to the first SQL task is in, then executes the SQL task from that database, and after the data returns, it determines which database the data table corresponding to the second SQL task is in, and so on. If one of the SQL tasks fails to execute, it returns to the calling party.

[0003] The drawback of the above methods is that there is no sequential execution relationship between multiple SQL tasks, but the current practice performs sequential execution for convenient and simple processing, which takes too long. Moreover, for each additional type of reading and writing, logical code maintenance needs to be increased, increasing labor costs. Summary of the Invention

[0004] In view of this, the present invention proposes a method for batch processing of data reading and writing, a server, and a computer-readable storage medium to solve at least one of the above technical problems.

[0005] First, to achieve the above object, the present invention proposes a method for batch processing of data reading and writing, the method comprising the steps of:

[0006] When receiving a batch data reading and writing request for a single database or multiple databases from a calling party, a unique request ID is assigned to each reading and writing request, and the request parameters in the reading and writing request include a database identifier and a reading and writing task, wherein each reading and writing task includes a reading and writing SQL number and a reading and writing SQL statement to be executed;

[0007] A storage structure for each of the reading and writing requests is created in the hash memory, including a key and a value structure. The request ID is written into the key, and the connection identifier of the calling party corresponding to each reading and writing request, as well as the reading and writing SQL statement and the reading and writing SQL number corresponding to each reading and writing task in the reading and writing request, are written into the value structure as an array;

[0008] Each reading and writing task of the reading and writing request is sent to the corresponding database for parallel processing according to the database identifier in each reading and writing request; and

[0009] Receive the processing results of each read / write task for each of the read / write requests returned by the database and update them to the value structure corresponding to the read / write request.

[0010] Optionally, the method further includes the steps of:

[0011] According to the value structure, determine whether all the processing results of the read / write tasks of the read / write request have been returned;

[0012] When all the processing results of the read / write tasks of the read / write request have been returned, combine the processing results of all the read / write tasks of the read / write request and return them to the caller.

[0013] Optionally, in the step of sending the read / write tasks of the read / write request to the corresponding database for parallel processing according to the database identifier in each read / write request, the data of the read / write tasks sent to each database includes the request ID, the read / write sql number, and the read / write sql statement.

[0014] Optionally, the processing result includes the request ID, the read / write sql number, the task return identifier, and the task return data.

[0015] Optionally, the value structure further includes the task return identifier and the task return data corresponding to each read / write task, and the step of receiving the processing results of the read / write tasks of the read / write request returned by the database and updating them to the value structure corresponding to the read / write request includes:

[0016] Asynchronously receive the processing results returned by each database, obtain the corresponding value structure from the hash memory according to the request ID in each processing result, then find the corresponding read / write task in the value structure according to the read / write sql number, and update the task return identifier and the task return data of the read / write task.

[0017] Optionally, the task return identifier is used to indicate whether the execution of the read / write sql statement is successful. Returning the value 0 indicates successful execution, and a non-zero value indicates failure.

[0018] Optionally, the task return data is used to indicate the return data corresponding to the read / write sql statement. When the read / write sql statement is a query task and the execution is successful, the specific query result data is returned; when the read / write sql statement is a non-query task or the query task execution fails, the null value is returned.

[0019] Optionally, the method further includes the steps of:

[0020] When not all the processing results of the read / write tasks of the read / write request have been returned and waiting is required, continue to process other read / write requests at the same time.

[0021] In addition, to achieve the above object, the present invention further provides a server, including a memory and a processor. A data reading and writing batch processing system that can run on the processor is stored on the memory. When the data reading and writing batch processing system is executed by the processor, the steps of the data reading and writing batch processing method as described above are implemented.

[0022] Furthermore, to achieve the above object, the present invention further provides a computer-readable storage medium. The computer-readable storage medium stores a data reading and writing batch processing system, which can be executed by at least one processor, so that the at least one processor executes the steps of the data reading and writing batch processing method as described above.

[0023] Compared with the prior art, the data reading and writing batch processing method, server and computer-readable storage medium proposed by the present invention can provide a unified multi-database reading and writing task interface. When receiving a batch data reading and writing request, the corresponding information of each reading and writing task of each reading and writing request is saved in the hash memory using the unique request ID of each reading and writing request as the key, and the batch of reading and writing tasks are simultaneously sent to the corresponding databases for parallel processing. When receiving the processing results returned for each reading and writing task, they are also written into the hash memory, so that the batch reading and writing tasks of multiple databases can be processed concurrently without data chaos, and the access time is greatly reduced. BRIEF DESCRIPTION OF THE DRAWINGS

[0024] Figure 1 is a schematic diagram of an optional application environment architecture of each embodiment of the present invention;

[0025] Figure 2 is Figure 1 a schematic diagram of an optional hardware architecture of the server in

[0026] Figure 3 is a schematic diagram of the program module of the first embodiment of the data reading and writing batch processing system of the present invention;

[0027] Figure 4 is a schematic diagram of the storage structure created by the present invention in the hash memory;

[0028] Figure 5 is a schematic diagram of the program module of the second embodiment of the data reading and writing batch processing system of the present invention;

[0029] Figure 6 is a schematic flowchart of the first embodiment of the data reading and writing batch processing method of the present invention;

[0030] Figure 7 is a schematic flowchart of the second embodiment of the data reading and writing batch processing method of the present invention;

[0031] The realization, functional features and advantages of the present invention will be further described with reference to the accompanying drawings and embodiments. Detailed implementation manners

[0032] In order to make the objectives, technical solutions and advantages of the present invention more clearly understood, the present invention will be further described in detail below with reference to the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are only used to explain the present invention and are not used to limit the present invention. All other embodiments obtained by those of ordinary skill in the art based on the embodiments of the present invention without creative efforts shall fall within the protection scope of the present invention.

[0033] It should be noted that the descriptions involving "first", "second", etc. in the present invention are only for descriptive purposes and cannot be understood as indicating or implying their relative importance or implicitly indicating the quantity of the indicated technical features. Thus, the features defined with "first", "second" may explicitly or implicitly include at least one of such features. In addition, the technical solutions between various embodiments may be combined with each other, but it must be based on the fact that those of ordinary skill in the art can implement them. When the combination of technical solutions is contradictory or cannot be implemented, it should be considered that such a combination of technical solutions does not exist and is not within the protection scope required by the present invention.

[0034] Refer to Figure 1 As shown, it is a schematic diagram of an optional application environment architecture of various embodiments of the present invention.

[0035] In this embodiment, the present invention can be applied to an application environment including, but not limited to, a caller 1, a server 2, and a database 3. Among them, the caller 1 can be a mobile device such as a mobile phone, a smart phone, a notebook computer, a digital broadcast receiver, a PDA (Personal Digital Assistant), a PAD (Tablet Computer), a PMP (Portable Multimedia Player), a navigation device, an in-vehicle device, etc., and fixed terminals such as a digital TV, a desktop computer, a notebook, a server, etc. The server 2 can be a computing device such as a rack server, a blade server, a tower server or a cabinet server. The server 2 can be an independent server or a server cluster composed of multiple servers.

[0036] The server 2 is communicatively connected to one or more of the callers 1 and the database 3 respectively through a network (not shown in the figure) for data transmission and interaction. The network can be a wireless or wired network such as an enterprise intranet (Intranet), the Internet, Global System of Mobile communication (GSM), Wideband Code Division Multiple Access (WCDMA), 4G network, 5G network, Bluetooth, Wi-Fi, a call network, etc.

[0037] Wherein, each database 3 corresponds to a database access module 30. Through the database access module 30, read and write requests for the database 3 can be received and processed, and then the processing results can be returned.

[0038] Refer to Figure 2 shown in the figure, which is a schematic diagram of an optional hardware architecture of the server 2 of the present invention.

[0039] In this embodiment, the server 2 may include, but is not limited to, a memory 11, a processor 12, and a network interface 13 that are communicatively connected to each other through a system bus. It should be noted that Figure 2 only the server 2 with components 11 - 13 is shown, but it should be understood that it is not required to implement all the shown components, and more or fewer components can be alternatively implemented.

[0040] Wherein, the server 2 can be a computing device such as a rack-mounted server, a blade server, a tower server, or a cabinet server. The server 2 can be an independent server or a server cluster composed of multiple servers.

[0041] The memory 11 at least includes one type of readable storage medium, and the readable storage medium includes flash memory, hard disk, multimedia card, card-type memory (such as SD or DX memory, etc.), random access memory (RAM), static random access memory (SRAM), read-only memory (ROM), electrically erasable programmable read-only memory (EEPROM), programmable read-only memory (PROM), magnetic memory, magnetic disk, optical disk, etc. In some embodiments, the memory 11 may be an internal storage unit of the server 2, such as the hard disk or memory of the server 2. In other embodiments, the memory 11 may also be an external storage device of the server 2, such as a plug-in hard disk, SmartMedia Card (SMC), Secure Digital (SD) card, Flash Card, etc. equipped on the server 2. Of course, the memory 11 may also include both the internal storage unit of the server 2 and its external storage devices. In this embodiment, the memory 11 is generally used to store the operating system and various application software installed on the server 2, such as the program code of the data reading and writing batch processing system 200. In addition, the memory 11 may also be used to temporarily store various data that have been output or will be output.

[0042] In some embodiments, the processor 12 may be a central processing unit (CPU), a controller, a microcontroller, a microprocessor, or other data processing chips. The processor 12 is generally used to control the overall operation of the server 2. In this embodiment, the processor 12 is used to run the program code stored in the memory 11 or process data, such as running the data reading and writing batch processing system 200, etc.

[0043] The network interface 13 may include a wireless network interface or a wired network interface, and the network interface 13 is generally used to establish a communication connection between the server 2 and other electronic devices.

[0044] So far, the application environment architecture of the present invention and the hardware structures and functions of related devices have been introduced in detail. Next, various embodiments of the present invention will be proposed based on the above introduction.

[0045] First, the present invention proposes a data reading and writing batch processing system 200.

[0046] Refer to Figure 3 As shown, it is a program module diagram of the first embodiment of the data reading and writing batch processing system 200 of the present invention.

[0047] In this embodiment, the data reading and writing batch processing system 200 includes a series of computer program instructions stored in the memory 11. When the computer program instructions are executed by the processor 12, the data reading and writing batch processing operations of the embodiments of the present invention can be implemented. In some embodiments, based on the specific operations implemented by each part of the computer program instructions, the data reading and writing batch processing system 200 can be divided into one or more modules. For example, in Figure 3 , the data reading and writing batch processing system 200 can be divided into an allocation module 201, a saving module 202, a sending module 203, and an updating module 204. Among them:

[0048] The allocation module 201 is used to allocate a unique request ID for each received reading and writing request.

[0049] Specifically, the caller 1 uses the json format to send a batch data reading and writing request for a single database 3 or multiple databases 3 to the server 2. The reading and writing request includes request parameters such as a database identifier (db_id), a reading and writing sql number (sql_id, and sql_id cannot be the same in the same request), and an sql statement to be executed. The database identifier is used to identify the database access module 30. For example, A indicates that the database A is accessed through the database access module A. Each reading and writing sql number and the corresponding sql statement represent a reading and writing task. Each reading and writing request may include one or more database identifiers and reading and writing tasks. The allocation module 201 receives the reading and writing request and allocates a unique request ID, that is, request_id, for each reading and writing request.

[0050] For a batch data reading and writing request for a single database 3, for example, there are a student table (stu), a teacher table (teac), and a student score table (stu_score) in a database. Now, a batch data reading and writing request for this database is received, and it is necessary to query the data of "Xiaowang" in the student table (sql1: select * from stu where stu_name = 'Xiaowang'), query the data of "Teacher Zhang" in the teacher table (sql2: select * from teac where teac_name = 'Teacher Zhang'), and update the data of "Xiaoli" in the student score table (sql3: update stu_score set score = 90 where stu_name = 'Xiaoli'). The allocation module 201 can allocate a unique request ID, such as 178652986, for this reading and writing request.

[0051] For batch data read and write requests for multiple databases 3, for example, database A contains a student table (stu) and a teacher table (teac), and database B contains a student score table (stu_score). Now, a batch data read and write request for databases A and B is received, and it is necessary to query the data of "Zhang San" in the teacher table and the data of "Zhang San" in the student table of database A at the same time (sql1: select * from stu where stu_name = 'Zhang San'; sql2: select * from teac where teac_name = 'Zhang San'), query the data of "Li Si" in the student score table of database B and modify the data of "Xiao Ming" (sql3: select * from stu_score where stu_name = 'Li Si'; sql4: update stu_score set score = 90 where stu_name = 'Xiao Ming'). Similarly, the allocation module 201 can also assign a unique request ID, such as 178652987, to this read and write request.

[0052] The saving module 202 is used to write the read and write request into the hash memory.

[0053] Specifically, a storage structure for each read and write request is created in the hash memory. Each storage structure includes a keyword (key) and a value structure. The read and write request is written into the hash memory. Among them, the request ID, that is, request_id, is used as the key corresponding to the read and write request in the hash memory, and the calling party connection identifier tcp socket corresponding to the read and write request, as well as the read and write sql statements and read and write sql_id corresponding to each read and write task in the read and write request, are written into the value structure corresponding to the request ID as an array. In addition, two items, namely, the task return identifier (sql_ret) and the task return data (sql_data), are also included in the array of the value structure, which are used to write the processing result returned for the read and write request subsequently. Refer to Figure 4 As shown, it is a schematic diagram of the storage structure created in the hash memory.

[0054] For example, using the request IDs of the above two read / write requests as keys, save the information of these two read / write requests in the hash memory, and use the connection identifiers of the callers of these two read / write requests, the read / write SQL statements, and the read / write SQL IDs as arrays, and write them into the value structure of the hash memory corresponding to their respective request IDs. Taking the above read / write requests for databases A and B as an example, in the hash memory, use the request ID "178652987" of the read / write request as the key, and use the connection identifier of the caller, the read / write SQL statements (select * from stu where stu_name = 'Zhang San'; select * from teac where teac_name = 'Zhang San'; select * from stu_score where stu_name = 'Li Si'; update stu_score set score = 90 where stu_name = 'Xiao Ming') and the read / write SQL IDs (sql1, sql2, sql3, sql4) as arrays, and write them into the corresponding value structure.

[0055] The sending module 203 is configured to send each read / write task of the read / write request to the corresponding database 3 for parallel processing according to the database identifier in the read / write request.

[0056] Specifically, obtain the request parameters in the read / write request, and according to the database identifier, i.e., db_id, in the request parameters, send multiple data read / write tasks belonging to different databases 3 (or the same database 3) to the corresponding databases 3 simultaneously. The read / write task data sent to each database 3 includes the request ID (request_id), task ID (task_id, which is the same as the sql_id), the read / write SQL statement, etc. Each database 3 includes a corresponding database access module 30, which is configured to receive and process the read / write request of this database 3, and then return the processing result. The returned processing result includes the request ID (request_id), task ID (task_id, which is the same as the sql_id), task return identifier (sql_ret), task return data (sql_data), etc. Among them, the task return identifier is used to indicate whether the SQL statement is executed successfully. Returning a value of 0 indicates successful execution, and a non-zero value indicates failure. The task return data is used to indicate the return data corresponding to the SQL statement. When the SQL statement is a query task and is executed successfully, the specific query result data is returned. When the SQL statement is a non-query task or the query execution fails, a null value is returned.

[0057] For example, for the query task of the above database B to query the data of "Xiaoli" in the student score table (sql3: select * from stu_score where stu_name = 'Xiaoli'), the processing result returned has a request ID of 178652987, a task ID of 3, a task return identifier of 0, and the task return data is {"stu_name": "Xiaoli", "stu_score": "95"}; for the non-query task of modifying the data of "Xiaoming" in the student score table (sql4: update stu_score set score = 90 where stu_name = 'Xiaoming'), the processing result returned has a request ID of 178652987, a task ID of 4, a task return identifier of 0, and the task return data is null.

[0058] In this embodiment, all read / write tasks in multiple read / write requests can be sent to the corresponding database 3 simultaneously, and all subtasks are processed in parallel. The time consumption is the time consumption of the largest subtask among them.

[0059] The update module 204 is configured to receive the processing result returned by the database 3 and update it to the value structure corresponding to the read / write request.

[0060] Specifically, the update module 204 asynchronously receives the processing results of each read / write task in each read / write request returned by each database 3, obtains the corresponding value structure from the hash memory according to the request ID in each processing result, and then finds the corresponding read / write task in the value structure according to the sql_id (which is the same as the task ID), and updates the task return identifier and task return data of the read / write task.

[0061] For example, update the processing result returned by the above database B to the value structure corresponding to the request ID "178652987" in the hash memory.

[0062] The data read / write batch processing system provided in this embodiment can provide a unified multi-database read / write task interface. When receiving a batch data read / write request, it uses the unique request ID of each read / write request as the key in the hash memory to save the corresponding information of each read / write task of each read / write request, and sends the batch of read / write tasks to the corresponding database for parallel processing at the same time. When receiving the processing result returned for each read / write task, it is also written into the hash memory, so that the batch read / write tasks of multiple databases can be processed concurrently without data chaos, and the access time consumption is greatly reduced.

[0063] Refer to Figure 5As shown, it is a program module diagram of the second embodiment of the data reading and writing batch processing system 200 of the present invention. In this embodiment, the data reading and writing batch processing system 200 further includes a judgment module 205 and a return module 206 in addition to the distribution module 201, the storage module 202, the sending module 203, and the update module 204 in the first embodiment.

[0064] The judgment module 205 is used to judge whether all the processing results of the reading and writing tasks of the reading and writing request have been returned.

[0065] Specifically, for each reading and writing request, judge whether all the processing results of the reading and writing tasks in the corresponding value structure have been returned. If not all are returned, continue to wait for the corresponding database 3 to return the processing results until all the processing results of the reading and writing tasks of the reading and writing request are returned.

[0066] It should be noted that in this embodiment, when not all the processing results of the reading and writing tasks of the reading and writing request have been returned and need to continue waiting, other reading and writing requests can be processed simultaneously, that is, other reading and writing requests do not need to queue and wait, and it will not affect the processing time of other reading and writing requests.

[0067] The return module 206 is used to combine the processing results and return them to the calling party 1 when all the processing results of the reading and writing tasks of the reading and writing request have been returned.

[0068] Specifically, if all the processing results of the reading and writing tasks of the reading and writing request have been returned, obtain the connection identifier of the calling party in the corresponding value structure, combine the processing results of all the reading and writing tasks of the reading and writing request, and return the combined result to the calling party 1. The combined processing result returned to the calling party 1 includes data such as database ID (db_id), reading and writing sql_id, task return identifier (sql_ret), and task return data (sql_data).

[0069] For example, for the read / write request with the above request ID 178652987, determine whether all four read / write tasks in the corresponding value structure (sql1: select * from stu where stu_name = 'Zhang San'; sql2: select * from teac where teac_name = 'Zhang San'; sql3: select * from stu_score where stu_name = 'Li Si'; sql4: update stu_score set score = 90 where stu_name = 'Xiao Ming') have been fully returned, that is, whether the task return flags and task return data corresponding to these four task IDs have been updated. If not all have been returned, for example, only database B has returned but database A has not, then continue to wait for the processing result returned by database A (at this time, other read / write requests can be processed simultaneously). If all have been returned, then combine the processing results of all four tasks and return them to the caller 1.

[0070] It should be noted that each module of the data read / write batch processing system 200 and the database access module 30 of each database 3 can be configured with multiple processes for synchronous processing. In this embodiment, the read / write requests for a single database 3 or multiple databases 3 are batch processed, and multiple different read / write requests can be processed simultaneously, that is, multiple read requests and multiple write requests for a single or multiple databases 3 can be mixed and batch processed concurrently.

[0071] The data read / write batch processing system provided in this embodiment can provide a unified multi-database read / write task interface. For batch read / write tasks, they are simultaneously sent to the corresponding databases for parallel processing, with high concurrency and greatly reduced access time. The caller only needs to assemble the read / write task list of the databases to be accessed according to the unified interface, and then perform logical processing based on the processing result data returned by the unified interface, achieving the effect of abstract isolation between the business layer and the data access layer and improving production efficiency. Moreover, the read / write in this embodiment does not depend on specific tables, and the fields are abstractly obtained, supporting dynamic table expansion without modifying the code, improving production capacity. In addition, this embodiment supports distributed deployment and configuration of multi-process processing, greatly utilizing the multi-core hardware resources of the CPU and improving processing capacity.

[0072] In addition, the present invention also proposes a data read / write batch processing method.

[0073] Refer to Figure 6 shown in the figure, which is a flowchart of the first embodiment of the data read / write batch processing method of the present invention. In this embodiment, according to different requirements, Figure 6 the execution order of the steps in the flowchart shown can be changed, and some steps can be omitted.

[0074] The method includes the following steps:

[0075] Step S400, assign a unique request ID to each received read / write request.

[0076] Specifically, the caller 1 initiates a batch data read / write request for a single database 3 or multiple databases 3 in json format to the server 2. The read / write request contains request parameters such as database identifier (db_id), read / write sql number (sql_id, sql_id cannot be the same in the same request), and sql statements to be executed. The database identifier is used to identify the database access module 30. For example, A indicates accessing database A through database access module A. Each read / write sql number and the corresponding sql statement represent a read / write task. Each read / write request may include one or more database identifiers and read / write tasks. The server 2 receives the read / write request and assigns a unique request ID, namely request_id, to each read / write request.

[0077] For a batch data read / write request for a single database 3, for example, there are student table (stu), teacher table (teac), and student score table (stu_score) in a database. Now a batch data read / write request for this database is received, and it is necessary to query the data of 'Xiaowang' in the student table (sql1: select * from stu where stu_name = 'Xiaowang'), query the data of 'Teacher Zhang' in the teacher table (sql2: select * from teac where teac_name = 'Teacher Zhang'), and update the data of 'Xiaoli' in the student score table (sql3: update stu_score set score = 90 where stu_name = 'Xiaoli'). The server 2 can assign a unique request ID, such as 178652986, to this read / write request.

[0078] For batch data read and write requests for multiple databases 3, for example, database A contains a student table (stu) and a teacher table (teac), and database B contains a student score table (stu_score). Now, a batch data read and write request for databases A and B is received. It is necessary to query the data of 'Zhang Xiao' in the student table of database A and the data of 'Zhang San' in the teacher table at the same time (sql1: select * from stu where stu_name = 'Zhang Xiao'; sql2: select * from teac where teac_name = 'Zhang San'), query the data of 'Li Xiao' in the student score table of database B and modify the data of 'Xiao Ming' (sql3: select * from stu_score where stu_name = 'Li Xiao'; sql4: update stu_score set score = 90 where stu_name = 'Xiao Ming'). Similarly, the server 2 can also assign a unique request ID to this read and write request, such as 178652987.

[0079] Step S402, write the read and write request into the hash memory.

[0080] Specifically, create a storage structure for each read and write request in the hash memory. Each storage structure includes a keyword (key) and a value structure. Write the read and write request into the hash memory. Among them, the request ID, that is, request_id, serves as the key corresponding to the read and write request in the hash memory, and the calling party connection identifier tcp socket corresponding to the read and write request, as well as the read and write sql statements and read and write sql_id corresponding to each read and write task in the read and write request, are written as an array into the value structure corresponding to the request ID. In addition, the array in the value structure also contains two items, namely the task return identifier (sql_ret) and the task return data (sql_data), which are used to write the processing result returned for the read and write request later.

[0081] For example, using the request IDs of the above two read / write requests as keys, save the information of these two read / write requests in the hash memory, and use the connection identifiers of the callers of these two read / write requests, the read / write SQL statements, and the read / write SQL IDs as arrays, and write them into the value structures of the hash memory corresponding to their respective request IDs. Taking the above read / write requests for databases A and B as an example, in the hash memory, use the request ID "178652987" of the read / write request as the key, and use the connection identifier of the caller, the read / write SQL statements (select * from stu where stu_name = 'Zhang San'; select * from teac where teac_name = 'Zhang San'; select * from stu_score where stu_name = 'Li Si'; update stu_score set score = 90 where stu_name = 'Xiao Ming') and the read / write SQL IDs (sql1, sql2, sql3, sql4) as arrays, and write them into the corresponding value structures.

[0082] Step S404, send each read / write task of the read / write request to the corresponding database 3 for parallel processing according to the database identifier in the read / write request.

[0083] Specifically, obtain the request parameters in the read / write request, and according to the database identifier (i.e., db_id) in the request parameters, send multiple data read / write tasks belonging to different databases 3 (or the same database 3) to the corresponding databases 3 at the same time. The read / write task data sent to each database 3 includes the request ID (request_id), the task ID (task_id, which is the same as the sql_id), the read / write SQL statement, etc. Each database 3 includes a corresponding database access module 30 for receiving and processing the read / write requests of this database 3, and then returning the processing results. The returned processing results include the request ID (request_id), the task ID (task_id, which is the same as the sql_id), the task return identifier (sql_ret), the task return data (sql_data), etc. Among them, the task return identifier is used to indicate whether the SQL statement is executed successfully. Returning the value 0 indicates successful execution, and a non-zero value indicates failure. The task return data is used to indicate the return data corresponding to the SQL statement. When the SQL statement is a query task and is executed successfully, the specific query result data is returned. When the SQL statement is a non-query task or the query execution fails, a null value is returned.

[0084] For example, for the query task of querying the data of "Xiaoli" in the student score table in the above database B (sql3: select * from stu_score where stu_name = 'Xiaoli'), in the processing result returned, the request ID is 178652987, the task ID is 3, the task return identifier is 0, and the task return data is {"stu_name": "Xiaoli", "stu_score": "95"}; for the non-query task of modifying the data of "Xiaoming" in the student score table (sql4: update stu_score set score = 90 where stu_name = 'Xiaoming'), in the processing result returned, the request ID is 178652987, the task ID is 4, the task return identifier is 0, and the task return data is null.

[0085] In this embodiment, all read / write tasks in multiple read / write requests can be simultaneously sent to the corresponding database 3, and all subtasks are processed in parallel, and the time consumption is the time consumption of one of the largest subtasks.

[0086] Step S406, receive the processing result returned by database 3 and update it to the value structure corresponding to the read / write request.

[0087] Specifically, the server 2 asynchronously receives the processing results of each read / write task in each read / write request returned by each database 3, obtains the corresponding value structure from the hash memory according to the request ID in each processing result, and then finds the corresponding read / write task in the value structure according to the sql_id (consistent with the task ID), and updates the task return identifier and task return data of the read / write task.

[0088] For example, update the processing result returned by the above database B to the value structure corresponding to the request ID "178652987" in the hash memory.

[0089] The data read / write batch processing method provided in this embodiment can provide a unified multi-database read / write task interface. When receiving a batch data read / write request, use the unique request ID of each read / write request as the keyword in the hash memory to save the corresponding information of each read / write task of each read / write request, and simultaneously send the batch of read / write tasks to the corresponding database for parallel processing. When receiving the processing result returned for each read / write task, it is also written into the hash memory, so that the batch read / write tasks of multiple databases can be processed concurrently without data chaos, and the access time consumption is greatly reduced.

[0090] Such as Figure 7As shown in the figure, it is a schematic flowchart of the second embodiment of the data reading and writing batch processing method of the present invention. In this embodiment, steps S500 - S506 of the data reading and writing batch processing method are similar to steps S400 - S406 of the first embodiment, except that this method further includes steps S508 - S510.

[0091] The method includes the following steps:

[0092] Step S500, assign a unique request ID to each received read - write request.

[0093] Specifically, the caller 1 uses the json format to initiate a batch data read - write request for a single database 3 or multiple databases 3 to the server 2. The read - write request contains request parameters such as database identifier (db_id), read - write sql number (sql_id, sql_id cannot be the same in the same request), and the sql statement to be executed. Among them, the database identifier is used to identify the database access module 30. For example, A indicates accessing database A through database access module A. Each read - write sql number and the corresponding sql statement represent a read - write task. Each read - write request may include one or more database identifiers and read - write tasks. The server 2 receives the read - write request and assigns a unique request ID, that is, request_id, to each read - write request.

[0094] For the batch data read - write request for a single database 3, for example, there are student table (stu), teacher table (teac), and student score table (stu_score) in a database. Now a batch data read - write request for this database is received, and it is required to query the data of 'Xiaowang' in the student table (sql1: select * from stu where stu_name = 'Xiaowang'), query the data of 'Teacher Zhang' in the teacher table (sql2: select * from teac where teac_name = 'Teacher Zhang'), and update the data of 'Xiaoli' in the student score table (sql3: update stu_score set score = 90 where stu_name = 'Xiaoli'). The server 2 can assign a unique request ID, such as 178652986, to this read - write request.

[0095] For batch data read and write requests for multiple databases 3, for example, database A contains a student table (stu) and a teacher table (teac), and database B contains a student score table (stu_score). Now, a batch data read and write request for databases A and B is received. It is necessary to query the data of 'Zhang Xiao' in the student table of database A and the data of 'Zhang San' in the teacher table at the same time (sql1: select * from stu where stu_name = 'Zhang Xiao'; sql2: select * from teac where teac_name = 'Zhang San'), query the data of 'Li Xiao' in the student score table of database B and modify the data of 'Xiao Ming' (sql3: select * from stu_score where stu_name = 'Li Xiao'; sql4: update stu_score set score = 90 where stu_name = 'Xiao Ming'). Similarly, the server 2 can also assign a unique request ID to this read and write request, such as 178652987.

[0096] Step S502, write the read and write request into the hash memory.

[0097] Specifically, create a storage structure for each read and write request in the hash memory. Each storage structure includes a keyword (key) and a value structure. Write the read and write request into the hash memory. Among them, the request ID, that is, request_id, serves as the key corresponding to the read and write request in the hash memory, and the calling party connection identifier tcp socket corresponding to the read and write request, as well as the read and write sql statements and read and write sql_id corresponding to each read and write task in the read and write request, are written as an array into the value structure of the hash memory corresponding to the request ID. In addition, the array in the value structure also contains two items, namely the task return identifier (sql_ret) and the task return data (sql_data), which are used to write the processing results returned for the read and write request later.

[0098] For example, using the request IDs of the above two read / write requests as keys, save the information of these two read / write requests in the hash memory. And respectively use the connection identifiers of the callers of these two read / write requests, the read / write SQL statements, and the read / write SQL IDs as arrays, and write them into the value structures of the hash memory corresponding to their respective request IDs. Taking the above read / write requests for databases A and B as an example, in the hash memory, use the request ID "178652987" of this read / write request as the key, and use the connection identifier of the caller, the read / write SQL statements (select * from stu where stu_name = 'Zhang San'; select * from teac where teac_name = 'Zhang San'; select * from stu_score where stu_name = 'Li Si'; update stu_score set score = 90 where stu_name = 'Xiao Ming') and the read / write SQL IDs (sql1, sql2, sql3, sql4) as arrays, and write them into the corresponding value structure.

[0099] Step S504, send each read / write task of the read / write request to the corresponding database 3 for parallel processing according to the database identifier in the read / write request.

[0100] Specifically, obtain the request parameters in the read / write request. According to the database identifier, i.e., db_id, in the request parameters, send multiple data read / write tasks belonging to different databases 3 (or the same database 3) to the corresponding database 3 simultaneously. The read / write task data sent to each database 3 includes the request ID (request_id), task ID (task_id, which is the same as sql_id), read / write SQL statements, etc. Each database 3 includes a corresponding database access module 30 for receiving and processing the read / write requests of this database 3, and then returning the processing results. The returned processing results include the request ID (request_id), task ID (task_id, which is the same as sql_id), task return identifier (sql_ret), task return data (sql_data), etc. Among them, the task return identifier is used to indicate whether the SQL statement is executed successfully. Returning the value 0 indicates successful execution, and non-0 indicates failure. The task return data is used to indicate the return data corresponding to the SQL statement. When the SQL statement is a query task and is executed successfully, the specific query result data is returned. When the SQL statement is a non-query task or the query execution fails, a null value is returned.

[0101] For example, for the query task of querying the data of "Xiaoli" in the student score table in the above database B (sql3: select * from stu_score where stu_name = 'Xiaoli'), in the processing result returned, the request ID is 178652987, the task ID is 3, the task return identifier is 0, and the task return data is {"stu_name": "Xiaoli", "stu_score": "95"}; for the non-query task of modifying the data of "Xiaoming" in the student score table (sql4: update stu_score set score = 90 where stu_name = 'Xiaoming'), in the processing result returned, the request ID is 178652987, the task ID is 4, the task return identifier is 0, and the task return data is null.

[0102] In this embodiment, all read / write tasks in multiple read / write requests can be sent to the corresponding database 3 simultaneously, and all subtasks are processed in parallel. The time consumption is the time consumption of the largest subtask among them.

[0103] Step S506, receive the processing result returned by database 3 and update it to the value structure corresponding to the read / write request.

[0104] Specifically, the server 2 asynchronously receives the processing results of each read / write task in each read / write request returned by each database 3, obtains the corresponding value structure from the hash memory according to the request ID in each processing result, and then finds the corresponding read / write task in the value structure according to the sql_id (which is the same as the task ID), and updates the task return identifier and task return data of the read / write task.

[0105] For example, update the processing result returned by the above database B to the value structure corresponding to the request ID "178652987" in the hash memory.

[0106] Step S508, determine whether the processing results of all read / write tasks of the read / write request have been returned.

[0107] Specifically, for each read / write request, determine whether the processing results of all read / write tasks in the corresponding value structure have been returned. If not all have been returned, continue to wait for the processing result returned by the corresponding database 3 until the processing results of all read / write tasks of the read / write request have been returned.

[0108] It should be noted that in this embodiment, when not all the processing results of the read / write tasks of the read / write request have been returned and waiting is required, other read / write requests can be processed simultaneously, that is, other read / write requests do not need to queue up and wait, and it will not affect the processing time of other read / write requests.

[0109] Step S510, when all the read / write tasks of the read / write request have returned processing results, combine the processing results and return them to the caller 1.

[0110] Specifically, if all the read / write tasks of the read / write request have returned processing results, obtain the caller connection identifier in the corresponding value structure, combine the processing results of all the read / write tasks of the read / write request, and return the combined result to the caller 1. The combined processing result returned to the caller 1 includes data such as database ID (db_id), read / write sql_id, task return identifier (sql_ret), task return data (sql_data), etc.

[0111] For example, for the above read / write request with request ID 178652987, determine whether all four read / write tasks in the corresponding value structure (sql1: select * from stu where stu_name = 'Zhang San'; sql2: select * from teac where teac_name = 'Zhang San'; sql3: select * from stu_score where stu_name = 'Li Si'; sql4: update stu_score set score = 90 where stu_name = 'Xiao Ming') have returned, that is, whether the task return identifiers and task return data corresponding to these four task IDs have been updated. If not all have returned, for example, only database B has returned but database A has not, then continue to wait for the processing result returned by database A (at this time, other read / write requests can be processed simultaneously). If all have returned, combine the processing results of all four tasks and return them to the caller 1.

[0112] It should be noted that each step of the data read / write batch processing method and the database access module 30 of each database 3 can be configured for multiple processes to synchronously process. In this embodiment, the batch processing is for read / write requests for a single database 3 or multiple databases 3, and multiple different read / write requests can be processed simultaneously, that is, multiple read requests and multiple write requests for a single or multiple databases 3 can be mixed and batch processed concurrently.

[0113] The data reading and writing batch processing method provided in this embodiment can provide a unified multi-database reading and writing task interface. For batch reading and writing tasks, they are simultaneously sent to the corresponding databases for parallel processing, with high concurrency and greatly reduced access time. The caller only needs to assemble the list of reading and writing tasks for the databases to be accessed according to the unified interface, and then perform logical processing based on the processed result data returned by the unified interface, achieving the effect of abstract isolation between the business layer and the data access layer and improving production efficiency. Moreover, the reading and writing in this embodiment do not depend on specific tables, and the fields are abstractly obtained, supporting dynamic table expansion without modifying the code, improving production capacity. In addition, this embodiment supports distributed deployment and configuration of multi-process processing, greatly utilizing the multi-core hardware resources of the CPU and improving processing capacity.

[0114] The present invention also provides another implementation manner, that is, to provide a computer-readable storage medium storing a data reading and writing batch processing program, which can be executed by at least one processor to cause the at least one processor to execute the steps of the data reading and writing batch processing method as described above.

[0115] The serial numbers of the above embodiments of the present invention are only for description and do not represent the superiority or inferiority of the embodiments.

[0116] Through the description of the above embodiments, those skilled in the art can clearly understand that the above embodiment methods can be implemented by means of software plus a necessary general hardware platform. Of course, they can also be implemented by hardware, but in many cases the former is a better implementation manner. Based on such an understanding, the technical solution of the present invention, in essence, or the part that contributes to the prior art can be embodied in the form of a software product. This computer software product is stored in a storage medium (such as ROM / RAM, magnetic disk, optical disc) and includes several instructions for causing a terminal device (which can be a mobile phone, computer, server, air conditioner, or network device, etc.) to execute the methods described in the various embodiments of the present invention.

[0117] The above are only the preferred embodiments of the present invention, and do not limit the patent scope of the present invention accordingly. Any equivalent structural or equivalent process transformation made by using the content of the specification and drawings of the present invention, or directly or indirectly applied in other related technical fields, shall be equally included in the patent protection scope of the present invention.

Claims

1. A method for batch processing of data reading and writing, characterized in that, the method comprises the steps of: When receiving one or more batch data reading and writing requests for a single database or multiple databases from a calling party, assign a unique request ID to each reading and writing request respectively. The request parameters in the reading and writing requests include database identifiers and reading and writing tasks, and each reading and writing task includes a reading and writing SQL number and a reading and writing SQL statement to be executed; Create a storage structure for each of the reading and writing requests in the hash memory, including a key and a value structure. Write the request ID into the key, and write the connection identifier of the calling party corresponding to each reading and writing request, as well as the reading and writing SQL statements and reading and writing SQL numbers corresponding to each reading and writing task in the reading and writing request as an array into the value structure; Send each reading and writing task of the reading and writing requests to the corresponding database for parallel processing according to the database identifier in each reading and writing request; and Asynchronously receive the processing results returned by each database, obtain the corresponding value structure from the hash memory according to the request ID in each processing result, and then find the corresponding reading and writing task in the value structure according to the reading and writing SQL number, and update the task return identifier and task return data of the reading and writing task.

2. The method for batch processing of data reading and writing according to claim 1, characterized in that, the method further comprises the steps of: Judge whether all the processing results of the reading and writing tasks of the reading and writing requests have been returned according to the value structure; When all the processing results of the reading and writing tasks of the reading and writing requests have been returned, combine the processing results of all the reading and writing tasks of the reading and writing requests and return them to the calling party.

3. The method for batch processing of data reading and writing according to claim 1 or 2, characterized in that, In the step of sending each reading and writing task of the reading and writing requests to the corresponding database for parallel processing according to the database identifier in each reading and writing request, the reading and writing task data sent to each database includes the request ID, the reading and writing SQL number, and the reading and writing SQL statement.

4. The method for batch processing of data reading and writing according to claim 1 or 2, characterized in that, The processing result includes the request ID, the reading and writing SQL number, the task return identifier, and the task return data.

5. The method for batch processing of data reading and writing according to claim 1, characterized in that, The task return identifier is used to indicate whether the reading and writing SQL statement is executed successfully. Returning the value 0 indicates successful execution, and non-0 indicates failure.

6. The method for batch processing of data reading and writing according to claim 1, characterized in that, The task return data is used to indicate the return data corresponding to the reading and writing SQL statement. When the reading and writing SQL statement is a query task and is executed successfully, the specific query result data is returned; when the reading and writing SQL statement is a non-query task or the query task fails to be executed, a null value is returned.

7. The method for batch processing of data reading and writing according to claim 2, characterized in that, the method further comprises the steps of: When the read / write tasks of the read / write request have not all returned processing results and need to continue waiting, other read / write requests are processed continuously at the same time.

8. A server, characterized in that the server includes a memory and a processor, and a data read / write batch processing system that can run on the processor is stored on the memory. When the data read / write batch processing system is executed by the processor, the steps of the data read / write batch processing method according to any one of claims 1-7 are implemented.

9. A computer-readable storage medium, characterized in that the computer-readable storage medium stores a data read / write batch processing system, and the data read / write batch processing system can be executed by at least one processor, so that the at least one processor executes the steps of the data read / write batch processing method according to any one of claims 1-7.

Images