Database-based data processing methods, devices, and equipment

Abstract

This application provides a database-based data processing method, apparatus, and device, applicable to the field of computer technology. The method is applied to a business processing device and includes: receiving a batch processing request, the batch processing request including N data items to be stored and a maximum latency, where N is an integer greater than or equal to 1; determining the remaining resources of the business processing device, and determining a processing method corresponding to the batch processing request based on the remaining resources and the maximum latency, the processing method including: synchronous processing and asynchronous processing; obtaining N sequence numbers from the business processing device and / or a preset cache device; and storing the N data items to be stored into a corresponding database according to the processing method and the N sequence numbers. This method can improve data processing efficiency.

Description

Technical Field

[0001] This application relates to the field of computer technology, and in particular to a database-based data processing method, apparatus, and device. Background Technology

[0002] Business processing equipment can perform data processing on business data based on a database. For example, business data can be business data generated from batch processing, and data processing can include business data storage or business data updates.

[0003] Currently, when business processing equipment stores business data into a database, it can generate a corresponding number of serial numbers based on the volume of business data, and store the business data and serial numbers in the database using a preset processing method (synchronous processing or asynchronous processing). However, the serial number generation speed in the above process is slow, resulting in low data processing efficiency. Summary of the Invention

[0004] This application provides a database-based data processing method, apparatus, and device, which can improve data processing efficiency.

[0005] Firstly, this application provides a database-based data processing method applied to a business processing device, the method comprising:

[0006] Receive a batch processing request, wherein the batch processing request includes N data to be stored and a maximum delay, wherein N is an integer greater than or equal to 1;

[0007] The remaining resources of the business processing device are determined, and based on the remaining resources and the maximum latency, the processing method corresponding to the batch processing request is determined, including: synchronous processing method and asynchronous processing method;

[0008] Obtain N serial numbers from the business processing device and / or the preset cache device;

[0009] Based on the processing method and the N serial numbers, the N data to be stored are stored in the corresponding database.

[0010] In one possible implementation, the remaining resources of the service processing device are determined, and based on the remaining resources and the maximum latency, the processing method corresponding to the batch processing request is determined, including:

[0011] If the remaining resource quantity is 0, then the waiting time is determined, and the processing method corresponding to the batch processing request is determined based on the waiting time and the maximum delay;

[0012] If the remaining resource quantity is greater than 0, then based on the remaining resource quantity and the quantity N of data to be stored, the first duration required for synchronous processing and the second duration required for asynchronous processing are determined, and based on the first duration, the second duration, and the maximum latency, the processing method corresponding to the batch processing request is determined.

[0013] In one possible implementation, the processing method corresponding to the batch processing request is determined based on the waiting time and the maximum delay, including:

[0014] If the waiting time is greater than the maximum delay, then the processing method corresponding to the batch processing request is determined to be the asynchronous processing method;

[0015] If the waiting time is less than the maximum delay, the time difference between the maximum delay and the waiting time is obtained. If the time difference is less than or equal to a preset time difference, the processing method corresponding to the batch processing request is determined to be the asynchronous processing method. If the time difference is greater than the preset time difference, the processing method corresponding to the batch processing request is determined to be the synchronous processing method.

[0016] In one possible implementation, the processing method corresponding to the batch processing request is determined based on the first duration, the second duration, and the maximum delay, including:

[0017] If the first duration is less than the maximum delay, then the processing method corresponding to the batch processing request is determined to be the synchronous processing method;

[0018] If the first duration is greater than or equal to the maximum duration, then the processing method corresponding to the batch processing request is determined to be the asynchronous processing method.

[0019] In one possible implementation, N serial numbers are obtained from the business processing device and / or the preset cache device, including:

[0020] Determine the first number M of serial numbers existing in the business processing device, where M is an integer;

[0021] If M is greater than or equal to N, then the N serial numbers are obtained from the service processing device;

[0022] If M is less than N and M is 0, then the N serial numbers are obtained from the preset cache device;

[0023] If M is less than N and M is not 0, then M serial numbers are obtained in the service processing device and NM serial numbers are obtained in the preset cache device.

[0024] In one possible implementation, obtaining the N sequence numbers from the preset cache device includes:

[0025] Obtain the device status of the preset cache device, wherein the device status is either locked or unlocked.

[0026] When the device state of the preset cache device is unlocked, a serial number acquisition request is sent to the preset cache device, and the device state of the preset cache device is controlled to switch to the locked state. The serial number acquisition request includes N.

[0027] Receive N serial numbers sent by the preset cache device, and control the device state of the preset cache device to switch to the unlocked state.

[0028] In one possible implementation, the N data to be stored are stored in the corresponding database according to the processing method and the N sequence numbers, including:

[0029] The storage component corresponding to the processing method is determined, and the storage component is implemented through Java code;

[0030] Based on the N sequence numbers and the N data to be stored, generate N data pairs;

[0031] The storage component is invoked, and the N data pairs are stored in the corresponding database through the storage component.

[0032] In one possible implementation, N data pairs are generated based on the N sequence numbers and the N data to be stored, including:

[0033] Among the N sequence numbers, determine the sequence number corresponding to each piece of data to be stored;

[0034] According to the preset format, each piece of data to be stored and its corresponding sequence number are combined to obtain the N data pairs.

[0035] Secondly, this application provides a data processing apparatus applied to a business processing device. The data processing apparatus includes: a receiving module, a determining module, an acquiring module, and a storage module, wherein...

[0036] The receiving module is used to receive a batch processing request, wherein the batch processing request includes N data to be stored and a maximum delay, wherein N is an integer greater than or equal to 1;

[0037] The determining module is used to determine the remaining resources of the service processing device, and based on the remaining resources and the maximum latency, determine the processing method corresponding to the batch processing request, wherein the processing method includes: synchronous processing method and asynchronous processing method;

[0038] The acquisition module is used to acquire N serial numbers from the business processing device and / or the preset cache device;

[0039] The storage module is used to store the N data to be stored into the corresponding database according to the processing method and the N serial numbers.

[0040] In one possible implementation, the determining module is specifically used for,

[0041] If the remaining resource quantity is 0, then the waiting time is determined, and the processing method corresponding to the batch processing request is determined based on the waiting time and the maximum delay;

[0042] If the remaining resource quantity is greater than 0, then based on the remaining resource quantity and the quantity N of data to be stored, the first duration required for synchronous processing and the second duration required for asynchronous processing are determined, and based on the first duration, the second duration, and the maximum latency, the processing method corresponding to the batch processing request is determined.

[0043] In one possible implementation, the determining module is specifically used for,

[0044] If the waiting time is greater than the maximum delay, then the processing method corresponding to the batch processing request is determined to be the asynchronous processing method;

[0045] If the waiting time is less than the maximum delay, the time difference between the maximum delay and the waiting time is obtained. If the time difference is less than or equal to a preset time difference, the processing method corresponding to the batch processing request is determined to be the asynchronous processing method. If the time difference is greater than the preset time difference, the processing method corresponding to the batch processing request is determined to be the synchronous processing method.

[0046] In one possible implementation, the determining module is specifically used for,

[0047] If the first duration is less than the maximum delay, then the processing method corresponding to the batch processing request is determined to be the synchronous processing method;

[0048] If the first duration is greater than or equal to the maximum duration, then the processing method corresponding to the batch processing request is determined to be the asynchronous processing method.

[0049] In one possible implementation, the acquisition module is specifically used for,

[0050] Determine the first number M of serial numbers existing in the business processing device, where M is an integer;

[0051] If M is greater than or equal to N, then the N serial numbers are obtained from the service processing device;

[0052] If M is less than N and M is 0, then the N serial numbers are obtained from the preset cache device;

[0053] If M is less than N and M is not 0, then M serial numbers are obtained in the service processing device and NM serial numbers are obtained in the preset cache device.

[0054] In one possible implementation, the acquisition module is specifically used for,

[0055] Obtain the device status of the preset cache device, wherein the device status is either locked or unlocked.

[0056] When the device state of the preset cache device is unlocked, a serial number acquisition request is sent to the preset cache device, and the device state of the preset cache device is controlled to switch to the locked state. The serial number acquisition request includes N.

[0057] Receive N serial numbers sent by the preset cache device, and control the device state of the preset cache device to switch to the unlocked state.

[0058] In one possible implementation, the storage module is specifically used for,

[0059] The storage component corresponding to the processing method is determined, and the storage component is implemented through Java code;

[0060] Based on the N sequence numbers and the N data to be stored, generate N data pairs;

[0061] The storage component is invoked, and the N data pairs are stored in the corresponding database through the storage component.

[0062] In one possible implementation, the acquisition module is specifically used for,

[0063] Among the N sequence numbers, determine the sequence number corresponding to each piece of data to be stored;

[0064] According to the preset format, each piece of data to be stored and its corresponding sequence number are combined to obtain the N data pairs.

[0065] Thirdly, this application provides a data processing device, including: a processor, and a memory communicatively connected to the processor;

[0066] The memory stores computer-executed instructions;

[0067] The processor executes computer execution instructions stored in the memory to implement the method described in any of the first aspects.

[0068] Fourthly, this application provides a computer-readable storage medium storing computer-executable instructions, which, when executed by a processor, are used to implement the method as described in any of the first aspects.

[0069] Fifthly, this application provides a computer program product, including a computer program that, when executed by a computer, implements the method as described in any one of the first aspects.

[0070] The database-based data processing method, apparatus, and device provided in this application embodiment can receive batch processing requests, which include N data items to be stored and a maximum delay. The remaining resources of the processing device can be determined, and based on the remaining resources and the maximum delay, a processing method corresponding to the batch processing request can be determined. The processing methods include synchronous processing and asynchronous processing. N sequence numbers can be obtained from the processing device and / or a preset cache device. The N data items to be stored can be stored in the corresponding database according to the processing method and the N sequence numbers. Through the above method, the time required to store the data to be stored can be shortened, improving data processing efficiency. Attached Figure Description

[0071] To more clearly illustrate the technical solutions in the embodiments of the present invention or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0072] Figure 1 A schematic diagram illustrating the application scenarios provided in the embodiments of this application;

[0073] Figure 2 A schematic flowchart illustrating a database-based data processing method provided in an embodiment of this application;

[0074] Figure 3 This is a schematic diagram illustrating a serial number acquisition method provided in an embodiment of this application.

[0075] Figure 4 A schematic flowchart illustrating another database-based data processing method provided in this application embodiment;

[0076] Figure 5 This is a schematic diagram of the structure of a data processing device provided in an embodiment of this application;

[0077] Figure 6 This is a schematic diagram of the hardware structure of a data processing device provided in an embodiment of this application.

[0078] The accompanying drawings illustrate specific embodiments of this application, which will be described in more detail below. These drawings and descriptions are not intended to limit the scope of the concept in any way, but rather to illustrate the concept of this application to those skilled in the art through reference to particular embodiments. Detailed Implementation

[0079] Exemplary embodiments will now be described in detail, examples of which are illustrated in the accompanying drawings. When the following description relates to the drawings, unless otherwise indicated, the same numbers in different drawings denote the same or similar elements. The embodiments described in the following exemplary embodiments do not represent all embodiments consistent with this application. Rather, they are merely examples of apparatuses and methods consistent with some aspects of this application as detailed in the appended claims.

[0080] It should be noted that, in this document, the terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such a process, method, article, or apparatus. Unless otherwise specified, an element defined by the phrase "comprising one..." does not exclude the presence of other identical elements in the process, method, article, or apparatus that includes that element.

[0081] The collection, storage, use, processing, transmission, provision, and disclosure of financial data or user data involved in the technical solution of this application all comply with the provisions of relevant laws and regulations and do not violate public order and good morals.

[0082] To facilitate understanding, the technical terms involved in the embodiments of this application will be explained first.

[0083] Localization transformation: refers to replacing servers, operating systems, middleware, databases and other products with domestic products through distributed architecture transformation, getting rid of dependence on existing single commercial products, and forming a new pattern of CCB Cloud with domestic technology stack as the main body and multiple technology stacks coexisting.

[0084] Under the requirements of the localization policy, China Construction Bank (CCB) needs to integrate its financial capabilities into different scenarios and industry chains. Simultaneously, the rapid development of CCB's retail business demands that its core system can quickly respond to rapid iteration needs. Furthermore, CCB needs to return to building a dual-speed iteration capability for its core system, namely, designing its technical architecture guided by stable services and agile business operations. This involves achieving a separation of static and dynamic elements through stable and composable agile services, and ensuring that the core system can quickly orchestrate and reorganize services according to business requirements for rapid delivery.

[0085] During the localization process, core business systems place higher demands on the overall stability and high performance of domestically produced servers, operating systems, middleware, and databases. Regulatory requirements during this process have proposed different pathways for the transformation of core systems in the context of IT innovation, including production environments, disaster recovery environments, and testing environments.

[0086] In the current system, the database primarily uses Oracle's relational database management system; the middleware primarily uses Java application server software (WebLogic); the operating system primarily uses Advanced Interactive Executive (AIX); virtualization primarily uses VMware; and the servers primarily use Itanium. Oracle-supported stored procedures and Oracle feature functions are extensively used at the database level.

[0087] The domestically developed and domestically produced database solution aims to provide a comprehensive solution encompassing management, integration, development, and architecture, centered around Jinke's self-developed and open-source databases. It addresses all aspects of database issues related to domestically developed and domestically produced databases, including selection, development, testing, migration, architecture, and operation and maintenance.

[0088] Information technology innovation refers to the industry of information technology application innovation. It is the foundation of data security and network security, and an important component of new infrastructure.

[0089] TiDB (TiDB): This refers to an open-source distributed relational database designed and developed independently by the database vendor (PingCAP). It is a converged distributed database that supports both online transaction processing and in-house analytical processing, suitable for various application scenarios requiring high availability, strong consistency, and large data volumes.

[0090] To facilitate understanding, let's first combine Figure 1 The application scenarios involved in the embodiments of this application will be described.

[0091] Figure 1 This is a schematic diagram illustrating an application scenario provided in an embodiment of this application. Please refer to [link / reference]. Figure 1Business processing devices can store business data in a database. For example, business processing devices can be electronic devices such as servers, and business data can be business data generated from batch processing.

[0092] It should be noted that in the above scenarios, the number of business processing devices and databases can be one or more, and this application embodiment does not limit this.

[0093] In related technologies, when a business processing device stores business data, it can generate a corresponding number of serial numbers based on the volume of business data, and store the business data and serial numbers in a database according to a preset processing method. For example, the database can be an Oracle database. The preset processing method can be a pre-set synchronous processing method or an asynchronous processing method. However, the serial number generation speed in the above method is slow, resulting in low data processing efficiency.

[0094] In view of this, embodiments of this application provide a database-based data processing method to solve the technical problem of low data processing efficiency.

[0095] The database-based data processing method provided in this application can directly obtain the serial number and determine the processing method based on the data to be stored, so as to store the data to be stored in the database through the determined processing method. For example, the database can be an open-source distributed relational database. This method can shorten the time required to store the data and improve data processing efficiency.

[0096] The technical solution of this application and how the technical solution of this application solves the above-mentioned technical problems are described in detail below with specific embodiments. These specific embodiments can be combined with each other, and the same or similar concepts or processes may not be described again in some embodiments. The embodiments of this application will now be described with reference to the accompanying drawings.

[0097] Figure 2 This is a flowchart illustrating a database-based data processing method provided in an embodiment of this application. The execution entity of this method can be a business processing device or a data processing unit installed within the business processing device. The data processing unit can be implemented through software or a combination of software and hardware. Please refer to... Figure 2 The method may include:

[0098] S201. Receive a batch processing request. The batch processing request includes N data to be stored and the maximum delay, where N is an integer greater than or equal to 1.

[0099] This embodiment can be applied to business processing equipment. Business processing equipment can be an electronic device such as a server or computer. Business processing equipment can be used to process batch processing requests.

[0100] Batch processing requests can be used to request business processing devices to store data to be stored.

[0101] The maximum latency can be defined as the optimal time from when the business processing device receives a batch processing request to when it completes processing the batch processing request. In other words, after receiving a batch processing request, the business processing device can complete the processing of the batch processing request within the time range required by the maximum latency.

[0102] It should be noted that if the business processing device fails to complete the processing of batch processing requests within the maximum delay, it can continue to process batch processing requests.

[0103] Optionally, in other embodiments, the service processing device may also receive online service processing requests to process the corresponding online service data.

[0104] S202. Determine the remaining resources of the business processing equipment, and based on the remaining resources and the maximum latency, determine the processing method corresponding to the batch processing request.

[0105] The remaining resource quantity can be the amount of resources currently available on the business processing equipment. For example, the remaining resource quantity can be the current availability of the central processing unit (CPU) of the business processing equipment.

[0106] The processing methods include synchronous processing and asynchronous processing.

[0107] In this embodiment, if the business processing device processes batch processing requests synchronously, it can store the N data items to be stored in the batch processing request sequentially; if the business processing device processes batch processing requests asynchronously, it can store the N data items to be stored in the batch processing request simultaneously.

[0108] It should be understood that processing batch requests synchronously takes longer, while processing them asynchronously takes shorter.

[0109] In this embodiment, if the remaining resource quantity is 0, the processing method corresponding to the batch processing request can be determined based on the waiting time and the maximum latency, so as to complete the processing of the batch processing request within the time range required by the maximum latency as much as possible. The waiting time can be the duration for which the batch processing request waits to be processed.

[0110] Optionally, if the waiting time is greater than the maximum delay, the processing method for the batch processing request can be determined to be asynchronous, so as to complete the processing of the batch processing request as soon as possible; if the waiting time is less than the maximum delay, the processing method for the batch processing request can be determined according to the actual situation, so as to complete the processing of the batch processing request within the time range required by the maximum delay.

[0111] In this embodiment, if the remaining resource quantity is greater than 0, the time required to process the batch processing request using the current resources and according to different processing methods can be determined; and the processing method corresponding to the batch processing request can be determined according to the required time, so as to complete the processing of the batch processing request within the time range of the maximum latency requirement as much as possible.

[0112] S203. Obtain N serial numbers from the business processing device and / or the preset cache device.

[0113] The default cache device can be a default cache server. For example, the default cache device can be a cache for a remote dictionary server (Redis).

[0114] In this embodiment, serial numbers can be pre-stored in the service processing device and the preset cache device. In specific implementation, N serial numbers can be obtained from the service processing device and / or the preset cache device. That is, one serial number is obtained for each piece of data to be stored.

[0115] In this embodiment, when obtaining the serial number, the first number M of serial numbers existing in the service processing device can be determined, where M is an integer; if M is greater than or equal to N, then N serial numbers are obtained from the service processing device; if M is less than N and M is 0, then N serial numbers are obtained from the preset cache device; if M is less than N and M is not 0, then M serial numbers are obtained from the service processing device and NM serial numbers are obtained from the preset cache device.

[0116] In other words, the serial number for each piece of data to be stored can be obtained from the business processing device first, so as to obtain the serial number as quickly as possible. If there are no remaining serial numbers in the business processing device, or if there are not enough remaining serial numbers, the serial number can also be obtained from the preset cache device.

[0117] For example, assuming N is 10 and M is 13, it can be determined that the number of remaining serial numbers in the business processing device is greater than the amount of data to be stored, so 10 serial numbers can be obtained from the business processing device. Assuming N is 10 and M is 0, it can be determined that there are no remaining serial numbers in the business processing device, so 10 serial numbers can be obtained from the preset cache device. Assuming N is 10 and M is 5, it can be determined that the number of remaining serial numbers in the business processing device is insufficient, so 5 serial numbers can be obtained from the business processing device and 5 serial numbers can be obtained from the preset cache device.

[0118] In this embodiment, when N serial numbers are obtained from the preset cache device, the device status of the preset cache device can be obtained, which is either locked or unlocked. When the device status of the preset cache device is unlocked, a serial number acquisition request is sent to the preset cache device, and the device status of the preset cache device is controlled to switch to the locked state. The serial number acquisition request includes N. The N serial numbers sent by the preset cache device are received, and the device status of the preset cache device is controlled to switch to the unlocked state.

[0119] Specifically, in this embodiment, a distributed lock can be used to control a preset cache device to ensure the uniqueness of the serial number obtained by each business processing device from the preset cache device. For example, the distributed lock can be a Redis lock. In the specific implementation process, the serial number can be obtained from the preset cache device when it is in an unlocked state. During the process of obtaining the serial number, the preset cache device can be controlled to be in a locked state to prevent other business processing devices from obtaining the serial number from the preset cache device at the same time.

[0120] Below, in conjunction with Figure 3 The process of obtaining the serial number from the preset cache device is explained.

[0121] Figure 3 This is a schematic diagram illustrating a serial number acquisition method provided in an embodiment of this application. Please refer to [link / reference]. Figure 3 When the preset cache device is in an unlocked state, the service processing device can send a serial number retrieval request to the preset cache device and simultaneously control the preset cache device to be in a locked state. After obtaining the serial number from the preset cache device, the device can control the preset cache device to be in an unlocked state.

[0122] It should be noted that when the preset cache device is in a locked state, other business processing devices cannot obtain the serial number from the preset cache device, thus ensuring the uniqueness of the serial number obtained by the business processing device.

[0123] S204. Based on the processing method and N serial numbers, store the N data to be stored in the corresponding database.

[0124] In this embodiment, when storing N data to be stored into the corresponding database, the storage component corresponding to the processing method is determined. The storage component is implemented using object-oriented programmable language (Java) code. Based on the N sequence numbers and the N data to be stored, N data pairs are generated. The storage component is called, and the N data pairs are stored into the corresponding database through the storage component.

[0125] Specifically, if the batch processing request is processed synchronously, the storage component corresponding to synchronous processing can be called to store N data pairs in the database. If the batch processing request is processed asynchronously, the storage component corresponding to asynchronous processing can be called to store N data pairs in the database.

[0126] In this embodiment, when generating N data pairs based on N sequence numbers and N data to be stored, the sequence number corresponding to each data to be stored can be determined from the N sequence numbers; and each data to be stored and its corresponding sequence number can be combined according to a preset format to obtain N data pairs.

[0127] Specifically, a sequence number can be assigned to each piece of data to be stored, and each piece of data to be stored and its corresponding sequence number can be combined according to a preset format to obtain the corresponding data pair.

[0128] The preset format can be set according to actual needs, and this application embodiment does not limit it. For example, the preset format can be: for any data to be stored, the serial number can be recorded on the left side of the data to be stored.

[0129] It should be noted that the method for determining a sequence number for each piece of data to be stored in this application is not limited to this method, as long as the sequence number corresponding to each piece of data to be stored is different. For example, the sequence number can be determined for each piece of data to be stored sequentially.

[0130] The database-based data processing method provided in this embodiment allows a business processing device to receive batch processing requests, each including N data items to be stored and a maximum latency. The remaining resources of the business processing device can be determined, and based on these resources and the maximum latency, a processing method corresponding to the batch processing request can be determined. The processing methods include synchronous and asynchronous processing. N sequence numbers can be obtained from the business processing device and / or a preset cache device. Based on the processing method and the N sequence numbers, the N data items to be stored can be stored in the corresponding database. This method shortens the time required to store the data and improves data processing efficiency.

[0131] exist Figure 2 Based on the embodiments, the following, in conjunction with Figure 4The data processing method based on a database provided in the embodiments of this application will be further described.

[0132] Figure 4 This is a flowchart illustrating another database-based data processing method provided in an embodiment of this application. The executing entity of this method can be a business processing device or a data processing unit installed within the business processing device. The data processing unit can be implemented through software or a combination of software and hardware. Please refer to... Figure 4 The method may include:

[0133] S401. Receive a batch processing request. The batch processing request includes N data to be stored and the maximum delay, where N is an integer greater than or equal to 1.

[0134] It should be noted that the specific implementation of S401 can be found in S201, and will not be repeated here.

[0135] S402. Determine whether the remaining resources of the business processing equipment are greater than 0.

[0136] If so, execute S403;

[0137] If not, proceed with S405.

[0138] In this embodiment, the remaining resource amount can be the CPU availability of the service processing device. The service processing device can obtain its own remaining resource amount and determine whether the remaining resource amount is greater than 0.

[0139] S403. Based on the remaining resources and the amount of data to be stored N, determine the first duration required for synchronous processing and the second duration required for asynchronous processing.

[0140] In this embodiment, the service processing device can determine the first duration and the second duration based on the remaining resource quantity, the synchronous preset duration, and the asynchronous preset duration.

[0141] The synchronous preset duration can be the time required for the business processing device to process a piece of data to be stored synchronously using its remaining resources. The asynchronous preset duration can be the time required for the business processing device to process a piece of data to be stored asynchronously using its remaining resources.

[0142] Specifically, the service processing device can determine a first duration based on the quantity N and the synchronous preset duration, where the first duration can be the product of the synchronous preset duration and the quantity N. The service processing device can also determine a second duration based on an asynchronous preset duration, where the second duration can be equal to the asynchronous preset duration.

[0143] S404. Determine the processing method corresponding to the batch processing request based on the first duration, the second duration, and the maximum delay.

[0144] In this embodiment, if the first duration is less than the maximum delay, the processing method corresponding to the batch processing request is determined to be synchronous processing; if the first duration is greater than or equal to the maximum duration, the processing method corresponding to the batch processing request is determined to be asynchronous processing.

[0145] For example, assuming a maximum latency of 10 seconds and a first duration of 8 seconds, the processing method for batch processing requests can be determined to be synchronous. If the maximum latency is 10 seconds and the first duration is 12 seconds, the processing method for batch processing requests can be determined to be asynchronous.

[0146] S405. Determine the waiting time and, based on the waiting time and maximum delay, determine the processing method corresponding to the batch processing request.

[0147] The waiting time can be the time between the moment when the business processing device receives the batch processing request and the moment when the remaining resources of the business processing device are greater than 0.

[0148] In this embodiment, if the waiting time is greater than the maximum delay, the processing method corresponding to the batch processing request is determined to be asynchronous; if the waiting time is less than the maximum delay, the time difference between the maximum delay and the waiting time is obtained. If the time difference is less than or equal to the preset time difference, the processing method corresponding to the batch processing request is determined to be asynchronous; if the time difference is greater than the preset time difference, the processing method corresponding to the batch processing request is determined to be synchronous.

[0149] The preset time difference can be set according to actual needs, and this application embodiment does not limit it. For example, the preset time difference can be 2 seconds.

[0150] For example, assuming the maximum latency is 10 seconds and the waiting time is 12 seconds, it can be determined that the processing method corresponding to the batch processing request is the asynchronous processing method.

[0151] Assuming a maximum latency of 10 seconds, a waiting time of 9 seconds, and a preset time difference of 2 seconds, the time difference between the maximum latency and the waiting time is 1 second. Since the time difference is less than the preset time difference, it can be determined that the processing method corresponding to the batch processing request is asynchronous processing.

[0152] Assuming a maximum latency of 10 seconds, a waiting time of 5 seconds, and a preset time difference of 2 seconds, the time difference between the maximum latency and the waiting time is 5 seconds. Since this time difference is greater than the preset time difference, it can be determined that the processing method for batch processing requests is synchronous processing.

[0153] S406. Obtain N serial numbers from the business processing device and / or the preset cache device.

[0154] S407. Based on the processing method and N serial numbers, store the N data to be stored in the corresponding database.

[0155] It should be noted that the specific implementation methods of S406-S407 can be found in S203-S204, and will not be repeated here.

[0156] The database-based data processing method provided in this embodiment allows a business processing device to receive batch processing requests, each including N data items to be stored and a maximum latency. The device can determine if its remaining resources are greater than zero. If the remaining resources are greater than zero, the device can determine a first processing time required for synchronous processing and a second processing time required for asynchronous processing based on the remaining resources and the number of data items (N). It can then determine the processing method corresponding to the batch processing request based on the first processing time, the second processing time, and the maximum latency. If the remaining resources are zero, the device can determine a waiting time and, based on the waiting time and the maximum latency, determine the processing method corresponding to the batch processing request. This method shortens the time required to store the data and improves data processing efficiency.

[0157] Figure 5 This is a schematic diagram of a data processing apparatus provided in an embodiment of this application. The data processing apparatus is applied to business processing equipment; please refer to [link / reference needed]. Figure 5 The data processing module 10 includes: a receiving module 11, a determining module 12, an acquiring module 13, and a storage module 14, wherein...

[0158] The receiving module 11 is used to receive a batch processing request, wherein the batch processing request includes N data to be stored and a maximum delay, wherein N is an integer greater than or equal to 1;

[0159] The determining module 12 is used to determine the remaining resources of the service processing device, and to determine the processing method corresponding to the batch processing request based on the remaining resources and the maximum latency. The processing method includes: synchronous processing method and asynchronous processing method.

[0160] The acquisition module 13 is used to acquire N serial numbers from the business processing device and / or the preset cache device;

[0161] The storage module 14 is used to store the N data to be stored into the corresponding database according to the processing method and the N serial numbers.

[0162] The data processing device provided in this embodiment can be used to execute the database-based data processing method in the above method embodiments. Its implementation principle and technical effect are similar, and will not be described in detail here.

[0163] In one possible implementation, the determining module 12 is specifically used for,

[0164] If the remaining resource quantity is 0, then the waiting time is determined, and the processing method corresponding to the batch processing request is determined based on the waiting time and the maximum delay;

[0165] If the remaining resource quantity is greater than 0, then based on the remaining resource quantity and the quantity N of data to be stored, the first duration required for synchronous processing and the second duration required for asynchronous processing are determined, and based on the first duration, the second duration, and the maximum latency, the processing method corresponding to the batch processing request is determined.

[0166] In one possible implementation, the determining module 12 is specifically used for,

[0167] If the waiting time is greater than the maximum delay, then the processing method corresponding to the batch processing request is determined to be the asynchronous processing method;

[0168] If the waiting time is less than the maximum delay, the time difference between the maximum delay and the waiting time is obtained. If the time difference is less than or equal to a preset time difference, the processing method corresponding to the batch processing request is determined to be the asynchronous processing method. If the time difference is greater than the preset time difference, the processing method corresponding to the batch processing request is determined to be the synchronous processing method.

[0169] In one possible implementation, the determining module 12 is specifically used for,

[0170] If the first duration is less than the maximum delay, then the processing method corresponding to the batch processing request is determined to be the synchronous processing method;

[0171] If the first duration is greater than or equal to the maximum duration, then the processing method corresponding to the batch processing request is determined to be the asynchronous processing method.

[0172] In one possible implementation, the acquisition module 13 is specifically used for,

[0173] Determine the first number M of serial numbers existing in the business processing device, where M is an integer;

[0174] If M is greater than or equal to N, then the N serial numbers are obtained from the service processing device;

[0175] If M is less than N and M is 0, then the N serial numbers are obtained from the preset cache device;

[0176] If M is less than N and M is not 0, then M serial numbers are obtained in the service processing device and NM serial numbers are obtained in the preset cache device.

[0177] In one possible implementation, the acquisition module 13 is specifically used for,

[0178] Obtain the device status of the preset cache device, wherein the device status is either locked or unlocked.

[0179] When the device state of the preset cache device is unlocked, a serial number acquisition request is sent to the preset cache device, and the device state of the preset cache device is controlled to switch to the locked state. The serial number acquisition request includes N.

[0180] Receive N serial numbers sent by the preset cache device, and control the device state of the preset cache device to switch to the unlocked state.

[0181] In one possible implementation, the storage module 14 is specifically used for,

[0182] The storage component corresponding to the processing method is determined, and the storage component is implemented through Java code;

[0183] Based on the N sequence numbers and the N data to be stored, generate N data pairs;

[0184] The storage component is invoked, and the N data pairs are stored in the corresponding database through the storage component.

[0185] In one possible implementation, the acquisition module 13 is specifically used for,

[0186] Among the N sequence numbers, determine the sequence number corresponding to each piece of data to be stored;

[0187] According to the preset format, each piece of data to be stored and its corresponding sequence number are combined to obtain the N data pairs.

[0188] The data processing device provided in this embodiment can be used to execute the database-based data processing method in the above method embodiments. Its implementation principle and technical effect are similar, and will not be described in detail here.

[0189] Figure 6 This is a schematic diagram of the hardware structure of a data processing device provided in an embodiment of this application. Please refer to [link / reference]. Figure 6The data processing device 20 may include a processor 21 and a memory 22, wherein the processor 21 and the memory 22 can communicate; for example, the processor 21 and the memory 22 communicate via a communication bus 23, wherein the memory 22 is used to store computer execution instructions, and the processor 21 is used to call the computer execution instructions in the memory to execute the database-based data processing method shown in any of the above method embodiments.

[0190] Optionally, the data processing device 20 may also include a communication interface, which may include a transmitter and / or a receiver.

[0191] Optionally, the aforementioned processor can be a Central Processing Unit (CPU), or other general-purpose processors, digital signal processors (DSPs), application-specific integrated circuits (ASICs), etc. The general-purpose processor can be a microprocessor or any conventional processor. The steps of the method disclosed in this application can be directly implemented by a hardware processor, or implemented by a combination of hardware and software modules within the processor.

[0192] This application provides a computer-readable storage medium storing computer-executable instructions; the computer-executable instructions are used to implement the database-based data processing method as described in any of the above embodiments.

[0193] This application provides a computer program product, which includes a computer program that, when executed, causes a computer to perform the aforementioned database-based data processing method.

[0194] All or part of the steps in the above method embodiments can be implemented by hardware related to program instructions. The aforementioned program can be stored in a readable memory. When the program is executed, it performs the steps of the above method embodiments; and the aforementioned memory (storage medium) includes: read-only memory (ROM), RAM, flash memory, hard disk, solid-state drive, magnetic tape, floppy disk, optical disk, and any combination thereof.

[0195] This application describes embodiments with reference to flowchart illustrations and / or block diagrams of methods, apparatus (systems), and computer program products according to embodiments of this application. It should be understood that each block of the flowchart illustrations and / or block diagrams, and combinations of blocks in the flowchart illustrations and / or block diagrams, can be implemented by computer program instructions. These computer program instructions can be provided to a processing unit of a general-purpose computer, special-purpose computer, embedded processor, or other programmable terminal device to produce a machine, such that the instructions, which execute via the processing unit of the computer or other programmable terminal device, generate instructions for implementing the flowchart. Figure 1 One or more processes and / or boxes Figure 1 A device that provides the functions specified in one or more boxes.

[0196] These computer program instructions may also be stored in a computer-readable storage medium that can direct a computer or other programmable terminal device to function in a particular manner, such that the instructions stored in the computer-readable storage medium produce an article of manufacture including instruction means, which are implemented in a process Figure 1 One or more processes and / or boxes Figure 1 The function specified in one or more boxes.

[0197] These computer program instructions can also be loaded onto a computer or other programmable terminal device, causing a series of operational steps to be performed on the computer or other programmable device to produce a computer-implemented process, thereby providing instructions that execute on the computer or other programmable device for implementing the process. Figure 1 One or more processes and / or boxes Figure 1 The steps of the function specified in one or more boxes.

[0198] Obviously, those skilled in the art can make various modifications and variations to the embodiments of this application without departing from the spirit and scope of this application. Therefore, if these modifications and variations to the embodiments of this application fall within the scope of the claims of this application and their equivalents, this application also intends to include these modifications and variations.

[0199] In this application, the term "comprising" and its variations can refer to non-limiting inclusion; the term "or" and its variations can refer to "and / or". The terms "first", "second", etc., in this application are used to distinguish similar objects and are not necessarily used to describe a specific order or sequence. In this application, "multiple" refers to two or more. "And / or" describes the relationship between related objects, indicating that three relationships can exist. For example, A and / or B can represent: A existing alone, A and B existing simultaneously, and B existing alone. The character " / " generally indicates that the preceding and following related objects have an "or" relationship.

[0200] Other embodiments of this application will readily occur to those skilled in the art upon consideration of the invention disclosed in the specification and in practice. This application is intended to cover any variations, uses, or adaptations of this application that follow the general principles of this application and include common knowledge or customary techniques in the art not disclosed in this application.

Claims

1. A database-based data processing method, characterized by, The method is applied to a service processing device, and comprises: receiving a batch processing request, wherein the batch processing request comprises N pieces of to-be-stored data and a maximum time delay, N is an integer greater than or equal to 1, and the maximum time delay is a maximum allowed time consumption for processing the batch processing request; determining a remaining resource amount of the service processing device; if the remaining resource amount is 0, determining a waiting time length, when the waiting time length is greater than the maximum time delay, determining that a processing mode corresponding to the batch processing request is an asynchronous processing mode, when the waiting time length is less than the maximum time delay, determining that the processing mode corresponding to the batch processing request is the asynchronous processing mode or a synchronous processing mode, if the remaining resource amount is greater than 0, determining a first time length required by the synchronous processing mode according to the remaining resource amount and the number N of the to-be-stored data, and determining the processing mode corresponding to the batch processing request according to the first time length and the maximum time delay; obtaining N serial numbers in the service processing device and / or a preset cache device; storing the N pieces of to-be-stored data into a corresponding database according to the processing mode and the N serial numbers; wherein the determining the processing mode corresponding to the batch processing request according to the first time length and the maximum time delay comprises: when the first time length is less than the maximum time delay, determining that the processing mode corresponding to the batch processing request is the synchronous processing mode, and when the first time length is greater than or equal to the maximum time delay, determining that the processing mode corresponding to the batch processing request is the asynchronous processing mode.

2. The method of claim 1, wherein, if the remaining resource amount is greater than 0, determining a first time length required by the synchronous processing mode according to the remaining resource amount and the number N of the to-be-stored data, comprises: if the remaining resource amount is greater than 0, determining a first time length required by the synchronous processing mode, a second time length required by the asynchronous processing mode according to the remaining resource amount and the number N of the to-be-stored data; the determining the processing mode corresponding to the batch processing request according to the first time length and the maximum time delay comprises: determining the processing mode corresponding to the batch processing request according to the first time length, the second time length and the maximum time delay.

3. The method of claim 1, wherein, when the waiting time length is less than the maximum time delay, determining that the processing mode corresponding to the batch processing request is the asynchronous processing mode or the synchronous processing mode, comprises: when the waiting time length is less than the maximum time delay, obtaining a time difference between the maximum time delay and the waiting time length, if the time difference is less than or equal to a preset time difference, determining that the processing mode corresponding to the batch processing request is the asynchronous processing mode, and if the time difference is greater than the preset time difference, determining that the processing mode corresponding to the batch processing request is the synchronous processing mode.

4. The method according to any one of claims 1 to 3, characterized in that, the obtaining N serial numbers in the service processing device and / or a preset cache device comprises: judging a first number M of serial numbers existing in the service processing device, M is an integer; if the M is greater than or equal to the N, obtaining the N serial numbers in the service processing device; If the M is less than the N and the M is 0, the N serial numbers are acquired in the preset cache device; If the M is less than the N and the M is not 0, M serial numbers are acquired in the service processing device, and N-M serial numbers are acquired in the preset cache device.

5. The method of claim 4, wherein, The N serial numbers are acquired in the preset cache device, comprising: Acquiring a device state of the preset cache device, the device state being a locked state or an unlocked state; When the device state of the preset cache device is the unlocked state, sending a serial number acquisition request to the preset cache device, and controlling the device state of the preset cache device to switch to the locked state, the serial number acquisition request comprising the N; Receiving the N serial numbers sent by the preset cache device, and controlling the device state of the preset cache device to switch to the unlocked state.

6. The method according to any of claims 1 to 3, 5, characterized in that, According to the processing mode and the N serial numbers, the N to-be-stored data are stored into corresponding databases, comprising: Determining a storage component corresponding to the processing mode, the storage component being implemented by an object-oriented programmable language Java code; According to the N serial numbers and the N to-be-stored data, N data pairs are generated; Calling the storage component, and storing the N data pairs into corresponding databases through the storage component.

7. The method of claim 6, wherein, According to the N serial numbers and the N to-be-stored data, N data pairs are generated, comprising: In the N serial numbers, determining a serial number corresponding to each to-be-stored data; According to a preset format, combining each to-be-stored data and the corresponding serial number to obtain the N data pairs.

8. A data processing apparatus, characterized by, The data processing apparatus is applied to a service processing device, and the data processing apparatus comprises a receiving module, a determining module, an acquiring module and a storage module, wherein, The receiving module is configured to receive a batch processing request, the batch processing request comprising N to-be-stored data and a maximum time delay, the N being an integer greater than or equal to 1; the maximum time delay being a maximum allowed time consumption for processing the batch processing request; The determining module is configured to determine a remaining resource amount of the service processing device; if the remaining resource amount is 0, a waiting time length is determined; when the waiting time length is greater than the maximum time delay, a processing mode corresponding to the batch processing request is determined as an asynchronous processing mode; when the waiting time length is less than the maximum time delay, the processing mode corresponding to the batch processing request is determined as an asynchronous processing mode or a synchronous processing mode; if the remaining resource amount is greater than 0, a first time length required by the synchronous processing mode is determined according to the remaining resource amount and the number N of the to-be-stored data; the processing mode corresponding to the batch processing request is determined according to the first time length and the maximum time delay; The acquiring module is configured to acquire N serial numbers in the service processing device and / or a preset cache device; The storage module is configured to store the N to-be-stored data into corresponding databases according to the processing mode and the N serial numbers. The determining module, when determining the processing mode corresponding to the batch processing request according to the first time length and the maximum time delay, is specifically configured to: when the first time length is less than the maximum time delay, determine that the processing mode corresponding to the batch processing request is a synchronous processing mode; and when the first time length is greater than or equal to the maximum time delay, determine that the processing mode corresponding to the batch processing request is an asynchronous processing mode.

9. A data processing device, characterized by Comprising: a processor, and a memory connected with the processor in communication; the memory stores computer-executable instructions; the processor executes the computer-executable instructions stored in the memory to implement the method of any one of claims 1 to 7.

10. A computer readable storage medium characterized by, The computer-readable storage medium stores computer-executable instructions, and the computer-executable instructions, when executed by a processor, are used to implement the method of any one of claims 1 to 7.

11. A computer program product, characterised in that, comprising a computer program, which, when executed by a processor, implements the method of any one of claims 1 to 7.

Images