Concurrent data processing method, device and equipment for data unloading batch operation and medium

By dynamically adjusting the concurrency number of unloading batches, the problem of database resource competition caused by excessive fixed concurrency number during peak periods is solved, and customer experience and system stability are improved.

CN119066096APending Publication Date: 2024-12-03PING AN BANK CO LTD
View PDF 0 Cites 0 Cited by

Patent Information

Application Number
CN202411079586.5
Authority / Receiving Office
CN · China
Patent Type
Applications(China)
Current Assignee / Owner
Filing Date
2024-08-07
Publication Date
2024-12-03

AI Technical Summary

Technical Problem

During peak business hours of banks and other financial institutions, the number of unloading and batch operations are too large, resulting in database resources competing for each other, affecting customer experience.

Method used

By determining whether the current time is the peak business and the peak batch running period, dynamically adjust the concurrency number, perform pre-run batch operations and unload batch running periods, to avoid processing during peak periods and reduce the concurrency number in non-peak periods.

Benefits of technology

It effectively avoids the competition for database resources, improves the experience of customer business processing, and ensures the efficiency of data processing and system stability.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure CN119066096A_ABST
    Figure CN119066096A_ABST
Patent Text Reader

Abstract

The invention discloses an unloading batch job concurrent data processing method, device and equipment and a medium, and the method comprises the steps: carrying out the judgment of a pre-batch job and an unloading batch job on the unloading batch job, and judging whether the current time is a business peak handling period and a batch peak period or not; the pre-start time of the pre-batch operation and the current concurrency number of the unloading batch operation are optimized and adjusted, the unloading batch operation is prevented from being processed in the business peak handling period, the current concurrency number is adjusted according to the batch operation peak period, the current concurrency number in the non-batch operation peak period is increased, and the operation efficiency is improved. According to the method and the device, the current concurrence number in the batch run peak period is reduced, and the technical problems that database resources are scrambled, customer business handling is influenced and the customer experience is reduced due to the fact that the number unloading batch run operation is triggered and executed in a large quantity in the customer business handling peak period and the value of the fixed concurrence number is relatively large are solved.
Need to check novelty before this filing date? Find Prior Art

Description

Technical Field

[0001] The present application relates to the field of data processing technology, and in particular to a method, device, equipment and medium for concurrent data processing of batch unloading operations. Background Art

[0002] In the era of big data, banks and other financial institutions are faced with the challenge of massive amounts of data. This data not only needs to be stored effectively, but also needs to be processed and analyzed quickly to support decision making. Batch operations are an important part of data processing, which involves data collection, processing, and unloading to ensure the timeliness and accuracy of the data.

[0003] Banks use a variety of database systems, such as Oracle, MySQL, OceanBase, TiDB, Elasticsearch, TDSQL, and PostgreSQL, to store and manage data. The data collection job is responsible for extracting data from the upstream database system and transferring it to the Hive library in the Hadoop ecosystem.

[0004] In the Hive database, data is processed through cleaning, conversion, aggregation and other processing logic to meet the needs of downstream applications. The processed data needs to be unloaded to the downstream database system, a process called unloading.

[0005] Batch jobs are a series of automated data processing tasks that are usually executed during off-peak hours to reduce the impact on the production environment. To improve data processing efficiency, you can increase the number of concurrent jobs to achieve multi-threaded parallel data processing. However, too high a number of concurrent jobs may cause excessive database load and affect system stability.

[0006] Currently, batch jobs are basically configured by setting the number of concurrent operations to a fixed value, such as 4, 8, or 16. Usually, after the data processing in the Hive database is completed, the call and execution of the data unloading job will be automatically triggered.

[0007] Since processing operations in the Hive database are generally handled offline, processing operations do not cause resource pressure in the Hive database. The DB database downstream of the Hive database is a real-time application database. If the batch unloading operation is triggered and executed in large quantities during the peak period of customer business processing, and the fixed concurrent number value is large, it will cause competition for database resources, affect customer business processing, and reduce customer experience. Summary of the invention

[0008] The present application provides a method, device, equipment and medium for processing the concurrent number of batch unloading operations, which solves the technical problem that if the batch unloading operations are triggered and executed in large quantities during the peak period of customer business processing, and the fixed concurrent number value is large, it will lead to competition for database resources, affect customer business processing, and reduce the customer experience.

[0009] In view of this, the first aspect of the present application provides a method for processing concurrent data of batch unloading operations, the method comprising:

[0010] Step S1: when the current time is not less than the end time of the processing operation and the pre-start time, determine whether to execute the pre-batch operation according to the unloading batch operation status;

[0011] Step S2: If the state of the batch unloading operation is not executed, the pre-batch operation is executed, and it is determined in turn whether the current time is a peak business handling period and whether the current time is a batch running peak period;

[0012] Step S3: If the current time is not a business peak processing period or a batch running peak period, the current concurrency number is set to the current concurrency number plus a preset value, and the batch running operation is performed based on the current concurrency number, and the current time is recorded as the actual start time;

[0013] Step S4: During the process of unloading and batching, it is determined in real time whether the current time is the peak business processing period and the batching peak period;

[0014] Step S5: If the current time is not the peak business processing period and the batch running peak period, set the current concurrency number to the current concurrency number plus the preset value until the batch unloading operation is completed, update the expected start time to the actual start time, and the batch unloading operation status is the executed status.

[0015] Optionally, the step S3 further includes:

[0016] If the current time is a peak business processing period, the pre-start time is updated so that the pre-start time is increased by a preset time, the pre-run batch job is terminated, and the process returns to step S1.

[0017] Optionally, the step S3 further includes:

[0018] If the current time is the batch running peak period, the current concurrency number is updated to the current concurrency number minus the preset value, and the pre-batch running job is terminated, and the execution returns to step S1.

[0019] Optionally, the step S5 further includes:

[0020] If the current time is a peak business period, the pre-start time is updated so that the pre-start time is increased by a preset time, the batch unloading operation is terminated, and the process returns to step S1.

[0021] Optionally, the step S5 further includes:

[0022] If the current time is the peak period for batch running, the current concurrent number is updated to the current concurrent number minus the preset value, and the batch running operation is terminated, returning to step S4.

[0023] Optionally, the determination of whether the current time is a peak business handling period is specifically as follows:

[0024] Get real-time traffic data from the DB application front end;

[0025] The real-time traffic data is compared with the traffic threshold to determine whether the current time is a peak business processing period.

[0026] Optionally, the determination of whether the current time is a peak business handling period is specifically as follows:

[0027] Obtain target indicator data for databases and application servers;

[0028] The target indicator data of the database and the application server are respectively compared with the corresponding indicator threshold data to determine whether the current time is a business peak processing period.

[0029] A second aspect of the present application provides a device for processing concurrent data in batch unloading operations, the device comprising:

[0030] The pre-batch run judgment unit is used to judge whether to execute the pre-batch run operation according to the unloading batch run operation status when the current time is not less than the end time of the processing operation and the pre-start time;

[0031] The first processing unit is used to execute the pre-batch operation if the state of the unloading batch operation is not executed, and to determine whether the current time is a business peak handling period and whether the current time is a batch running peak period in turn;

[0032] The second processing unit is used to set the current concurrency number to the current concurrency number plus a preset value if the current time is not a business peak processing period and a batch running peak period, and perform the batch unloading operation based on the current concurrency number, and record the current time as the actual start time;

[0033] The data unloading and batch running judgment unit is used to judge in real time whether the current time is the peak business processing period and the batch running peak period during the data unloading and batch running operation;

[0034] The third processing unit is used to set the current concurrency number to the current concurrency number plus the preset value if the current time is not a business peak processing period and a batch running peak period, until the unloading and batch running job is completed, update the expected start time to the actual start time, and the unloading and batch running job status is the executed status.

[0035] Optionally, the second processing unit is further used for:

[0036] If the current time is a peak business processing period, the pre-start time is updated so that after the pre-start time is increased by a preset time, the pre-batch operation is terminated and the process is transferred to the pre-batch determination unit.

[0037] Optionally, the second processing unit is further used for:

[0038] If the current time is the batch running peak period, the current concurrency number is updated to the current concurrency number minus the preset value, and the pre-batch running operation is terminated, jumping to the pre-batch running judgment unit.

[0039] Optionally, the third processing unit is further configured to:

[0040] If the current time is a peak business period, the pre-start time is updated so that after the pre-start time is increased by a preset time, the batch unloading operation is terminated and the process is transferred to the pre-batch determination unit.

[0041] Optionally, the third processing unit is further configured to:

[0042] If the current time is the batch running peak period, the current concurrent number is updated to the current concurrent number minus the preset value, and the batch unloading operation is terminated, jumping to the batch unloading judgment unit.

[0043] Optionally, the determination of whether the current time is a peak business handling period is specifically as follows:

[0044] Get real-time traffic data from the DB application front end;

[0045] The real-time traffic data is compared with the traffic threshold to determine whether the current time is a peak business processing period.

[0046] Optionally, the determination of whether the current time is a peak business handling period is specifically as follows:

[0047] Obtain target indicator data for databases and application servers;

[0048] The target indicator data of the database and the application server are respectively compared with the corresponding indicator threshold data to determine whether the current time is a business peak processing period.

[0049] A third aspect of the present application provides a device for concurrently processing batch jobs, the device comprising a processor and a memory:

[0050] The memory is used to store program code and transmit the program code to the processor;

[0051] The processor is used to execute the steps of the method for unloading batch jobs and concurrently processing data as described in the first aspect according to the instructions in the program code.

[0052] A fourth aspect of the present application provides a computer-readable storage medium, wherein the computer-readable storage medium is used to store program code, and the program code is used to execute the method described in the first aspect above.

[0053] It can be seen from the above technical solutions that the embodiments of the present application have the following advantages:

[0054] In the present application, a method, device, equipment and medium for processing the concurrent number of batch unloading operations are provided. The pre-batch operation and the unloading batch operation are judged respectively for the unloading batch operation. By judging whether the current time is the peak business processing period and the batch running peak period, the pre-start time of the pre-batch operation and the current concurrent number of the unloading batch operation are optimized and adjusted, so as to avoid the unloading batch operation from being processed during the peak business processing period, and adjust the current concurrent number according to the batch running peak period, increase the current concurrent number during non-batch running peak period, and reduce the current concurrent number during batch running peak period, so as to solve the technical problem that if the unloading batch operation is triggered and executed in large quantities during the peak period of customer business processing, and the fixed concurrent number value is large, the database resource competition will be caused, which will affect the customer business processing and reduce the customer experience. BRIEF DESCRIPTION OF THE DRAWINGS

[0055] Figure 1 This is a method flow chart of a method for processing concurrent data in batch unloading operations in an embodiment of the present application;

[0056] Figure 2 This is a schematic diagram of the structure of a concurrent data processing device for batch unloading operations in an embodiment of the present application;

[0057] Figure 3 This is a schematic diagram of the structure of the concurrent data processing equipment for batch unloading operations in an embodiment of the present application. DETAILED DESCRIPTION

[0058] In order to enable those skilled in the art to better understand the solution of the present application, the technical solution in the embodiments of the present application will be clearly and completely described below in conjunction with the drawings in the embodiments of the present application. Obviously, the described embodiments are only part of the embodiments of the present application, not all of the embodiments. Based on the embodiments in the present application, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of this application.

[0059] The present application designs a method, device, equipment and medium for processing the concurrent number of batch unloading operations, which solves the technical problem that if the batch unloading operations are triggered and executed in large quantities during the peak period of customer business processing, and the fixed concurrent number value is large, it will lead to competition for database resources, affect customer business processing, and reduce the customer experience.

[0060] For easier understanding, see Figure 1 , Figure 1 Flow chart of the method for processing concurrent batch operations of unloading data in the embodiment of the present application. Figure 1 As shown, specifically:

[0061] Step S1: when the current time is not less than the end time of the processing operation and the pre-start time, determine whether to execute the pre-batch operation according to the unloading batch operation status;

[0062] It should be noted that the end time of the processing operation and the expected start time should be consistent when first determined, and the expected start time will be optimized and adjusted after being determined during the subsequent business peak processing period.

[0063] When the unloading and batching job is executed daily, it is necessary to check the status of the unloading and batching job. It is understandable that since the unloading and batching job is a T+1 time-limited job, the status of the unloading and batching job can be reset to the unexecuted state after 0:00 every day. After the unloading and batching job is completed, the status of the unloading and batching job is updated to the executed state.

[0064] Since the end time of the daily hive database processing operation is uncertain, it may end in the morning, noon, afternoon or evening. If the end time of the processing operation for the day is not detected at the current time, the pre-batch operation and the unloading batch operation will not be executed at all.

[0065] Step S2: If the batch unloading operation status is not executed, the pre-batch operation is executed, and it is determined in turn whether the current time is a peak business handling period and whether the current time is a batch running peak period;

[0066] It should be noted that if the status of the unloading and batch running job is not executed, the pre-batch running job will be entered. The pre-batch running job is a detection status, which makes a preliminary judgment on whether the unloading and batch running job can be entered. It mainly needs to determine whether the current time is the peak business processing period and the batch running peak period.

[0067] Step S3: If the current time is not the peak business processing period or the batch running peak period, the current concurrency number is set to the current concurrency number plus the preset value, and the batch unloading operation is performed based on the current concurrency number, and the current time is recorded as the actual start time;

[0068] It should be noted that when the current time is not the peak business processing period and the batch running peak period, it can be determined that the unloading and batch running job can be executed. At the same time, the current concurrency number is increased accordingly. When entering the unloading and batch running job execution, the efficiency of the unloading and batch running job can be improved.

[0069] During the initial determination, the current number of concurrent connections is the initial value, and when the unloading batch job status is updated daily, the current number of concurrent connections will be reset to the initial value. You can choose a suitable value for the initial value, such as 8.

[0070] The current concurrency number also has a minimum threshold and a maximum threshold, which are determined according to the executable concurrency number. When the current concurrency number reaches the minimum threshold, it cannot be further reduced; when the current concurrency number reaches the maximum threshold, it cannot be further increased.

[0071] Step S4: during the process of unloading and batching, determine in real time whether the current time is the peak business processing period and the batching peak period;

[0072] It should be noted that during the process of unloading and batching operations, it will also be determined in real time whether the current time is the peak business processing period or the peak batching period, in order to prevent the business peak processing period or the peak batching period from suddenly entering during the process of unloading and batching operations.

[0073] Step S5: If the current time is not the peak business processing period or the batch running peak period, set the current concurrency number to the current concurrency number plus the preset value until the batch unloading operation is completed, update the expected start time to the actual start time, and the batch unloading operation status is executed.

[0074] It should be noted that, only when the current time is not the peak business processing period and the batch running peak period, the current concurrency number will continue to increase based on the pre-batch running job judgment until the unloading batch running job is completed.

[0075] After the data unloading is completed, update the batch job status to the executed status.

[0076] Update the expected start time to the actual start time to record the time when the activity actually started.

[0077] Furthermore, step S3 also includes:

[0078] If the current time is a peak business processing period, the estimated start time is updated so that the estimated start time increases by a preset time, the pre-batch operation is terminated, and the process returns to step S1.

[0079] Furthermore, step S3 also includes:

[0080] If the current time is the batch running peak period, the current concurrency number is updated to the current concurrency number minus the preset value, and the pre-batch running job is terminated, and the execution returns to step S1.

[0081] Furthermore, step S5 further includes:

[0082] If the current time is during the peak business hours, the estimated start time is updated so that the estimated start time increases by a preset time, and then the batch unloading operation is terminated and the process returns to step S1.

[0083] Furthermore, step S5 further includes:

[0084] If the current time is the peak period for batch running, the current concurrent number is updated to the current concurrent number minus the preset value, and the batch running operation is terminated, returning to step S4.

[0085] Further, whether the current time is a peak business processing period is determined as follows:

[0086] Get real-time traffic data from the DB application front end;

[0087] Compare the real-time traffic data with the traffic threshold to determine whether the current time is the peak business processing period.

[0088] Further, whether the current time is a peak business processing period is determined as follows:

[0089] Obtain target indicator data for databases and application servers;

[0090] Compare the target indicator data of the database and application server with the corresponding indicator threshold data to determine whether the current time is the peak business processing period.

[0091] It should be noted that the peak business processing period can be determined by the DB application front-end page traffic, such as the number of page visits, page visit duration, and active number.

[0092] You can also use database and server monitoring indicators to make judgments, such as qps, tps, IOPS, etc.

[0093] In one embodiment, the peak business period can be simply determined when the application server's cup usage reaches 50% of the previous day's peak, or the database's real-time qps value reaches 50% of the previous day's peak, or the database's real-time tps value reaches 50% of the previous day's peak.

[0094] Furthermore, the peak period of batch operation can be determined based on the IOPS value of the database. For example, the real-time IOPS value of the database reaches 50% of the peak value of the previous day.

[0095] In addition, if you encounter abnormal situations during the batch running process, such as excessive database load or system failure, you can configure the corresponding exception handling and recovery mechanism.

[0096] See also Figure 2 , Figure 2 Schematic diagram of the structure of the device for processing batch operations and concurrent data in the embodiment of the present application. Figure 2 As shown, specifically:

[0097] The pre-batch run judgment unit 201 is used to judge whether to execute the pre-batch run operation according to the unloading batch run operation state when the current time is not less than the end time of the processing operation and the pre-start time;

[0098] It should be noted that the end time of the processing operation and the expected start time should be consistent when first determined, and the expected start time will be optimized and adjusted after being determined during the subsequent business peak processing period.

[0099] When the unloading and batching job is executed daily, it is necessary to check the status of the unloading and batching job. It is understandable that since the unloading and batching job is a T+1 time-limited job, the status of the unloading and batching job can be reset to the unexecuted state after 0:00 every day. After the unloading and batching job is completed, the status of the unloading and batching job is updated to the executed state.

[0100] Since the end time of the daily hive database processing operation is uncertain, it may end in the morning, noon, afternoon or evening. If the end time of the processing operation for the day is not detected at the current time, the pre-batch operation and the unloading batch operation will not be executed at all.

[0101] The first processing unit 202 is used to execute the pre-batch operation if the unloading batch operation status is not executed, and determine whether the current time is a business peak handling period and whether the current time is a batch operation peak period in sequence;

[0102] It should be noted that if the status of the unloading and batch running job is not executed, the pre-batch running job will be entered. The pre-batch running job is a detection status, which makes a preliminary judgment on whether the unloading and batch running job can be entered. It mainly needs to determine whether the current time is the peak business processing period and the batch running peak period.

[0103] The second processing unit 203 is used to set the current concurrency number to the current concurrency number plus the preset value if the current time is not the business peak processing period and the batch running peak period, and perform the batch unloading operation based on the current concurrency number, and record the current time as the actual start time;

[0104] It should be noted that when the current time is not the peak business processing period and the batch running peak period, it can be determined that the unloading and batch running job can be executed. At the same time, the current concurrency number is increased accordingly. When entering the unloading and batch running job execution, the efficiency of the unloading and batch running job can be improved.

[0105] During the initial determination, the current number of concurrent connections is the initial value, and when the unloading batch job status is updated daily, the current number of concurrent connections will be reset to the initial value. You can choose a suitable value for the initial value, such as 8.

[0106] The current concurrency number also has a minimum threshold and a maximum threshold.

[0107] The data unloading and batch running judgment unit 204 is used to judge in real time whether the current time is a business peak handling period and a batch running peak period during the data unloading and batch running operation;

[0108] It should be noted that during the process of unloading and batching operations, it will also be determined in real time whether the current time is the peak business processing period or the peak batching period, in order to prevent the business peak processing period or the peak batching period from suddenly entering during the process of unloading and batching operations.

[0109] The third processing unit 205 is used to set the current concurrency number to the current concurrency number plus the preset value if the current time is not the peak business processing period and the batch running peak period, until the unloading and batch running job is completed, update the expected start time to the actual start time, and the unloading and batch running job status is the executed status.

[0110] Furthermore, the second processing unit 203 is further configured to:

[0111] If the current time is a peak business processing period, the pre-start time is updated so that the pre-start time increases by a preset time, the pre-batch operation is terminated, and the process jumps to the pre-batch determination unit 201 .

[0112] Furthermore, the second processing unit 203 is further configured to:

[0113] If the current time is the batch running peak period, the current concurrency number is updated to the current concurrency number minus the preset value, and the pre-batch running operation is terminated, jumping to the pre-batch running judgment unit 201.

[0114] Furthermore, the third processing unit 205 is further configured to:

[0115] If the current time is a peak business period, the pre-start time is updated so that the pre-start time increases by a preset time, the batch unloading operation is terminated, and the process jumps to the pre-batch determination unit 201.

[0116] Furthermore, the third processing unit 205 is further configured to:

[0117] If the current time is the batch running peak period, the current concurrent number is updated to the current concurrent number minus the preset value, and the batch unloading operation is terminated, jumping to the batch unloading judgment unit 204.

[0118] It should be noted that, only when the current time is not the peak business processing period and the batch running peak period, the current concurrency number will continue to increase based on the pre-batch running job judgment until the unloading batch running job is completed.

[0119] Further, whether the current time is a peak business processing period is determined as follows:

[0120] Get real-time traffic data from the DB application front end;

[0121] Compare the real-time traffic data with the traffic threshold to determine whether the current time is the peak business processing period.

[0122] Further, whether the current time is a peak business processing period is determined as follows:

[0123] Obtain target indicator data for databases and application servers;

[0124] Compare the target indicator data of the database and application server with the corresponding indicator threshold data to determine whether the current time is the peak business processing period.

[0125] It should be noted that the peak business processing period can be determined by the DB application front-end page traffic, such as the number of page visits, page visit duration, and active number.

[0126] You can also use database and server monitoring indicators to make judgments, such as qps, tps, IOPS, etc.

[0127] In one embodiment, the peak business period can be simply determined when the application server's cup usage reaches 50% of the previous day's peak, or the database's real-time qps value reaches 50% of the previous day's peak, or the database's real-time tps value reaches 50% of the previous day's peak.

[0128] Furthermore, the peak period of batch operation can be determined based on the IOPS value of the database. For example, the real-time IOPS value of the database reaches 50% of the peak value of the previous day.

[0129] In addition, if you encounter abnormal situations during the batch running process, such as excessive database load or system failure, you can configure the corresponding exception handling and recovery mechanism.

[0130] The present application embodiment also provides another unloading batch operation concurrent data processing device, such as Figure 3 For the sake of convenience, only the parts related to the embodiments of the present application are shown. For specific technical details not disclosed, please refer to the method part of the embodiments of the present application. The terminal can be any terminal device including a mobile phone, a tablet computer, a personal digital assistant (English full name: Personal Digital Assistant, English abbreviation: PDA), a sales terminal (English full name: Point of Sales, English abbreviation: POS), a car computer, etc., taking the terminal as a mobile phone as an example:

[0131] Figure 3 FIG. 1 is a block diagram showing a partial structure of a mobile phone related to a terminal provided in an embodiment of the present application. Figure 3 The mobile phone includes: a radio frequency (RF) circuit 1010, a memory 1020, an input unit 1030, a display unit 1040, a sensor 1050, an audio circuit 1060, a wireless fidelity (WiFi) module 1070, a processor 1080, and a power supply 1090. Those skilled in the art can understand that Figure 3 The mobile phone structure shown in the figure does not constitute a limitation on the mobile phone, and may include more or fewer components than shown in the figure, or combine certain components, or arrange the components differently.

[0132] Combine the following Figure 3 A detailed introduction to the various components of the mobile phone:

[0133] The RF circuit 1010 can be used for receiving and sending signals during information transmission or calls. In particular, after receiving the downlink information of the base station, it is sent to the processor 1080 for processing; in addition, the designed uplink data is sent to the base station. Generally, the RF circuit 1010 includes but is not limited to an antenna, at least one amplifier, a transceiver, a coupler, a low noise amplifier (full name: Low Noise Amplifier, English abbreviation: LNA), a duplexer, etc. In addition, the RF circuit 1010 can also communicate with the network and other devices through wireless communication. The above-mentioned wireless communications may use any communication standard or protocol, including but not limited to Global System of Mobile communications (Global System of Mobile communication, English abbreviation: GSM), General Packet Radio Service (General Packet Radio Service, GPRS), Code Division Multiple Access (Code Division Multiple Access, English abbreviation: CDMA), Wideband Code Division Multiple Access (WCDMA), Long Term Evolution (Long Term Evolution, English abbreviation: LTE), e-mail, Short Messaging Service (SMS), etc.

[0134] The memory 1020 can be used to store software programs and modules. The processor 1080 executes various functional applications and data processing of the mobile phone by running the software programs and modules stored in the memory 1020. The memory 1020 can mainly include a program storage area and a data storage area, wherein the program storage area can store an operating system, an application required for at least one function (such as a sound playback function, an image playback function, etc.), etc.; the data storage area can store data created according to the use of the mobile phone (such as audio data, a phone book, etc.), etc. In addition, the memory 1020 can include a high-speed random access memory, and can also include a non-volatile memory, such as at least one disk storage device, a flash memory device, or other volatile solid-state storage devices.

[0135] The input unit 1030 can be used to receive input digital or character information, and to generate key signal input related to the user settings and function control of the mobile phone. Specifically, the input unit 1030 may include a touch panel 1031 and other input devices 1032. The touch panel 1031, also known as a touch screen, can collect the user's touch operation on or near it (such as the user's operation on the touch panel 1031 or near the touch panel 1031 using any suitable object or accessory such as a finger, stylus, etc.), and drive the corresponding connection device according to a pre-set program. Optionally, the touch panel 1031 may include two parts: a touch detection device and a touch controller. Among them, the touch detection device detects the user's touch orientation, detects the signal brought by the touch operation, and transmits the signal to the touch controller; the touch controller receives the touch information from the touch detection device, converts it into contact coordinates, and then sends it to the processor 1080, and can receive and execute commands sent by the processor 1080. In addition, the touch panel 1031 can be implemented in various types such as resistive, capacitive, infrared, and surface acoustic waves. In addition to the touch panel 1031, the input unit 1030 may further include other input devices 1032. Specifically, the other input devices 1032 may include but are not limited to one or more of a physical keyboard, function keys (such as volume control keys, switch keys, etc.), a trackball, a mouse, a joystick, and the like.

[0136] The display unit 1040 can be used to display information input by the user or information provided to the user and various menus of the mobile phone. The display unit 1040 may include a display panel 1041. Optionally, the display panel 1041 may be configured in the form of a liquid crystal display (full name in English: Liquid Crystal Display, English abbreviation: LCD), an organic light-emitting diode (full name in English: Organic Light-Emitting Diode, English abbreviation: OLED), etc. Further, the touch panel 1031 may cover the display panel 1041. When the touch panel 1031 detects a touch operation on or near it, it is transmitted to the processor 1080 to determine the type of touch event. Subsequently, the processor 1080 provides a corresponding visual output on the display panel 1041 according to the type of touch event. Although in Figure 3 In the embodiment, the touch panel 1031 and the display panel 1041 are used as two independent components to realize the input and output functions of the mobile phone, but in some embodiments, the touch panel 1031 and the display panel 1041 can be integrated to realize the input and output functions of the mobile phone.

[0137] The mobile phone may also include at least one sensor 1050, such as a light sensor, a motion sensor, and other sensors. Specifically, the light sensor may include an ambient light sensor and a proximity sensor, wherein the ambient light sensor may adjust the brightness of the display panel 1041 according to the brightness of the ambient light, and the proximity sensor may turn off the display panel 1041 and / or the backlight when the mobile phone is moved to the ear. As a type of motion sensor, the accelerometer sensor can detect the magnitude of acceleration in all directions (generally three axes), and can detect the magnitude and direction of gravity when stationary. It can be used for applications that identify the posture of the mobile phone (such as horizontal and vertical screen switching, related games, magnetometer posture calibration), vibration recognition related functions (such as pedometer, tapping), etc.; as for other sensors that can be configured in the mobile phone, such as gyroscopes, barometers, hygrometers, thermometers, infrared sensors, etc., they will not be repeated here.

[0138] The audio circuit 1060, the speaker 1061, and the microphone 1062 can provide an audio interface between the user and the mobile phone. The audio circuit 1060 can transmit the received audio data to the speaker 1061 after converting the received audio data into an electrical signal, which is converted into a sound signal for output; on the other hand, the microphone 1062 converts the collected sound signal into an electrical signal, which is received by the audio circuit 1060 and converted into audio data, and then the audio data is output to the processor 1080 for processing, and then sent to another mobile phone through the RF circuit 1010, or the audio data is output to the memory 1020 for further processing.

[0139] WiFi is a short-range wireless transmission technology. The mobile phone can help users send and receive emails, browse web pages and access streaming media through the WiFi module 1070. It provides users with wireless broadband Internet access. Figure 3 A WiFi module 1070 is shown, but it is understandable that it is not an essential component of the mobile phone and can be omitted as needed without changing the essence of the invention.

[0140] The processor 1080 is the control center of the mobile phone. It uses various interfaces and lines to connect various parts of the entire mobile phone. By running or executing software programs and / or modules stored in the memory 1020 and calling data stored in the memory 1020, it executes various functions of the mobile phone and processes data, thereby monitoring the mobile phone as a whole. Optionally, the processor 1080 may include one or more processing units; preferably, the processor 1080 may integrate an application processor and a modem processor, wherein the application processor mainly processes the operating system, user interface and application programs, etc., and the modem processor mainly processes wireless communications. It is understandable that the above-mentioned modem processor may not be integrated into the processor 1080.

[0141] The mobile phone also includes a power supply 1090 (such as a battery) for supplying power to various components. Preferably, the power supply can be logically connected to the processor 1080 through a power management system, so that the power management system can manage functions such as charging, discharging, and power consumption.

[0142] Although not shown, the mobile phone may also include a camera, a Bluetooth module, etc., which will not be described in detail here.

[0143] In the embodiment of the present application, the processor 1080 included in the terminal also has the following functions:

[0144] Step S1: when the current time is not less than the end time of the processing operation and the pre-start time, determine whether to execute the pre-batch operation according to the unloading batch operation status;

[0145] It should be noted that the end time of the processing operation and the expected start time should be consistent when first determined, and the expected start time will be optimized and adjusted after being determined during the subsequent business peak processing period.

[0146] When the unloading and batching job is executed daily, it is necessary to check the status of the unloading and batching job. It is understandable that since the unloading and batching job is a T+1 time-limited job, the status of the unloading and batching job can be reset to the unexecuted state after 0:00 every day. After the unloading and batching job is completed, the status of the unloading and batching job is updated to the executed state.

[0147] Since the end time of the daily hive database processing operation is uncertain, it may end in the morning, noon, afternoon or evening. If the end time of the processing operation for the day is not detected at the current time, the pre-batch operation and the unloading batch operation will not be executed at all.

[0148] Step S2: If the batch unloading operation status is not executed, the pre-batch operation is executed, and it is determined in turn whether the current time is a peak business handling period and whether the current time is a batch running peak period;

[0149] It should be noted that if the status of the unloading and batch running job is not executed, the pre-batch running job will be entered. The pre-batch running job is a detection status, which makes a preliminary judgment on whether the unloading and batch running job can be entered. It mainly needs to determine whether the current time is the peak business processing period and the batch running peak period.

[0150] Step S3: If the current time is not the peak business processing period or the batch running peak period, the current concurrency number is set to the current concurrency number plus the preset value, and the batch unloading operation is performed based on the current concurrency number, and the current time is recorded as the actual start time;

[0151] It should be noted that when the current time is not the peak business processing period and the batch running peak period, it can be determined that the unloading and batch running job can be executed. At the same time, the current concurrency number is increased accordingly. When entering the unloading and batch running job execution, the efficiency of the unloading and batch running job can be improved.

[0152] During the initial determination, the current number of concurrent connections is the initial value, and when the unloading batch job status is updated daily, the current number of concurrent connections will be reset to the initial value. You can choose a suitable value for the initial value, such as 8.

[0153] The current concurrency number also has a minimum threshold and a maximum threshold.

[0154] Step S4: during the process of unloading and batching, determine in real time whether the current time is the peak business processing period and the batching peak period;

[0155] It should be noted that during the process of unloading and batching operations, it will also be determined in real time whether the current time is the peak business processing period or the peak batching period, in order to prevent the business peak processing period or the peak batching period from suddenly entering during the process of unloading and batching operations.

[0156] Step S5: If the current time is not the peak business processing period or the batch running peak period, set the current concurrency number to the current concurrency number plus the preset value until the batch unloading operation is completed, update the expected start time to the actual start time, and the batch unloading operation status is executed.

[0157] It should be noted that, only when the current time is not the peak business processing period and the batch running peak period, the current concurrency number will continue to increase based on the pre-batch running job judgment until the unloading batch running job is completed.

[0158] Furthermore, step S3 further includes:

[0159] If the current time is a peak business processing period, the estimated start time is updated so that the estimated start time increases by a preset time, the pre-batch operation is terminated, and the process returns to step S1.

[0160] Furthermore, step S3 further includes:

[0161] If the current time is the batch running peak period, the current concurrency number is updated to the current concurrency number minus the preset value, and the pre-batch running job is terminated, and the execution returns to step S1.

[0162] Furthermore, step S5 further includes:

[0163] If the current time is during the peak business hours, the estimated start time is updated so that the estimated start time increases by a preset time, and then the batch unloading operation is terminated and the process returns to step S1.

[0164] Furthermore, step S5 further includes:

[0165] If the current time is the peak period for batch running, the current concurrent number is updated to the current concurrent number minus the preset value, and the batch running operation is terminated, returning to step S4.

[0166] Further, whether the current time is a peak business processing period is determined as follows:

[0167] Get real-time traffic data from the DB application front end;

[0168] Compare the real-time traffic data with the traffic threshold to determine whether the current time is the peak business processing period.

[0169] Further, whether the current time is a peak business processing period is determined as follows:

[0170] Obtain target indicator data for databases and application servers;

[0171] Compare the target indicator data of the database and application server with the corresponding indicator threshold data to determine whether the current time is the peak business processing period.

[0172] It should be noted that the peak business processing period can be determined by the DB application front-end page traffic, such as the number of page visits, page visit duration, and active number.

[0173] You can also use database and server monitoring indicators to make judgments, such as qps, tps, IOPS, etc.

[0174] In one embodiment, the peak business period can be simply determined when the application server's cup usage reaches 50% of the previous day's peak, or the database's real-time qps value reaches 50% of the previous day's peak, or the database's real-time tps value reaches 50% of the previous day's peak.

[0175] Furthermore, the peak period of batch operation can be determined based on the IOPS value of the database. For example, the real-time IOPS value of the database reaches 50% of the peak value of the previous day.

[0176] In addition, if you encounter abnormal situations during the batch running process, such as excessive database load or system failure, you can configure the corresponding exception handling and recovery mechanism.

[0177] The embodiment of the present application also provides a computer-readable storage medium for storing program code, wherein the program code is used to execute any one of the implementation methods of the method for processing concurrent batch operations of unloading data described in the aforementioned embodiments.

[0178] In the embodiments of the present application, a method, device, equipment and medium for processing the concurrent number of batch unloading operations are provided, by performing pre-batch operation and batch unloading operation judgments on the batch unloading operations respectively, wherein by judging whether the current time is a business peak processing period and a batch running peak period, the pre-start time of the pre-batch operation and the current concurrent number of the batch unloading operations are optimized and adjusted, so as to avoid the batch unloading operations being processed during the business peak processing period, and adjust the current concurrent number according to the batch running peak period, increase the current concurrent number during non-batch running peak periods, and reduce the current concurrent number during batch running peak periods, thereby solving the technical problem that if the batch unloading operations are triggered and executed in large quantities during the peak period of customer business processing, and the fixed concurrent number value is large, database resource competition will be caused, affecting customer business processing and reducing customer experience.

[0179] Those skilled in the art can clearly understand that, for the convenience and brevity of description, the specific working processes of the systems, devices and units described above can refer to the corresponding processes in the aforementioned method embodiments and will not be repeated here.

[0180] The terms "first", "second", "third", "fourth", etc. (if any) in the specification of the present application and the above-mentioned drawings are used to distinguish similar objects, and are not necessarily used to describe a specific order or sequence. It should be understood that the data used in this way can be interchangeable where appropriate, so that the embodiments of the present application described herein can be implemented in an order other than those illustrated or described herein, for example. In addition, the terms "including" and "having" and any of their variations are intended to cover non-exclusive inclusions, for example, a process, method, system, product or device comprising a series of steps or units is not necessarily limited to those steps or units clearly listed, but may include other steps or units that are not clearly listed or inherent to these processes, methods, products or devices.

[0181] It should be understood that in the present application, "at least one (item)" means one or more, and "plurality" means two or more. "And / or" is used to describe the association relationship of associated objects, indicating that three relationships may exist. For example, "A and / or B" can mean: only A exists, only B exists, and A and B exist at the same time, where A and B can be singular or plural. The character " / " generally indicates that the objects associated before and after are in an "or" relationship. "At least one of the following" or similar expressions refers to any combination of these items, including any combination of single or plural items. For example, at least one of a, b or c can mean: a, b, c, "a and b", "a and c", "b and c", or "a and b and c", where a, b, c can be single or multiple.

[0182] In the several embodiments provided in the present application, it should be understood that the disclosed systems, devices and methods can be implemented in other ways. For example, the device embodiments described above are only schematic. For example, the division of the units is only a logical function division. There may be other division methods in actual implementation, such as multiple units or components can be combined or integrated into another system, or some features can be ignored or not executed. Another point is that the mutual coupling or direct coupling or communication connection shown or discussed can be an indirect coupling or communication connection through some interfaces, devices or units, which can be electrical, mechanical or other forms.

[0183] The units described as separate components may or may not be physically separated, and the components shown as units may or may not be physical units, that is, they may be located in one place or distributed on multiple network units. Some or all of the units may be selected according to actual needs to achieve the purpose of the solution of this embodiment.

[0184] In addition, each functional unit in each embodiment of the present application may be integrated into one processing unit, or each unit may exist physically separately, or two or more units may be integrated into one unit. The above-mentioned integrated unit may be implemented in the form of hardware or in the form of software functional units.

[0185] If the integrated unit is implemented in the form of a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present application is essentially or the part that contributes to the prior art or all or part of the technical solution can be embodied in the form of a software product, and the computer software product is stored in a storage medium, including a number of instructions to enable a computer device (which can be a personal computer, a server, or a network device, etc.) to perform all or part of the steps of the method described in each embodiment of the present application. The aforementioned storage medium includes: U disk, mobile hard disk, read-only memory (full name in English: Read-Only Memory, English abbreviation: ROM), random access memory (full name in English: Random Access Memory, English abbreviation: RAM), disk or optical disk and other media that can store program codes.

[0186] As described above, the above embodiments are only used to illustrate the technical solutions of the present application, rather than to limit them. Although the present application has been described in detail with reference to the aforementioned embodiments, those skilled in the art should understand that they can still modify the technical solutions described in the aforementioned embodiments, or make equivalent replacements for some of the technical features therein. However, these modifications or replacements do not deviate the essence of the corresponding technical solutions from the spirit and scope of the technical solutions of the embodiments of the present application.

Claims

1. A method for processing concurrent data in batch unloading operations, characterized in that: include: Step S1: when the current time is not less than the end time of the processing operation and the pre-start time, determine whether to execute the pre-batch operation according to the unloading batch operation status; Step S2: If the state of the batch unloading operation is not executed, the pre-batch operation is executed, and it is determined in turn whether the current time is a peak business handling period and whether the current time is a batch running peak period; Step S3: If the current time is not a business peak processing period or a batch running peak period, the current concurrency number is set to the current concurrency number plus a preset value, and the batch running operation is performed based on the current concurrency number, and the current time is recorded as the actual start time; Step S4: during the process of unloading and batching, determine in real time whether the current time is the peak business processing period and the batching peak period; Step S5: If the current time is not the peak business processing period and the batch running peak period, set the current concurrency number to the current concurrency number plus the preset value until the batch unloading operation is completed, update the expected start time to the actual start time, and the batch unloading operation status is the executed status.

2. The method for processing concurrent batch operations of unloading data according to claim 1 is characterized in that: The step S3 further comprises: If the current time is a peak business processing period, the pre-start time is updated so that the pre-start time is increased by a preset time, the pre-run batch job is terminated, and the process returns to step S1.

3. The method for processing concurrent batch operations of unloading data according to claim 1 is characterized in that: The step S3 further comprises: If the current time is the batch running peak period, the current concurrency number is updated to the current concurrency number minus the preset value, and the pre-batch running job is terminated, and the execution returns to step S1.

4. The method for processing concurrent batch operations of unloading data according to claim 1 is characterized in that: The step S5 further comprises: If the current time is a peak business period, the pre-start time is updated so that the pre-start time is increased by a preset time, the batch unloading operation is terminated, and the process returns to step S1.

5. The method for processing concurrent data of batch unloading operations according to claim 1, characterized in that: The step S5 further comprises: If the current time is the peak period for batch running, the current concurrent number is updated to the current concurrent number minus the preset value, and the batch running operation is terminated, returning to step S4.

6. The method for processing concurrent batch operations of unloading data according to claim 1, characterized in that: The specific method of determining whether the current time is a peak business handling period is as follows: Get real-time traffic data from the DB application front end; The real-time traffic data is compared with the traffic threshold to determine whether the current time is a peak business processing period.

7. The method for processing concurrent data of batch unloading operations according to claim 1, characterized in that: The specific method of determining whether the current time is a peak business handling period is as follows: Obtain target indicator data for databases and application servers; The target indicator data of the database and the application server are respectively compared with the corresponding indicator threshold data to determine whether the current time is a business peak processing period.

8. A concurrent data processing device for batch unloading operations, characterized in that: include: The pre-batch run judgment unit is used to judge whether to execute the pre-batch run operation according to the unloading batch run operation status when the current time is not less than the end time of the processing operation and the pre-start time; The first processing unit is used to execute the pre-batch operation if the state of the unloading batch operation is not executed, and to determine whether the current time is a business peak handling period and whether the current time is a batch running peak period in turn; The second processing unit is used to set the current concurrency number to the current concurrency number plus a preset value if the current time is not a business peak processing period and a batch running peak period, and perform the batch unloading operation based on the current concurrency number, and record the current time as the actual start time; The data unloading and batch running judgment unit is used to judge in real time whether the current time is the peak business processing period and the batch running peak period during the data unloading and batch running operation; The third processing unit is used to set the current concurrency number to the current concurrency number plus the preset value if the current time is not a business peak processing period and a batch running peak period, until the unloading and batch running job is completed, update the expected start time to the actual start time, and the unloading and batch running job status is the executed status.

9. A data unloading batch operation concurrent data processing device, characterized in that: The device comprises a processor and a memory: The memory is used to store program code and transmit the program code to the processor; The processor is used to execute the method for concurrently processing batch jobs of unloading data according to the instructions in the program code as described in any one of claims 1-7.

10. A computer-readable storage medium, characterized in that: The computer-readable storage medium is used to store program codes, and the program codes are used to execute the method for concurrently processing batch jobs of unloading data as described in any one of claims 1-7.