Mass data processing method and device, electronic equipment and storage medium

Abstract

The embodiment of the invention provides a mass data processing method and device, electronic equipment and a storage medium. The method comprises the following steps: acquiring a plurality of pieces of first data information to be processed and an identification number corresponding to each piece of first data information; according to a preset queue state in a blocking queue, sequentially adding identification numbers corresponding to the first data information into the blocking queue, and updating the queue state; according to the queue sequence of the blocking queues, pushing identification numbers to message queues of preset message middleware in sequence, and updating the blocking queues and queue states; converting the first data information corresponding to the identification number into second data information according to the identification number in the message queue; and pushing the plurality of pieces of second data information and the corresponding identification numbers to a task thread pool for task allocation and processing. According to the embodiment of the invention, task division can be coordinated through the message middleware, parallel processing of mass data is realized, the processing efficiency is improved, and the method is simple to use and convenient to expand.

Description

Technical Field

[0001] The present application relates to the field of data processing technology, and in particular to a method and device for processing massive data, an electronic device and a storage medium. Background Art

[0002] In today's digital age, the amount of data in enterprises, especially in the financial sector, is exploding. Take banks as an example. Every day, a large amount of employee and customer outbound call record data is generated. This data is crucial for the bank's business analysis and decision-making. However, when processing this massive amount of data, there are many technical problems.

[0003] One of the main technical problems is that the traditional single-machine multi-threaded concurrent processing method is often unable to process all data within the specified time frame when processing large amounts of data, because the business process of processing a piece of data takes a long time and there is a bottleneck limit to the processing capacity of a single machine. Although it is possible to achieve multi-host parallel operation of data processing services and improve the overall processing speed by building a distributed scheduled task scheduling platform, this method has some disadvantages, such as the need for additional development and deployment costs, high requirements for platform disaster recovery capabilities and the possibility of load imbalance, and the need for customized development of sharding algorithms that are not universal. Summary of the invention

[0004] The main purpose of the embodiments of the present application is to propose a method and device for processing massive data, an electronic device and a storage medium, aiming to coordinate task division through message middleware, realize parallel processing of massive data, improve processing efficiency, and be simple to use and easy to expand.

[0005] To achieve the above-mentioned purpose, a first aspect of an embodiment of the present application proposes a method for processing massive data, the method comprising:

[0006] Acquire a plurality of first data information to be processed and an identification number corresponding to each of the first data information;

[0007] According to the queue state in the preset blocking queue, the identification numbers corresponding to the first data information are sequentially added to the blocking queue, and the queue state is updated;

[0008] According to the queue order of the blocking queue, the identification number is pushed to the message queue of the preset message middleware in sequence, and the blocking queue and the queue state are updated;

[0009] According to the identification number in the message queue, converting the first data information corresponding to the identification number into second data information;

[0010] Pushing a plurality of the second data information and the corresponding identification numbers to a task thread pool for task allocation and processing; wherein the task thread pool includes a plurality of task processing threads, and each task processing thread is responsible for processing one piece of the second data information.

[0011] In some embodiments, the adding the identification numbers corresponding to the first data information to the blocking queue in sequence and updating the queue state includes:

[0012] Add the identification number to a preset blocking queue, and update the task cumulative value corresponding to the blocking queue;

[0013] When the task cumulative value is less than a preset full load threshold, the queue state of the blocking queue is updated to a smooth state;

[0014] When the task cumulative value is greater than or equal to the full load threshold, the queue state of the blocking queue is updated to a busy state.

[0015] In some embodiments, according to a queue state in a preset blocking queue, sequentially adding the identification number corresponding to the first data information to the blocking queue and updating the queue state includes:

[0016] When the queue state is a smooth state, adding the identification number to the blocked queue and updating the queue state;

[0017] When the queue state is a busy state, stop adding the identification number to the blocking queue until the queue state is updated to a smooth state.

[0018] In some embodiments, according to a queue state in a preset blocking queue, sequentially adding the identification number corresponding to the first data information to the blocking queue and updating the queue state includes:

[0019] According to the change of the message queue, the processing speed of the task thread pool is obtained;

[0020] According to the processing speed, adjusting the queue entry speed of adding the identification number to the blocking queue;

[0021] According to the queue state in the preset blocking queue and the queue entry speed, the identification number corresponding to the first data information is added to the blocking queue in sequence, and the queue state is updated.

[0022] In some embodiments, after the step of pushing the plurality of second data information to the task thread pool for task allocation, the method further includes:

[0023] When the task processing thread completes data processing of the current second data information, a completion feedback signal is generated; wherein the completion feedback signal includes the identification number corresponding to the current second data information;

[0024] In response to the completion feedback signal, the corresponding identification number in the message queue is deleted.

[0025] In some embodiments, after the step of pushing the plurality of second data information to the task thread pool for task allocation, the method further includes:

[0026] When the task processing thread fails to complete data processing of the current second data information, a retry feedback signal and retry count information are generated; wherein the retry feedback signal includes the identification number corresponding to the current second data information;

[0027] In response to the completion feedback signal, re-pushing the second data information corresponding to the identification number to the task thread pool for task allocation and processing;

[0028] When the number of retries is greater than or equal to a preset abnormal threshold, the task allocation for the second data information is stopped, and an abnormal identification is generated for the identification number.

[0029] In some embodiments, obtaining the processing speed of the task thread pool according to the change of the message queue includes:

[0030] Marking the identification numbers corresponding to the first data information as test marking numbers;

[0031] Recording the entry timestamp of the test identification number into the message queue and the exit timestamp of the test identification number in the message queue;

[0032] Obtaining a first processing time of the first data information corresponding to the test mark number according to the queue entry timestamp and the queue exit timestamp;

[0033] Obtaining an average processing time according to the first processing times corresponding to all the test mark numbers;

[0034] The processing speed of the task thread pool is obtained according to the average processing time.

[0035] To achieve the above-mentioned purpose, a second aspect of an embodiment of the present application provides a massive data processing device, the device comprising:

[0036] An acquisition module, used for acquiring a plurality of first data information to be processed and an identification number corresponding to each of the first data information;

[0037] A first push module, configured to sequentially add the identification numbers corresponding to the first data information to the blocking queue according to a queue state in a preset blocking queue, and update the queue state;

[0038] A second push module is used to push the identification number to the message queue of the preset message middleware in sequence according to the queue order of the blocking queue, and update the blocking queue and the queue state;

[0039] a conversion module, configured to convert the first data information corresponding to the identification number into second data information according to the identification number in the message queue;

[0040] An allocation and processing module is used to push a number of the second data information and the corresponding identification numbers to a task thread pool for task allocation and processing; wherein the task thread pool includes a number of task processing threads, and each task processing thread is responsible for processing one second data information.

[0041] To achieve the above objectives, a third aspect of an embodiment of the present application proposes an electronic device, which includes a memory and a processor, wherein the memory stores a computer program, and when the processor executes the computer program, the method described in the first aspect is implemented.

[0042] To achieve the above-mentioned purpose, the fourth aspect of an embodiment of the present application proposes a computer-readable storage medium, wherein the computer-readable storage medium stores a computer program, and when the computer program is executed by a processor, the method described in the first aspect is implemented.

[0043] The method and device for processing massive data, the electronic device and the storage medium proposed in the present application obtain several first data information to be processed and the identification number corresponding to each first data information. According to the queue state in the preset blocking queue, the identification number corresponding to the first data information is sequentially added to the blocking queue, and the queue state is updated, so that the data can be ensured to enter the processing flow in an orderly manner to avoid confusion. According to the queue order of the blocking queue, the identification number is sequentially pushed to the message queue of the preset message middleware, and the blocking queue and the queue state are updated. The message middleware can play a role of buffering and coordination to ensure the reliable transmission of data. According to the identification number in the message queue, the first data information corresponding to the identification number is converted into the second data information. This conversion process can be carried out according to specific business needs, such as format conversion, cleaning and other operations on the data. Several second data information and corresponding identification numbers are pushed to the task thread pool for task allocation and processing. The task thread pool includes several task processing threads, each task processing thread is responsible for processing one second data information, thereby realizing parallel processing, improving processing efficiency, and allocating data processing tasks to multiple task processing threads to realize parallel processing. At the same time, the use of message middleware can coordinate the division of tasks in a multi-host environment, avoiding the bottleneck of single-machine processing. When the amount of data increases, the processing capacity can be improved by increasing the number of task processing threads or hosts, thereby effectively solving the problem of insufficient single-machine capacity. Furthermore, this method avoids the use of a complex distributed timed task scheduling platform. Through the combination of blocking queues and message middleware, the orderly transmission and processing of data is achieved without the need to build an additional complex platform. The implementation of this method is relatively simple, easy to maintain and expand, and reduces the cost of development and deployment. BRIEF DESCRIPTION OF THE DRAWINGS

[0044] Figure 1 is a flow chart of a method for processing massive data provided by an embodiment of the present application;

[0045] Figure 2 yes Figure 1 Flow chart of step S120 in FIG.

[0046] Figure 3 yes Figure 1 Flow chart of step S120 in FIG.

[0047] Figure 4 yes Figure 1 Flow chart of step S120 in FIG.

[0048] Figure 5 yes Figure 1 The flowchart after step S150 in FIG.

[0049] Figure 6 yes Figure 1The flowchart after step S150 in FIG.

[0050] Figure 7 yes Figure 4 Flow chart of step S410 in FIG.

[0051] Figure 8 is a structural schematic diagram of a mass data processing device provided in an embodiment of the present application;

[0052] Fig. 9 It is a schematic diagram of the hardware structure of the electronic device provided in the embodiment of the present application. DETAILED DESCRIPTION

[0053] In order to make the purpose, technical solution and advantages of the present application more clearly understood, the present application is further described in detail below in conjunction with the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are only used to explain the present application and are not used to limit the present application.

[0054] It should be noted that, although the functional modules are divided in the device schematic diagram and the logical order is shown in the flowchart, in some cases, the steps shown or described may be performed in a different order than the module division in the device or the order in the flowchart. The terms "first", "second", etc. in the specification, claims and the above drawings are used to distinguish similar objects, and are not necessarily used to describe a specific order or sequence.

[0055] Unless otherwise defined, all technical and scientific terms used herein have the same meaning as those commonly understood by those skilled in the art to which this application belongs. The terms used herein are only for the purpose of describing the embodiments of this application and are not intended to limit this application.

[0056] First, some nouns involved in this application are analyzed:

[0057] Message-Oriented Middleware: It is a software that uses efficient and reliable message delivery mechanism to exchange data independent of the platform and integrate distributed systems based on data communication. It can pass messages between different applications, allowing these applications to communicate and collaborate in a loosely coupled situation.

[0058] Message Queue (MQ): is a queue data structure that stores messages and is used to transfer messages between different application components or different systems. It is the core component of message middleware and provides an asynchronous and loosely coupled communication method.

[0059] Task thread pool: is a mechanism for managing and reusing threads. It mainly consists of a group of pre-created threads that can be used to perform various tasks. The thread pool maintains a task queue to store tasks waiting to be executed. When a task needs to be executed, the thread pool will take the task from the task queue and assign it to an idle thread for execution.

[0060] In today's digital age, the amount of data in enterprises, especially in the financial sector, is exploding. Take banks as an example. Every day, a large amount of employee and customer outbound call record data is generated. This data is crucial for the bank's business analysis and decision-making. However, when processing this massive amount of data, there are many technical problems.

[0061] One of the main technical problems is that the traditional single-machine multi-threaded concurrent processing method is often unable to process all data within the specified time frame when processing large amounts of data, because the business process of processing a piece of data takes a long time and there is a bottleneck limit to the processing capacity of a single machine. Although it is possible to achieve multi-host parallel operation of data processing services and improve the overall processing speed by building a distributed scheduled task scheduling platform, this method has some disadvantages, such as the need for additional development and deployment costs, high requirements for platform disaster recovery capabilities and the possibility of load imbalance, and the need for customized development of sharding algorithms that are not universal.

[0062] Specifically, in an application scenario: a bank generates a large amount of outbound call record data of employees and customers every day. These data need to be further processed by a scheduled task at 2 o'clock every night (such as downloading outbound call recording files to the business system, judging whether to generate bank employee activity indicator data based on the outbound call duration, etc.), and all data records are required to be processed before 8 o'clock in the morning. The common solutions for using single-machine multi-threaded concurrent processing of these data are as follows:

[0063] (1) Data processing task dispatcher: executes data query SQL, queries all records with the status of "unprocessed", traverses each record, creates a data processing task for it, and puts it into the data processing thread pool. If all thread pools in the data processing thread pool are busy at this time, the data processing task dispatcher temporarily sleeps and waits until an idle thread is available to continue developing and distributing tasks.

[0064] (2) The data processing thread pool obtains the distributed data processing tasks, runs the business processing flow, and modifies the status of the corresponding record in the database to "processed" after the processing is completed.

[0065] Due to the large amount of data to be processed, the business process of processing a piece of data takes a long time, and the bottleneck limit of the processing capacity of a single machine, these factors will make it impossible to process all the data within the specified time frame. Therefore, when the existing processing capacity is insufficient, it is necessary to add new machines on demand to run data processing services in parallel to improve the overall processing speed. In an environment where multiple hosts run in parallel, it is necessary to ensure that each host coordinates the division of tasks to avoid processing the same data at the same time and achieve load balancing.

[0066] A common solution is to build an additional distributed scheduled task scheduling platform, which detects all data processing service hosts registered to the platform and in operation, and initiates scheduled tasks on multiple hosts at the same time, passing the shard sequence number and total number of shards in the task. When each host runs scheduled tasks in parallel, it adopts appropriate algorithms based on their respective shard sequence numbers and total number of shards to process massive data in parallel, thereby improving the overall processing speed.

[0067] The disadvantages of this solution are as follows:

[0068] (1) Building a distributed scheduled task calling platform requires additional development and deployment costs.

[0069] (2) The platform needs to have disaster recovery capabilities. When a host that is executing a task goes offline abnormally, its task needs to be transferred to other machines. At this time, some hosts may have more tasks than other hosts, resulting in an unbalanced workload.

[0070] (3) Sharding algorithms need to be customized and developed according to the actual data situation and are not universal. If there is no column for increasing numbers, the algorithm of "number% total number of shards = shard number - 1" cannot be used. Other sharding algorithms need to be considered, and the load balancing of the algorithm must be considered to evenly distribute tasks to each host to avoid overload of individual hosts leading to task backlogs.

[0071] Based on this, the embodiments of the present application provide a method and device for processing massive data, an electronic device and a storage medium, aiming to coordinate task division through message middleware, realize parallel processing of massive data, improve processing efficiency, and be simple to use and easy to expand.

[0072] The embodiments of the present application can acquire and process relevant data based on artificial intelligence technology. Among them, artificial intelligence (AI) is the theory, method, technology and application system that uses digital computers or machines controlled by digital computers to simulate, extend and expand human intelligence, perceive the environment, acquire knowledge and use knowledge to obtain the best results.

[0073] AI basic technologies generally include sensors, dedicated AI chips, cloud computing, distributed storage, big data processing technology, operation / interaction systems, mechatronics, etc. AI software technologies mainly include computer vision technology, robotics technology, biometrics technology, speech processing technology, natural language processing technology, and machine learning / deep learning.

[0074] The present application can be used in many general or special computer system environments or configurations. For example: personal computers, server computers, handheld or portable devices, tablet devices, multiprocessor systems, microprocessor-based systems, set-top boxes, programmable consumer electronics, network PCs, minicomputers, mainframe computers, distributed computing environments including any of the above systems or devices, etc. The present application can be described in the general context of computer-executable instructions executed by a computer, such as program modules. Generally, program modules include routines, programs, objects, components, data structures, etc. that perform specific tasks or implement specific abstract data types. The present application can also be practiced in distributed computing environments, in which tasks are performed by remote processing devices connected through a communication network. In a distributed computing environment, program modules can be located in local and remote computer storage media including storage devices.

[0075] It should be noted that in each specific implementation of the present application, when it comes to the need to perform relevant processing based on data related to user identity or characteristics such as user information, user behavior data, user historical data, and user location information, the user's permission or consent will be obtained first, and the collection, use, and processing of these data will comply with the relevant laws, regulations, and standards of the relevant countries and regions. In addition, when the embodiment of the present application needs to obtain the user's sensitive personal information, the user's separate permission or consent will be obtained through a pop-up window or by jumping to a confirmation page. After clearly obtaining the user's separate permission or consent, the necessary user-related data for the normal operation of the embodiment of the present application will be obtained.

[0076] Figure 1 is an optional flow chart of the method for processing massive data provided by an embodiment of the present application. Figure 1 The method may include but is not limited to steps S110 to S150.

[0077] Step S110, obtaining a plurality of first data information to be processed and an identification number corresponding to each first data information;

[0078] Step S120, according to the queue state in the preset blocking queue, the identification number corresponding to the first data information is sequentially added to the blocking queue, and the queue state is updated;

[0079] Step S130, pushing identification numbers to the message queues of the preset message middleware in sequence according to the queue order of the blocking queues, and updating the blocking queues and queue states;

[0080] Step S140, according to the identification number in the message queue, converting the first data information corresponding to the identification number into the second data information;

[0081] Step S150, pushing a number of second data information and corresponding identification numbers to a task thread pool for task allocation and processing; wherein the task thread pool includes a number of task processing threads, and each task processing thread is responsible for processing a second data information.

[0082] Steps S110 to S160 shown in the embodiment of the present application are obtained by obtaining several first data information to be processed and the identification number corresponding to each first data information. According to the queue state in the preset blocking queue, the identification number corresponding to the first data information is added to the blocking queue in sequence, and the queue state is updated, so that the data can be ensured to enter the processing flow in an orderly manner to avoid confusion. According to the queue order of the blocking queue, the identification number is pushed to the message queue of the preset message middleware in sequence, and the blocking queue and the queue state are updated. The message middleware can play a role of buffering and coordination to ensure the reliable transmission of data. According to the identification number in the message queue, the first data information corresponding to the identification number is converted into the second data information. This conversion process can be carried out according to specific business needs, such as format conversion, cleaning and other operations on the data. Several second data information and corresponding identification numbers are pushed to the task thread pool for task allocation and processing. The task thread pool includes several task processing threads, each task processing thread is responsible for processing a second data information, thereby realizing parallel processing, improving processing efficiency, and allocating data processing tasks to multiple task processing threads to realize parallel processing. At the same time, the use of message middleware can coordinate the division of tasks in a multi-host environment, avoiding the bottleneck of single-machine processing. When the amount of data increases, the processing capacity can be improved by increasing the number of task processing threads or hosts, thereby effectively solving the problem of insufficient single-machine capacity. Furthermore, this method avoids the use of a complex distributed timed task scheduling platform. Through the combination of blocking queues and message middleware, the orderly transmission and processing of data is achieved without the need to build an additional complex platform. The implementation of this method is relatively simple, easy to maintain and expand, and reduces the cost of development and deployment.

[0083] In step S110 of some embodiments, the first data information can be obtained by querying the database SQL from the preset. In the database, the query conditions are set according to business needs and data characteristics, such as filtering according to the "pending" status, time, etc. It is also possible to obtain data from an external system by establishing a network connection or calling through an API, and doing authentication and authorization. Or directly use the corresponding tool to open it according to its storage path and format. This is not limited in this application.

[0084] In step S140 of some embodiments, after the first data information (record information to be processed) is converted into the second data information (MQ message), the coupling between the data producer (the part that sends data) and the data consumer (the part that processes data) is reduced. Before the conversion, the data producer may need to directly pass the unprocessed record information to the consumer in a certain format, and the consumer also needs to rely closely on the implementation details of the producer. By converting to MQ messages, as long as the communication protocol of the message middleware is followed, the producer and the consumer can be modified and expanded independently. For example, if the logic of data processing needs to be updated or new consumer services need to be added in the banking system, as long as the message format remains unchanged, it will not have much impact on the generation and transmission of data. For example, in the bank's outbound call record processing scenario, the speed of data generation (such as the generation of outbound call records) may be very fast, while data processing (such as downloading outbound call recording files, generating employee activity indicator data, etc.) may be time-consuming. By converting to MQ messages, the outbound call record data can be sent to the message queue first, allowing the downstream processing service to process at its own pace, thereby improving the overall throughput of the system.

[0085] In step S150 of some embodiments, it can be processed by a single machine or multiple machines. In the data processing of multiple machines, a centralized task thread pool management method can be adopted, that is, all machines share a logical task thread pool. For example, a central scheduler is needed to manage the distribution of tasks. It must clearly know the load of each machine, the number of idle threads in the task thread pool, and other information, and then accurately distribute the second data information and identification number to the appropriate thread for processing. It is also possible that each machine can be configured with its own independent task thread pool, and the second data information and the corresponding identification number can be distributed to the task thread pools of different machines for processing according to a certain strategy.

[0086] See also Figure 2 It is understandable that step S120 may include but is not limited to the following steps:

[0087] Step S210: adding the identification number to a preset blocking queue, and updating the task cumulative value corresponding to the blocking queue;

[0088] Step S220: When the task cumulative value is less than the preset full load threshold, the queue state of the blocked queue is updated to a smooth state;

[0089] Step S230: When the task cumulative value is greater than or equal to the full load threshold, the queue state of the blocking queue is updated to a busy state.

[0090] In step S210 of some embodiments, the identification number corresponding to the first data information is added to a preset blocking queue. This blocking queue can be regarded as a task waiting area for orderly storing the task identification numbers to be processed. Each time an identification number is added, the task cumulative value corresponding to the blocking queue is updated. The task cumulative value is a counter for recording the number of task identification numbers stored in the current blocking queue. For example, after an identification number is added, the task cumulative value is increased by 1.

[0091] In steps S220 to S230 of some embodiments, the task cumulative value is compared with a preset full load threshold. The full load threshold is a preset value used to measure the busyness of the blocking queue. If the task cumulative value is less than the full load threshold, the queue state of the blocking queue is updated to a smooth state. This means that the blocking queue currently has enough space to receive new task identification numbers, the processing progress of the task is relatively ideal, and there is no task backlog. On the contrary, if the task cumulative value is greater than or equal to the full load threshold, the queue state of the blocking queue is updated to a busy state. This means that the number of tasks in the blocking queue is already large, and may be close to or reach the limit of its processing capacity, and the new task identification number may need to wait for a period of time before it can be processed. By setting the full load threshold and updating the queue state, the task flow entering the blocking queue can be effectively controlled. When the queue is in a busy state, corresponding measures can be taken, such as pausing or slowing down the addition of new tasks, to avoid excessive backlog of tasks in the queue, resulting in exhaustion of system resources. This helps to make rational use of system resources and ensure that data processing tasks can be carried out in an orderly manner within the tolerance range of the system. On the other hand, the update of queue status provides intuitive information for system monitoring. Administrators can timely understand the progress and pressure of task processing by observing the queue status. For example, when the queue is busy for a long time, it may be a signal that the system may need to increase processing resources (such as increasing the number of threads, expanding servers, etc.), or optimize the data processing process to improve the processing efficiency of the system.

[0092] Correspondingly, in step S130, after pushing the identification number to the message queue of the preset message middleware in sequence according to the queue order of the blocking queue, the blocking queue and the queue status will be updated. Specifically, the corresponding identification number in the blocking queue will be deleted, and the task cumulative value will be reduced by 1. Then, according to the task cumulative value, the queue status corresponding to the blocking queue will be updated through steps S220 to S230.

[0093] Specifically, in step S210 to step S230, appropriate adjustments can be made according to the increase or decrease in the number of task processing hosts. Assuming that one task processing host can process 50 tasks in parallel, and 10 task processing hosts as a whole can process 500 tasks in parallel, the blocking queue is usually set to twice the overall parallel processing capacity, and the full load threshold can be set to 1000. When the first order ID is added, the cumulative value of the task changes from 0 to 1, and the queue state of the blocking queue is still in a smooth state. When order IDs are continuously added, assuming that when it is added to the 500th, the cumulative value of the task becomes 500, and the queue state is still in a smooth state. When it is added to the 1000th order ID, the cumulative value of the task reaches 1000, which is equal to the full load threshold of 1000. At this time, the queue state of the blocking queue is updated to a busy state. After taking an order ID from the blocking queue and pushing it to the message queue, the cumulative value of the task in the blocking queue is reduced by 1 and becomes 999. At this time, it is necessary to judge the relationship between the cumulative value of the task and the full load threshold again. When the cumulative value of the task is less than the full load threshold again, the queue state of the blocking queue is updated back to the smooth state.

[0094] In addition, a smooth threshold can be preset, which can be set to 900. When the blocked queue enters the busy state, it is necessary to wait until the task accumulation value returns to 900 before updating the queue state of the blocked queue back to the smooth state. During the data processing process, there may be a short task peak, which causes the task accumulation value to just fall below the full load threshold and then rise rapidly. If the task accumulation value is only slightly below the full load threshold and is updated to the smooth state, the administrator may misjudge the system's processing capacity and increase the inflow of new tasks too early. Setting a smooth threshold can avoid such frequent state switching, because the queue state will only be updated to the smooth state when the task accumulation value drops to a relatively low level. This can more accurately reflect the actual processing capacity and task pressure of the queue, reduce such misjudgments, and make system monitoring more stable and effective.

[0095] See also Figure 3 It is understandable that step S120 may include but is not limited to the following steps:

[0096] Step S310: When the queue state is smooth, add the identification number to the blocked queue and update the queue state;

[0097] Step S320: When the queue state is busy, stop adding identification numbers to the blocked queue until the queue state is updated to a smooth state.

[0098] In steps S310 to S320 of some embodiments, when the system is preparing to add the identification number corresponding to the first data information to the blocking queue, it will first check the current queue state of the blocking queue. By deciding whether to add the identification number according to the queue state, the blocking queue can be effectively prevented from being overloaded. Stopping the addition of new identification numbers when the queue is busy can prevent excessive backlog of tasks, thereby avoiding excessive consumption of system resources due to too many tasks waiting to be processed. Through the above-mentioned method steps, it is ensured that the system operates within a reasonable resource range, and the overall performance and stability of the system are improved. For example, when processing a large amount of data, if tasks are added to the blocking queue without control, memory overflow or slow system response may occur, and this mechanism can avoid this situation.

[0099] See also Figure 4 It is understandable that step S120 may include but is not limited to the following steps:

[0100] Step S410: Obtaining the processing speed of the task thread pool according to the change of the message queue;

[0101] Step S420: adjusting the queue entry speed of adding the identification number to the blocking queue according to the processing speed;

[0102] Step S430: According to the queue state in the preset blocking queue and the queue entry speed, the identification number corresponding to the first data information is added to the blocking queue in sequence, and the queue state is updated.

[0103] In steps S410 to S430 of some embodiments, the system infers the processing speed of the task thread pool by monitoring the changes in the message queue. According to the obtained processing speed of the task thread pool, the system adjusts the entry speed of adding the identification number to the blocking queue. If the processing speed is fast, it means that the task thread pool has a high processing capacity and the entry speed can be appropriately increased; conversely, if the processing speed is slow, it may be necessary to reduce the entry speed to avoid excessive backlog of tasks in the blocking queue. Thus, according to the adjusted entry speed and the current queue state of the blocking queue, the identification number corresponding to the first data information is added to the blocking queue in sequence.

[0104] In the above steps, the queue entry speed can be dynamically adjusted according to the actual processing capacity of the task thread pool, realizing the dynamic load balancing of the system. When processing massive data, the load of the system may be different in different time periods. In this way, the number of tasks in the blocking queue can be matched with the processing capacity of the task thread pool. For example, during business peaks, the processing speed of the task thread pool may decrease. At this time, reducing the queue entry speed can avoid excessive task accumulation; while during business troughs, increasing the queue entry speed can make full use of system resources and improve data processing efficiency. Therefore, the queue entry speed is dynamically adjusted according to the processing situation of the task thread pool to ensure the timeliness of data processing and avoid the problem of task backlog caused by too fast queue entry speed, thereby delaying data processing. At the same time, the stability of the system is also guaranteed, because a reasonable queue entry speed can prevent the system from crashing or abnormal due to task overload.

[0105] See also Figure 5 It is understandable that after step S150, the following steps may also be included but not limited to:

[0106] Step S510: When the task processing thread completes data processing of the current second data information, a completion feedback signal is generated; wherein the completion feedback signal includes an identification number corresponding to the current second data information;

[0107] Step S520: In response to the completion feedback signal, the corresponding identification number in the message queue is deleted.

[0108] In step S510 to step S520 of some embodiments, when the task processing thread completes the data processing of the current second data information, a completion feedback signal is generated, and the feedback signal contains the identification number corresponding to the second data information that has just been processed. For example, in a bank outbound call record processing system, the task processing thread completes the processing of a certain outbound call record, such as completing the download of the outbound call recording file and the generation of employee activity index data, and a completion feedback signal with the identification number of the outbound call record is created. After the system receives this completion feedback signal, it will search and delete the corresponding identification number in the message queue according to the identification number contained in the signal. By generating a completion feedback signal and deleting the identification number in the message queue, the system can accurately track the processing progress of each data, and can ensure that each second data information is correctly processed once, avoiding repeated processing or missing processing. When processing financial transaction data, accurately tracking the processing status of each transaction data ensures the integrity and accuracy of the transaction data and prevents financial risks caused by data processing errors. At the same time, timely deleting the processed identification numbers can effectively utilize the space resources of the message queue. The message queue will not take up too much space due to storing a large number of data identification numbers that have been processed, and can accommodate more new data identification numbers to be processed.

[0109] See also Figure 6 It is understandable that after step S150, the following steps may also be included but not limited to:

[0110] Step S610: when the task processing thread fails to complete data processing of the current second data information, a retry feedback signal and retry count information are generated; wherein the retry feedback signal includes an identification number corresponding to the current second data information;

[0111] Step S620: in response to the completion feedback signal, the second data information corresponding to the identification number is pushed to the task thread pool for task allocation and processing;

[0112] Step S630: When the number of retries is greater than or equal to a preset abnormal threshold, stop allocating tasks to the second data information and generate an abnormal identification for the identification number.

[0113] In step S610 of some embodiments, when the task processing thread fails to complete the data processing of the current second data information, it generates a retry feedback signal and retry count information. For example, when processing bank outbound call record data, if the task processing thread encounters a network failure when trying to download the outbound call recording file, resulting in a download failure, this mechanism will be triggered. The retry feedback signal contains an identification number corresponding to the current second data information so that the system can identify which data processing task has a problem. At the same time, the current number of retries will be recorded as the number of retries information. Initially, the number of retries is 1. If it is the second retry, the number of retries is 2, and so on.

[0114] In step S620 of some embodiments, the system responds to the retry feedback signal and pushes the second data information corresponding to the identification number to the task thread pool for task allocation and processing according to the identification number therein. The task thread pool will try to process the task again, just like the first time, and assign a task processing thread to perform related data processing operations. For example, when the task processing thread A is responsible for processing a certain second data information but fails during the first processing, after re-pushing, the task thread pool may assign task processing thread B to try to process the data again.

[0115] In step S630 of some embodiments, if the number of retries reaches or exceeds the abnormal threshold, the system stops assigning tasks to the second data information and generates an abnormal flag for its identification number. This abnormal flag can be used for subsequent troubleshooting and data recording. For example, the identification number is marked as "abnormal pending" and related error information is recorded for subsequent analysis and processing by the administrator.

[0116] In the above steps S610 to S630, through the retry mechanism, when the task processing thread fails for the first time, there is a chance to try again, which increases the possibility of successful data processing. In a complex system environment, various temporary problems may be encountered, such as network fluctuations, temporary server failures, etc. The retry mechanism can effectively deal with these situations, ensure that the data can be processed correctly as much as possible, and improve the reliability of the entire data processing process. At the same time, setting an abnormal threshold can also avoid unlimited retries and stop losses in time, allowing the system to focus resources on processing other processable data and improve overall resource utilization efficiency.

[0117] See also Figure 7 It is understandable that, in step S410, the following steps may be included but not limited to:

[0118] Step S710: marking identification numbers corresponding to a plurality of first data information as test identification numbers;

[0119] Step S720: Record the entry timestamp of the test identification number into the message queue and the exit timestamp of the test identification number in the message queue;

[0120] Step S730: Obtaining the first processing time of the first data information corresponding to the test mark number according to the queue entry timestamp and the queue exit timestamp;

[0121] Step S740: Obtaining an average processing time according to the first processing times corresponding to all test mark numbers;

[0122] Step S750: Obtain the processing speed of the task thread pool according to the average processing time.

[0123] In step S710 of some embodiments, a part or specific identification numbers are selected from the identification numbers corresponding to the first data information and marked as test identification numbers, and these test identification numbers will be used for subsequent measurement of the processing speed of the task thread pool. For example, when processing bank outbound call record data, a certain number of outbound call record identification numbers can be randomly selected as test identification numbers.

[0124] In steps S720 to S730 of some embodiments, when an identification number with a test tag number enters the message queue, its entry timestamp is recorded. This timestamp can be accurate to milliseconds or even smaller time units to accurately measure the processing time. At the same time, when these test identification numbers are taken out and processed by the task thread pool in the message queue, their dequeue timestamps are recorded. If the entry timestamp of a test tag number is 10:00:00.500 (hours: minutes: seconds. milliseconds), the dequeue timestamp is 10:00:05.750. Subsequently, the first processing time of the first data information corresponding to the test tag number is calculated based on its entry timestamp and dequeue timestamp. The specific calculation method is to subtract the entry timestamp from the dequeue timestamp, and the first processing time is 5.250 milliseconds.

[0125] In steps S740 to S750 of some embodiments, the average processing time is obtained according to the first processing time corresponding to all test mark numbers, specifically, the first processing time corresponding to all test mark numbers is accumulated, and then divided by the total number of test mark numbers to obtain the average processing time. If there are 100 test mark numbers, their first processing times are 5 milliseconds, 6 milliseconds, 4 milliseconds, etc., respectively, and these times are accumulated to 500 milliseconds, then the average processing time is 500÷100=5 milliseconds. The processing speed of the task thread pool is obtained according to the average processing time. The processing speed can be expressed by the amount of data processed per unit time. According to the above embodiment, if the average processing time is 5 milliseconds, and the task thread pool processes 100 data within a certain period of time, then the processing speed can be expressed as processing 100÷0.005=20000 data per second.

[0126] In the above steps, accurate processing speed information helps to better allocate resources. According to the processing speed, it is decided whether to increase the number of task processing threads, adjust server resource allocation, etc., to ensure that the system can run efficiently under different load conditions and improve resource utilization.

[0127] See also Figure 8 The present application also provides a mass data processing device 800, which can implement the above-mentioned mass data processing method, and the device includes:

[0128] An acquisition module 810 is used to acquire a plurality of first data information to be processed and an identification number corresponding to each first data information;

[0129] The first push module 820 is used to add the identification number corresponding to the first data information to the blocking queue in sequence according to the queue state in the preset blocking queue, and update the queue state;

[0130] The second push module 830 is used to push the identification number to the message queue of the preset message middleware in sequence according to the queue order of the blocking queue, and update the blocking queue and queue status;

[0131] A conversion module 840, configured to convert the first data information corresponding to the identification number into the second data information according to the identification number in the message queue;

[0132] The allocation processing module 850 is used to push a number of second data information and corresponding identification numbers to the task thread pool for task allocation and processing; wherein the task thread pool includes a number of task processing threads, each task processing thread is responsible for processing a second data information.

[0133] The specific implementation of the massive data processing device is substantially the same as the specific implementation of the aforementioned massive data processing method, and will not be described in detail herein.

[0134] The embodiment of the present application also provides an electronic device, the electronic device includes a memory and a processor, the memory stores a computer program, and the processor implements the above-mentioned mass data processing method when executing the computer program. The electronic device can be any intelligent terminal including a tablet computer, a car computer, etc.

[0135] See also Fig. 9 , Fig. 9 The hardware structure of an electronic device of another embodiment is illustrated, and the electronic device includes:

[0136] The processor 901 may be implemented by a general-purpose CPU (Central Processing Unit), a microprocessor, an application-specific integrated circuit (Application Specific Integrated Circuit, ASIC), or one or more integrated circuits, and is used to execute relevant programs to implement the technical solutions provided in the embodiments of the present application;

[0137] The memory 902 can be implemented in the form of a read-only memory (ROM), a static storage device, a dynamic storage device, or a random access memory (RAM). The memory 902 can store an operating system and other application programs. When the technical solution provided in the embodiment of this specification is implemented by software or firmware, the relevant program code is stored in the memory 902, and the processor 901 calls and executes the mass data processing method of the embodiment of this application;

[0138] Input / output interface 903, used to implement information input and output;

[0139] Communication interface 904, used to realize communication interaction between the device and other devices, which can be realized by wired mode (such as USB, network cable, etc.) or wireless mode (such as mobile network, WIFI, Bluetooth, etc.);

[0140] A bus 905 that transmits information between various components of the device (e.g., the processor 901, the memory 902, the input / output interface 903, and the communication interface 904);

[0141] The processor 901 , the memory 902 , the input / output interface 903 and the communication interface 904 are connected to each other in communication within the device via a bus 905 .

[0142] An embodiment of the present application also provides a computer-readable storage medium, which stores a computer program. When the computer program is executed by a processor, the above-mentioned mass data processing method is implemented.

[0143] The memory, as a non-transient computer-readable storage medium, can be used to store non-transient software programs and non-transient computer executable programs. In addition, the memory may include a high-speed random access memory, and may also include a non-transient memory, such as at least one disk storage device, a flash memory device, or other non-transient solid-state storage device. In some embodiments, the memory may optionally include a memory remotely disposed relative to the processor, and these remote memories may be connected to the processor via a network. Examples of the above-mentioned network include, but are not limited to, the Internet, an intranet, a local area network, a mobile communication network, and combinations thereof.

[0144] The method, device, electronic device and storage medium for processing massive data provided by the embodiment of the present application obtain several first data information to be processed and the identification number corresponding to each first data information. According to the queue state in the preset blocking queue, the identification number corresponding to the first data information is added to the blocking queue in sequence, and the queue state is updated, so that the data can be ensured to enter the processing flow in an orderly manner to avoid confusion. According to the queue order of the blocking queue, the identification number is pushed to the message queue of the preset message middleware in sequence, and the blocking queue and the queue state are updated. The message middleware can play a role of buffering and coordination to ensure the reliable transmission of data. According to the identification number in the message queue, the first data information corresponding to the identification number is converted into the second data information. This conversion process can be carried out according to specific business needs, such as format conversion, cleaning and other operations on the data. Several second data information and corresponding identification numbers are pushed to the task thread pool for task allocation and processing. The task thread pool includes several task processing threads, each task processing thread is responsible for processing a second data information, thereby realizing parallel processing, improving processing efficiency, and allocating data processing tasks to multiple task processing threads to realize parallel processing. At the same time, the use of message middleware can coordinate the division of tasks in a multi-host environment, avoiding the bottleneck of single-machine processing. When the amount of data increases, the processing capacity can be improved by increasing the number of task processing threads or hosts, thereby effectively solving the problem of insufficient single-machine capacity. Furthermore, this method avoids the use of a complex distributed timed task scheduling platform. Through the combination of blocking queues and message middleware, the orderly transmission and processing of data is achieved without the need to build an additional complex platform. The implementation of this method is relatively simple, easy to maintain and expand, and reduces the cost of development and deployment.

[0145] The embodiments described in the embodiments of the present application are intended to more clearly illustrate the technical solutions of the embodiments of the present application and do not constitute a limitation on the technical solutions provided in the embodiments of the present application. Those skilled in the art will appreciate that with the evolution of technology and the emergence of new application scenarios, the technical solutions provided in the embodiments of the present application are also applicable to similar technical problems.

[0146] Those skilled in the art will appreciate that the technical solutions shown in the figures do not constitute a limitation on the embodiments of the present application, and may include more or fewer steps than shown in the figures, or a combination of certain steps, or different steps.

[0147] The device embodiments described above are merely illustrative, and the units described as separate components may or may not be physically separated, that is, they may be located in one place or distributed on multiple network units. Some or all of the modules may be selected according to actual needs to achieve the purpose of the solution of this embodiment.

[0148] Those skilled in the art will appreciate that all or some of the steps in the methods disclosed above, and the functional modules / units in the systems and devices may be implemented as software, firmware, hardware, or a suitable combination thereof.

[0149] The terms "first", "second", "third", "fourth", etc. (if any) in the specification of the present application and the above-mentioned drawings are used to distinguish similar objects, and are not necessarily used to describe a specific order or sequence. It should be understood that the data used in this way can be interchangeable where appropriate, so that the embodiments of the present application described herein can be implemented in an order other than those illustrated or described herein. In addition, the terms "including" and "having" and any of their variations are intended to cover non-exclusive inclusions, for example, a process, method, system, product or device comprising a series of steps or units is not necessarily limited to those steps or units clearly listed, but may include other steps or units that are not clearly listed or inherent to these processes, methods, products or devices.

[0150] It should be understood that in the present application, "at least one (item)" means one or more, and "plurality" means two or more. "And / or" is used to describe the association relationship of associated objects, indicating that three relationships may exist. For example, "A and / or B" can mean: only A exists, only B exists, and A and B exist at the same time, where A and B can be singular or plural. The character " / " generally indicates that the objects associated before and after are in an "or" relationship. "At least one of the following" or similar expressions refers to any combination of these items, including any combination of single or plural items. For example, at least one of a, b or c can mean: a, b, c, "a and b", "a and c", "b and c", or "a and b and c", where a, b, c can be single or multiple.

[0151] In the several embodiments provided in the present application, it should be understood that the disclosed devices and methods can be implemented in other ways. For example, the device embodiments described above are only schematic. For example, the division of the above units is only a logical function division. There may be other division methods in actual implementation, such as multiple units or components can be combined or integrated into another system, or some features can be ignored or not executed. Another point is that the mutual coupling or direct coupling or communication connection shown or discussed can be through some interfaces, indirect coupling or communication connection of devices or units, which can be electrical, mechanical or other forms.

[0152] The units described above as separate components may or may not be physically separated, and the components shown as units may or may not be physical units, that is, they may be located in one place or distributed on multiple network units. Some or all of the units may be selected according to actual needs to achieve the purpose of the solution of this embodiment.

[0153] In addition, each functional unit in each embodiment of the present application may be integrated into one processing unit, or each unit may exist physically separately, or two or more units may be integrated into one unit. The above-mentioned integrated unit may be implemented in the form of hardware or in the form of software functional units.

[0154] If the integrated unit is implemented in the form of a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present application, or the part that contributes to the prior art, or all or part of the technical solution can be embodied in the form of a software product, and the computer software product is stored in a storage medium, including multiple instructions to enable a computer device (which can be a personal computer, server, or network device, etc.) to execute all or part of the steps of the methods of various embodiments of the present application. The aforementioned storage medium includes: U disk, mobile hard disk, read-only memory (Read-Only Memory, referred to as ROM), random access memory (Random Access Memory, referred to as RAM), disk or optical disk and other media that can store programs.

[0155] The preferred embodiments of the present invention are described above with reference to the accompanying drawings, but the scope of the rights of the present invention is not limited thereto. Any modification, equivalent substitution and improvement made by a person skilled in the art without departing from the scope and essence of the present invention should be within the scope of the rights of the present invention.

Claims

1. A method for processing massive data, characterized in that: The method comprises: Acquire a plurality of first data information to be processed and an identification number corresponding to each of the first data information; According to the queue state in the preset blocking queue, the identification numbers corresponding to the first data information are sequentially added to the blocking queue, and the queue state is updated; According to the queue order of the blocking queue, the identification number is pushed to the message queue of the preset message middleware in sequence, and the blocking queue and the queue state are updated; According to the identification number in the message queue, converting the first data information corresponding to the identification number into second data information; Pushing a plurality of the second data information and the corresponding identification numbers to a task thread pool for task allocation and processing; wherein the task thread pool includes a plurality of task processing threads, and each task processing thread is responsible for processing one piece of the second data information.

2. The method for processing massive data according to claim 1, characterized in that: The step of sequentially adding the identification numbers corresponding to the first data information to the blocking queue and updating the queue state includes: Add the identification number to a preset blocking queue, and update the task cumulative value corresponding to the blocking queue; When the task cumulative value is less than a preset full load threshold, the queue state of the blocking queue is updated to a smooth state; When the task cumulative value is greater than or equal to the full load threshold, the queue state of the blocking queue is updated to a busy state.

3. The method for processing massive data according to claim 2, characterized in that: The step of sequentially adding the identification numbers corresponding to the first data information to the blocking queue according to the queue state in the preset blocking queue, and updating the queue state, includes: When the queue state is a smooth state, adding the identification number to the blocked queue and updating the queue state; When the queue state is a busy state, stop adding the identification number to the blocking queue until the queue state is updated to a smooth state.

4. The method for processing massive data according to claim 1, characterized in that: The step of sequentially adding the identification numbers corresponding to the first data information to the blocking queue according to the queue state in the preset blocking queue, and updating the queue state, includes: According to the change of the message queue, the processing speed of the task thread pool is obtained; According to the processing speed, adjusting the queue entry speed of adding the identification number to the blocking queue; According to the queue state in the preset blocking queue and the queue entry speed, the identification number corresponding to the first data information is added to the blocking queue in sequence, and the queue state is updated.

5. The method for processing massive data according to claim 4, after the step of pushing the plurality of second data information to the task thread pool for task allocation, further comprising: When the task processing thread completes data processing of the current second data information, a completion feedback signal is generated; wherein the completion feedback signal includes the identification number corresponding to the current second data information; In response to the completion feedback signal, the corresponding identification number in the message queue is deleted.

6. The method for processing massive data according to claim 4, after the step of pushing the plurality of second data information to the task thread pool for task allocation, further comprising: When the task processing thread fails to complete data processing of the current second data information, a retry feedback signal and retry count information are generated; wherein the retry feedback signal includes the identification number corresponding to the current second data information; In response to the completion feedback signal, re-pushing the second data information corresponding to the identification number to the task thread pool for task allocation and processing; When the number of retries is greater than or equal to a preset abnormal threshold, the task allocation for the second data information is stopped, and an abnormal identification is generated for the identification number.

7. The method for processing massive data according to claim 5, characterized in that: The obtaining the processing speed of the task thread pool according to the change of the message queue includes: Marking the identification numbers corresponding to the first data information as test marking numbers; Recording the entry timestamp of the test identification number into the message queue and the exit timestamp of the test identification number in the message queue; Obtaining a first processing time of the first data information corresponding to the test mark number according to the queue entry timestamp and the queue exit timestamp; Obtaining an average processing time according to the first processing times corresponding to all the test mark numbers; The processing speed of the task thread pool is obtained according to the average processing time.

8. A mass data processing device, characterized in that: The device comprises: An acquisition module, used for acquiring a plurality of first data information to be processed and an identification number corresponding to each of the first data information; A first push module, configured to sequentially add the identification numbers corresponding to the first data information to the blocking queue according to a queue state in a preset blocking queue, and update the queue state; A second push module is used to push the identification number to the message queue of the preset message middleware in sequence according to the queue order of the blocking queue, and update the blocking queue and the queue state; a conversion module, configured to convert the first data information corresponding to the identification number into second data information according to the identification number in the message queue; An allocation and processing module is used to push a number of the second data information and the corresponding identification numbers to a task thread pool for task allocation and processing; wherein the task thread pool includes a number of task processing threads, and each task processing thread is responsible for processing one second data information.

9. An electronic device, characterized in that: The electronic device comprises a memory and a processor, the memory stores a computer program, and the processor implements the mass data processing method according to any one of claims 1 to 7 when executing the computer program.

10. A computer-readable storage medium storing a computer program, characterized in that: When the computer program is executed by a processor, the method for processing massive data according to any one of claims 1 to 7 is implemented.

Images