Massive data processing method and device, electronic equipment and storage medium

Abstract

Embodiments of the present application provide a mass data processing method and device, electronic equipment and storage medium. The method comprises: obtaining a plurality of first data information to be processed and an identifier corresponding to each first data information; according to a queue state in a preset blocking queue, adding the identifier corresponding to the first data information to the blocking queue in turn, and updating the queue state; according to the queue order of the blocking queue, pushing the identifier to the message queue of the preset message middleware in turn, and updating the blocking queue and the queue state; according to the identifier in the message queue, converting the first data information corresponding to the identifier into second data information; and pushing the plurality of second data information and the corresponding identifier to a task thread pool for task allocation and processing. Embodiments of the present application can coordinate task division through the message middleware, realize parallel processing of mass data, improve processing efficiency, and are simple to use and easy to expand.

Description

Technical Field

[0001] This application relates to the field of data processing technology, and in particular to a method and apparatus for processing massive amounts of data, electronic equipment, and storage medium. Background Technology

[0002] In today's digital age, the amount of data generated by businesses, especially in the financial sector, is exploding. Taking banks as an example, they generate massive amounts of outbound call records from employees and customers every day, data that is crucial for business analysis and decision-making. However, processing this massive amount of data presents numerous technical challenges.

[0003] One major technical problem is that traditional single-machine multi-threaded concurrent processing methods often fail to process all data within the allotted time when handling large amounts of data. This is because the business process of processing a single data entry is time-consuming, and the processing capacity of a single machine has a bottleneck limit. While a distributed scheduled task platform can be built to enable parallel operation of data processing services across multiple hosts and improve overall processing speed, this approach has several drawbacks. These include additional development and deployment costs, high requirements for platform disaster recovery capabilities that may lead to uneven load distribution, and the need for custom-developed sharding algorithms that lack versatility. Summary of the Invention

[0004] The main objective of this application is to propose a method, apparatus, electronic device, and storage medium for processing massive amounts of data. It aims to coordinate task allocation through message middleware, thereby achieving parallel processing of massive amounts of data, improving processing efficiency, and being simple to use and easy to expand.

[0005] To achieve the above objectives, a first aspect of this application proposes a method for processing massive amounts of data, the method comprising:

[0006] Acquire several pieces of first data information to be processed and the identifier number corresponding to each piece of first data information;

[0007] According to the queue status in the preset blocking queue, the identifier corresponding to the first data information is added to the blocking queue in sequence, and the queue status is updated.

[0008] According to the queue order of the blocking queue, the identifier is pushed sequentially to the message queue of the preset message middleware, and the blocking queue and the queue status are updated.

[0009] Based on the identifier in the message queue, the first data information corresponding to the identifier is converted into second data information;

[0010] Several pieces of the second data information and the corresponding identifier are pushed to the task thread pool for task allocation and processing; wherein, the task thread pool includes several task processing threads, and each task processing thread is responsible for processing one piece of the second data information.

[0011] In some embodiments, adding the identifier corresponding to the first data information sequentially to the blocking queue and updating the queue status includes:

[0012] Add the identifier to the preset blocking queue and update the cumulative task value corresponding to the blocking queue;

[0013] When the cumulative value of the tasks is less than the preset full load threshold, the queue status of the blocked queue is updated to the smooth state.

[0014] When the cumulative value of the tasks is greater than or equal to the full load threshold, the queue status of the blocking queue is updated to busy.

[0015] In some embodiments, the step of sequentially adding the identifier corresponding to the first data information to the blocking queue according to the queue status in the preset blocking queue, and updating the queue status, includes:

[0016] When the queue status is in a smooth state, the identifier is added to the blocked queue, and the queue status is updated;

[0017] When the queue status is busy, stop adding the identifier to the blocked queue until the queue status is updated to smooth.

[0018] In some embodiments, the step of sequentially adding the identifier corresponding to the first data information to the blocking queue according to the queue status in the preset blocking queue, and updating the queue status, includes:

[0019] The processing speed of the task thread pool is obtained based on the changes in the message queue.

[0020] Adjust the enqueue speed at which the identifier is added to the blocking queue based on the processing speed.

[0021] According to the queue status in the preset blocking queue, the identifier corresponding to the first data information is added to the blocking queue in sequence according to the queuing speed, and the queue status is updated.

[0022] In some embodiments, after the step of pushing a plurality of the second data information to the task thread pool for task allocation, the method further includes:

[0023] When the task processing thread completes the data processing of the current second data information, it generates a completion feedback signal; wherein, the completion feedback signal includes the identifier corresponding to the current second data information;

[0024] In response to the completion feedback signal, the corresponding identifier is deleted from the message queue.

[0025] In some embodiments, after the step of pushing a plurality of the second data information to the task thread pool for task allocation, the method further includes:

[0026] When the task processing thread fails to complete the data processing of the current second data information, a retry feedback signal and retry count information are generated; wherein, the retry feedback signal includes the identifier corresponding to the current second data information;

[0027] In response to the completion feedback signal, the second data information corresponding to the identifier is pushed back to the task thread pool for task allocation and processing.

[0028] When the number of retries is greater than or equal to a preset abnormal threshold, task allocation for the second data information is stopped, and an abnormal identifier is generated for the identifier number.

[0029] In some embodiments, determining the processing speed of the task thread pool based on changes in the message queue includes:

[0030] Mark the identifiers corresponding to several of the first data information as test identifiers;

[0031] Record the enqueue timestamp of the test identifier entering the message queue, and the dequeue timestamp of the test identifier in the message queue;

[0032] Based on the enqueue timestamp and the dequeue timestamp, the first processing time of the first data information corresponding to the test tag number is obtained;

[0033] The average processing time is obtained based on the first processing time corresponding to all the test markers;

[0034] The processing speed of the task thread pool is obtained based on the average processing time.

[0035] To achieve the above objectives, a second aspect of this application provides a massive data processing apparatus, the apparatus comprising:

[0036] The acquisition module is used to acquire several pieces of first data information to be processed and the identifier number corresponding to each piece of first data information.

[0037] The first push module is used to add the identifier corresponding to the first data information to the blocking queue in sequence according to the queue status in the preset blocking queue, and update the queue status.

[0038] The second push module is used to push the identifier number sequentially to the message queue of the preset message middleware according to the queue order of the blocking queue, and update the blocking queue and the queue status.

[0039] The conversion module is used to convert the first data information corresponding to the identifier number in the message queue into second data information.

[0040] The allocation and processing module is used to push several pieces of the second data information and the corresponding identifier to the task thread pool for task allocation and processing; wherein, the task thread pool includes several task processing threads, and each task processing thread is responsible for processing one piece of the second data information.

[0041] To achieve the above objectives, a third aspect of this application provides an electronic device, which includes a memory and a processor. The memory stores a computer program, and the processor executes the computer program to implement the method described in the first aspect.

[0042] To achieve the above objectives, a fourth aspect of the present application provides a computer-readable storage medium storing a computer program that, when executed by a processor, implements the method described in the first aspect.

[0043] The massive data processing method, apparatus, electronic device, and storage medium proposed in this application acquire several pieces of first data information to be processed and an identifier corresponding to each piece of first data information. Based on the queue status in a preset blocking queue, the identifiers corresponding to the first data information are sequentially added to the blocking queue, and the queue status is updated. This ensures that data enters the processing flow in an orderly manner, avoiding chaos. Following the queue order of the blocking queue, identifiers are sequentially pushed to the message queue of a preset message middleware, and the blocking queue and queue status are updated. The message middleware acts as a buffer and coordinator, ensuring reliable data transmission. Based on the identifiers in the message queue, the first data information corresponding to the identifiers is converted into second data information. This conversion process can be performed according to specific business needs, such as data format conversion and cleaning. Several pieces of second data information and their corresponding identifiers are pushed to a task thread pool for task allocation and processing. The task thread pool includes several task processing threads, each responsible for processing one piece of second data information, thereby achieving parallel processing and improving processing efficiency. Distributing data processing tasks to multiple task processing threads achieves parallel processing. Meanwhile, using message middleware enables coordination of task allocation in a multi-host environment, avoiding the bottleneck of single-machine processing. When the data volume increases, processing capacity can be improved by increasing the number of task processing threads or hosts, thus effectively solving the problem of insufficient single-machine capacity. Furthermore, this method avoids the use of complex distributed scheduled task platforms. Through the combination of blocking queues and message middleware, it achieves ordered data transmission and processing without the need to build an additional complex platform. The implementation of this method is relatively simple, easy to maintain and extend, and reduces development and deployment costs. Attached Figure Description

[0044] Figure 1 This is a flowchart of the massive data processing method provided in the embodiments of this application;

[0045] Figure 2 yes Figure 1 The flowchart of step S120 in the middle;

[0046] Figure 3 yes Figure 1 The flowchart of step S120 in the middle;

[0047] Figure 4 yes Figure 1 The flowchart of step S120 in the middle;

[0048] Figure 5 yes Figure 1 The flowchart following step S150;

[0049] Figure 6 yes Figure 1The flowchart following step S150;

[0050] Figure 7 yes Figure 4 The flowchart of step S410 in the middle;

[0051] Figure 8 This is a schematic diagram of the structure of the massive data processing device provided in the embodiments of this application;

[0052] Figure 9 This is a schematic diagram of the hardware structure of the electronic device provided in the embodiments of this application. Detailed Implementation

[0053] To make the objectives, technical solutions, and advantages of this application clearer, the following detailed description is provided in conjunction with the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are merely illustrative and not intended to limit the scope of this application.

[0054] It should be noted that although functional modules are divided in the device schematic diagram and a logical order is shown in the flowchart, in some cases, the steps shown or described may be performed in a different order than the module division in the device or the order in the flowchart. The terms "first," "second," etc., in the specification, claims, and the aforementioned drawings are used to distinguish similar objects and are not necessarily used to describe a specific order or sequence.

[0055] Unless otherwise defined, all technical and scientific terms used herein have the same meaning as commonly understood by one of ordinary skill in the art to which this application belongs. The terminology used herein is for the purpose of describing embodiments of this application only and is not intended to limit the scope of this application.

[0056] First, let's analyze some of the terms used in this application:

[0057] Message-Oriented Middleware (MOM) is software that utilizes efficient and reliable message passing mechanisms for platform-independent data exchange and for distributed system integration based on data communication. It enables message passing between different applications, allowing these applications to communicate and collaborate in a loosely coupled manner.

[0058] Message Queue (MQ): A queue data structure for storing messages, used for message passing between different application components or different systems. It is a core component of message middleware, providing an asynchronous, loosely coupled communication method.

[0059] A task thread pool is a mechanism for managing and reusing threads. It primarily consists of a set of pre-created threads that can be used to execute various tasks. The thread pool maintains a task queue to store tasks waiting to be executed. When a task needs to be executed, the thread pool retrieves a task from the task queue and assigns it to an idle thread for execution.

[0060] In today's digital age, the amount of data generated by businesses, especially in the financial sector, is exploding. Taking banks as an example, they generate massive amounts of outbound call records from employees and customers every day, data that is crucial for business analysis and decision-making. However, processing this massive amount of data presents numerous technical challenges.

[0061] One major technical problem is that traditional single-machine multi-threaded concurrent processing methods often fail to process all data within the allotted time when handling large amounts of data. This is because the business process of processing a single data entry is time-consuming, and the processing capacity of a single machine has a bottleneck limit. While a distributed scheduled task platform can be built to enable parallel operation of data processing services across multiple hosts and improve overall processing speed, this approach has several drawbacks. These include additional development and deployment costs, high requirements for platform disaster recovery capabilities that may lead to uneven load distribution, and the need for custom-developed sharding algorithms that lack versatility.

[0062] Specifically, in one application scenario: banks generate a large amount of outbound call records for employees and customers every day. This data needs to be further processed by a scheduled task starting at 2 AM every night (e.g., downloading outbound call recordings to the business system, determining whether to generate activity metrics for bank employees based on call duration, etc.), and all data records must be processed before 8 AM the next morning. A common solution for processing this data using single-machine multi-threaded concurrent processing is as follows:

[0063] (1) Data processing task dispatcher: Executes the data query SQL to query all records with the status of "unprocessed", iterates through each record, creates a data processing task for it, and puts it into the data processing thread pool. If all threads in the data processing thread pool are busy at this time, the data processing task dispatcher will temporarily sleep and wait until an idle thread becomes available to continue developing and distributing tasks.

[0064] (2) Data processing thread pool, obtains the distributed data processing tasks, runs the business processing flow, and modifies the status of the corresponding record in the database to "processed" after processing.

[0065] Due to the large volume of data to be processed, the lengthy processing time for each data entry, and the limitations of single-machine processing capacity, it is impossible to process all data within the specified timeframe. Therefore, when existing processing capacity is insufficient, it is necessary to add new machines as needed to run the data processing service in parallel, thereby improving the overall processing speed. In a multi-host parallel operation environment, it is essential to ensure that each host coordinates its task allocation, avoids processing the same data simultaneously, and achieves load balancing.

[0066] A common solution is to build an additional distributed scheduled task platform. This platform detects all running data processing service hosts registered with it and initiates scheduled tasks simultaneously on multiple hosts at regular intervals, passing the shard sequence number and total number of shards in the task. When each host runs the scheduled task in parallel, it uses an appropriate algorithm based on its own shard sequence number and total number of shards to perform sharded parallel processing of massive amounts of data, thereby improving the overall processing speed.

[0067] The drawbacks of this solution are as follows:

[0068] (1) Building a distributed scheduled task invocation platform requires additional development and deployment costs.

[0069] (2) The platform needs to have disaster recovery capabilities. When a host that is executing a task goes offline abnormally, its task needs to be transferred to other machines. This may cause some hosts to bear more tasks than other hosts, resulting in an unbalanced workload.

[0070] (3) The sharding algorithm needs to be customized according to the actual data situation and is not universal. If there is no column for incremental number, the algorithm of "number%total number of shards = shard number-1" cannot be used. Other sharding algorithms need to be considered, and the load balancing of the algorithm should be considered to distribute the tasks evenly to each host and avoid the backlog of tasks caused by excessive load on individual hosts.

[0071] Based on this, embodiments of this application provide a method and apparatus for massive data processing, an electronic device and a storage medium, which aim to coordinate task division through message middleware, realize parallel processing of massive data, improve processing efficiency, and are simple to use and easy to expand.

[0072] The embodiments of this application can acquire and process relevant data based on artificial intelligence technology. Artificial intelligence (AI) refers to the theories, methods, technologies, and application systems that use digital computers or machines controlled by digital computers to simulate, extend, and expand human intelligence, perceive the environment, acquire knowledge, and use that knowledge to obtain optimal results.

[0073] Foundational technologies for artificial intelligence generally include sensors, dedicated AI chips, cloud computing, distributed storage, big data processing, operating / interactive systems, and mechatronics. AI software technologies mainly encompass computer vision, robotics, biometrics, speech processing, natural language processing, and machine learning / deep learning.

[0074] This application can be used in a wide variety of general-purpose or special-purpose computer system environments or configurations. Examples include: personal computers, server computers, handheld or portable devices, tablet devices, multiprocessor systems, microprocessor-based systems, set-top boxes, programmable consumer electronics, network PCs, minicomputers, mainframe computers, and distributed computing environments including any of the above systems or devices. This application can be described in the general context of computer-executable instructions executed by a computer, such as program modules. Generally, program modules include routines, programs, objects, components, data structures, etc., that perform specific tasks or implement specific abstract data types. This application can also be practiced in distributed computing environments where tasks are performed by remote processing devices connected via a communication network. In distributed computing environments, program modules can reside in local and remote computer storage media, including storage devices.

[0075] It should be noted that in all specific embodiments of this application, when processing data related to user identity or characteristics, such as user information, user behavior data, user historical data, and user location information, user permission or consent is obtained first. Furthermore, the collection, use, and processing of this data comply with relevant laws, regulations, and standards of the relevant countries and regions. In addition, when embodiments of this application require access to sensitive personal information of users, separate permission or consent from the user is obtained through pop-ups or redirects to confirmation pages. Only after obtaining the user's separate permission or consent is the necessary user-related data for the normal operation of the embodiments of this application obtained.

[0076] Figure 1 This is an optional flowchart of the massive data processing method provided in the embodiments of this application. Figure 1 The method may include, but is not limited to, steps S110 to S150.

[0077] Step S110: Obtain several pieces of first data information to be processed and the identifier number corresponding to each piece of first data information;

[0078] Step S120: According to the queue status in the preset blocking queue, add the identifier corresponding to the first data information to the blocking queue in sequence, and update the queue status.

[0079] Step S130: According to the queue order of the blocking queue, push the identifier number to the message queue of the preset message middleware in sequence, and update the blocking queue and the queue status.

[0080] Step S140: Based on the identifier in the message queue, convert the first data information corresponding to the identifier into the second data information;

[0081] Step S150: Push several second data information and their corresponding identifiers to the task thread pool for task allocation and processing; wherein, the task thread pool includes several task processing threads, and each task processing thread is responsible for processing one second data information.

[0082] Steps S110 to S160 of this embodiment involve acquiring several pieces of first data information to be processed and an identifier corresponding to each piece of first data information. Based on the queue status in a preset blocking queue, the identifiers corresponding to the first data information are sequentially added to the blocking queue, and the queue status is updated. This ensures that data enters the processing flow in an orderly manner, avoiding chaos. Following the queue order of the blocking queue, identifiers are sequentially pushed to the message queue of a preset message middleware, and the blocking queue and queue status are updated. The message middleware acts as a buffer and coordinator, ensuring reliable data transmission. Based on the identifiers in the message queue, the first data information corresponding to the identifiers is converted into second data information. This conversion process can be performed according to specific business needs, such as data format conversion and cleaning. Several pieces of second data information and their corresponding identifiers are pushed to a task thread pool for task allocation and processing. The task thread pool includes several task processing threads, each responsible for processing one piece of second data information, thereby achieving parallel processing and improving processing efficiency. Distributing data processing tasks to multiple task processing threads achieves parallel processing. Meanwhile, using message middleware enables coordination of task allocation in a multi-host environment, avoiding the bottleneck of single-machine processing. When the data volume increases, processing capacity can be improved by increasing the number of task processing threads or hosts, thus effectively solving the problem of insufficient single-machine capacity. Furthermore, this method avoids the use of complex distributed scheduled task platforms. Through the combination of blocking queues and message middleware, it achieves ordered data transmission and processing without the need to build an additional complex platform. The implementation of this method is relatively simple, easy to maintain and extend, and reduces development and deployment costs.

[0083] In step S110 of some embodiments, the first data information can be obtained from a preset database by querying SQL. In the database, query conditions are defined according to business needs and data characteristics, such as filtering based on "pending" status, time, etc. Alternatively, data can be obtained from external systems, requiring the establishment of a network connection or API calls, with proper authentication and authorization. Or, it can be directly opened using appropriate tools according to its storage path and format. This application does not limit this approach.

[0084] In step S140 of some embodiments, after converting the first data information (the record information to be processed) into the second data information (MQ message), the coupling between the data producer (the part that sends data) and the data consumer (the part that processes data) is reduced. Before the conversion, the data producer may need to directly pass the unprocessed record information to the consumer in a specific format, and the consumer also needs to rely heavily on the producer's implementation details. However, by converting to an MQ message, as long as the communication protocol of the message middleware is followed, the producer and consumer can modify and extend independently. For example, if the data processing logic needs to be updated or a new consumer service needs to be added in a banking system, as long as the message format remains unchanged, it will not have a significant impact on the generation and transmission of data. For example, in the scenario of processing outbound call records in a bank, the speed of data generation (such as the generation of outbound call records) may be fast, while data processing (such as downloading outbound call recording files, generating employee activity index data, etc.) may be time-consuming. By converting to an MQ message, the outbound call record data can be sent to the message queue first, allowing downstream processing services to process it at their own pace, thereby improving the overall throughput of the system.

[0085] In step S150 of some embodiments, data processing can be performed using either single-machine or multi-machine processing. In multi-machine data processing, a centralized task thread pool management approach can be adopted, where all machines share a logical task thread pool. For example, a central scheduler is needed to manage task allocation. It needs to clearly know the load status of each machine, the number of idle threads in the task thread pool, and other information, and then accurately allocate the second data information and identifier to the appropriate thread for processing. Alternatively, each machine can configure its own independent task thread pool, allowing the second data information and corresponding identifier to be allocated to the task thread pools of different machines for processing according to a certain strategy.

[0086] Please see Figure 2 It is understood that step S120 may include, but is not limited to, the following steps:

[0087] Step S210: Add the identifier to the preset blocking queue and update the cumulative task value corresponding to the blocking queue;

[0088] Step S220: When the cumulative task value is less than the preset full load threshold, update the queue status of the blocked queue to the smooth state.

[0089] Step S230: When the cumulative task value is greater than or equal to the full load threshold, update the queue status of the blocking queue to busy status.

[0090] In step S210 of some embodiments, the identifier corresponding to the first data information is added to a preset blocking queue. This blocking queue can be regarded as a task waiting area, used to store the task identifiers to be processed in an orderly manner. Each time an identifier is added, the task accumulation value corresponding to the blocking queue is updated. The task accumulation value is a counter used to record the number of task identifiers currently stored in the blocking queue. For example, after adding an identifier, the task accumulation value is incremented by 1.

[0091] In steps S220 to S230 of some embodiments, the cumulative task value is compared with a preset full-load threshold. The full-load threshold is a pre-set value used to measure the busyness of the blocking queue. If the cumulative task value is less than the full-load threshold, the queue status of the blocking queue is updated to a smooth state. This indicates that the blocking queue currently has enough space to receive new task identifiers, the task processing progress is ideal, and there is no task backlog. Conversely, if the cumulative task value is greater than or equal to the full-load threshold, the queue status of the blocking queue is updated to a busy state. This means that the number of tasks in the blocking queue is already large, possibly approaching or reaching its processing capacity limit, and new task identifiers may need to wait for a period of time before being processed. By setting the full-load threshold and updating the queue status, the flow of tasks entering the blocking queue can be effectively controlled. When the queue is in a busy state, corresponding measures can be taken, such as pausing or slowing down the addition of new tasks, to avoid excessive task backlog in the queue, leading to exhaustion of system resources. This helps to rationally utilize system resources and ensure that data processing tasks can be carried out in an orderly manner within the system's capacity. On the other hand, queue status updates provide intuitive information for system monitoring. Administrators can observe queue status to understand the progress and pressure of task processing in a timely manner. For example, when the queue is busy for a long time, this may be a signal that the system may need to increase processing resources (such as increasing the number of threads, expanding servers, etc.) or optimize the data processing flow to improve the system's processing efficiency.

[0092] Correspondingly, in step S130, after pushing the identifier number to the message queue of the preset message middleware in the order of the blocking queue, the blocking queue and the queue status will be updated. Specifically, the corresponding identifier number in the blocking queue will be deleted and the task accumulation value will be decremented by 1. Then, based on the task accumulation value, the queue status corresponding to the blocking queue will be updated through steps S220 to S230.

[0093] Specifically, in steps S210 to S230, the parameters can be adjusted according to the number of task processing hosts. Assuming one task processing host can process 50 tasks in parallel, and 10 task processing hosts can collectively process 500 tasks in parallel, the blocking queue is typically set to twice the overall parallel processing capacity, and the full load threshold can be set to 1000. When the first order ID is added, the cumulative task value changes from 0 to 1, and the blocking queue remains in a smooth state. As order IDs are continuously added, assuming the 500th order ID is added, the cumulative task value becomes 500, and the queue state remains smooth. When the 1000th order ID is added, the cumulative task value reaches 1000, equal to the full load threshold of 1000, and the blocking queue state is updated to a busy state. After retrieving an order ID from the blocked queue and pushing it to the message queue, the cumulative task value in the blocked queue is decremented by 1, becoming 999. At this point, it is necessary to check the relationship between the cumulative task value and the full load threshold again. When the cumulative task value is less than the full load threshold again, the queue status of the blocked queue is updated back to the smooth state.

[0094] Additionally, a smoothness threshold can be preset, which can be set to 900. When the blocked queue enters a busy state, it needs to wait until the cumulative task value returns to 900 before updating the queue status back to smooth. During data processing, there may be brief task spikes, causing the cumulative task value to just fall below the full load threshold and then rise rapidly again. If the queue status is updated to smooth when the cumulative task value is only slightly below the full load threshold, administrators may misjudge the system's processing capacity and prematurely add new tasks. Setting a smoothness threshold can avoid such frequent state switching because the queue status will only be updated to smooth when the cumulative task value drops to a relatively low level. This more accurately reflects the queue's actual processing capacity and task pressure, reducing such misjudgments and making system monitoring more stable and effective.

[0095] Please see Figure 3 It is understood that step S120 may include, but is not limited to, the following steps:

[0096] Step S310: When the queue status is smooth, add the identifier to the blocked queue and update the queue status;

[0097] Step S320: When the queue status is busy, stop adding identifiers to the blocked queue until the queue status is updated to smooth.

[0098] In steps S310 to S320 of some embodiments, when the system is preparing to add the identifier corresponding to the first data information to the blocking queue, it first checks the current queue status of the blocking queue. By deciding whether to add the identifier based on the queue status, the blocking queue can be effectively prevented from becoming overloaded. Stopping the addition of new identifiers when the queue is busy prevents excessive task backlog, thereby avoiding excessive consumption of system resources due to too many tasks waiting to be processed. Through the above method steps, the system is ensured to operate within a reasonable resource range, improving the overall performance and stability of the system. For example, when processing large amounts of data, adding tasks to the blocking queue without control may lead to memory overflow or slow system response, while this mechanism can prevent this from happening.

[0099] Please see Figure 4 It is understood that step S120 may include, but is not limited to, the following steps:

[0100] Step S410: Obtain the processing speed of the task thread pool based on the changes in the message queue;

[0101] Step S420: Adjust the enqueue speed for adding the identifier to the blocking queue according to the processing speed;

[0102] Step S430: Based on the queue status in the preset blocking queue, add the identifier corresponding to the first data information to the blocking queue in sequence according to the enqueue speed, and update the queue status.

[0103] In steps S410 to S430 of some embodiments, the system infers the processing speed of the task thread pool by monitoring changes in the message queue. Based on the obtained task thread pool processing speed, the system adjusts the enqueue speed for adding identifiers to the blocking queue. If the processing speed is fast, it indicates that the task thread pool has high processing capacity, and the enqueue speed can be appropriately increased; conversely, if the processing speed is slow, the enqueue speed may need to be reduced to avoid excessive task backlog in the blocking queue. Therefore, according to the adjusted enqueue speed and the current queue state of the blocking queue, the identifiers corresponding to the first data information are sequentially added to the blocking queue.

[0104] The above steps allow for dynamic adjustment of the queuing speed based on the actual processing capacity of the task thread pool, achieving dynamic load balancing of the system. When processing massive amounts of data, the system load may vary at different times. This method ensures that the number of tasks in the blocking queue matches the processing capacity of the task thread pool. For example, during peak business periods, the task thread pool's processing speed may decrease; reducing the queuing speed at this time can prevent excessive task backlog. Conversely, during off-peak periods, increasing the queuing speed can fully utilize system resources and improve data processing efficiency. Therefore, dynamically adjusting the queuing speed based on the task thread pool's processing status ensures timely data processing and avoids data processing delays caused by task backlog due to excessively fast queuing speeds. Simultaneously, it also guarantees system stability, as a reasonable queuing speed prevents system crashes or anomalies due to task overload.

[0105] Please see Figure 5 It is understood that after step S150, the following steps may be included, but are not limited to:

[0106] Step S510: When the task processing thread completes the data processing of the current second data information, it generates a completion feedback signal; wherein, the completion feedback signal includes the identifier corresponding to the current second data information;

[0107] Step S520: In response to the completion feedback signal, delete the corresponding identifier from the message queue.

[0108] In steps S510 to S520 of some embodiments, after the task processing thread completes the data processing of the current second data information, it generates a completion feedback signal. This feedback signal contains the identifier corresponding to the second data information that has just been processed. For example, in a bank outbound call record processing system, after the task processing thread completes the processing of a certain outbound call record, such as completing the download of the outbound call recording file and the generation of employee activity index data, it will create a completion feedback signal with the identifier of that outbound call record. After receiving this completion feedback signal, the system will search for and delete the corresponding identifier in the message queue according to the identifier contained in the signal. By generating a completion feedback signal and deleting the identifier in the message queue, the system can accurately track the processing progress of each data, ensuring that each piece of second data information is processed correctly once, avoiding duplicate processing or omissions. When processing financial transaction data, accurately tracking the processing status of each transaction data ensures the integrity and accuracy of the transaction data and prevents financial risks caused by data processing errors. At the same time, timely deletion of processed identifiers can effectively utilize the space resources of the message queue. The message queue will not occupy too much space due to storing a large number of processed data identifiers, thus accommodating more new data identifiers to be processed.

[0109] Please see Figure 6 It is understood that after step S150, the following steps may be included, but are not limited to:

[0110] Step S610: When the task processing thread fails to complete the data processing of the current second data information, a retry feedback signal and retry count information are generated; wherein, the retry feedback signal includes the identifier corresponding to the current second data information;

[0111] Step S620: In response to the completion feedback signal, push the second data information corresponding to the identifier to the task thread pool for task allocation and processing;

[0112] Step S630: When the number of retries is greater than or equal to the preset abnormal threshold, stop assigning tasks to the second data information and generate an abnormal identifier for the identifier number.

[0113] In step S610 of some embodiments, when the task processing thread fails to complete the data processing of the current second data information, it generates a retry feedback signal and retry count information. For example, when processing bank outbound call record data, if the task processing thread encounters a network failure while attempting to download the outbound call recording file, causing the download to fail, this mechanism will be triggered. The retry feedback signal contains an identifier corresponding to the current second data information so that the system can identify which data processing task has encountered a problem. At the same time, the number of retries already performed is recorded as the retry count information. Initially, the retry count is 1; if it is the second retry, the retry count is 2, and so on.

[0114] In step S620 of some embodiments, in response to the retry feedback signal, the system re-pushes the second data information corresponding to the identifier number to the task thread pool for task allocation and processing based on the identifier number therein. The task thread pool will attempt to process the task again, just like the first time, allocating a task processing thread to perform the relevant data processing operations. For example, if task processing thread A failed to process a certain second data information during the first processing, after re-pushing, the task thread pool may allocate task processing thread B to attempt to process the data again.

[0115] In step S630 of some embodiments, if the number of retries reaches or exceeds an exception threshold, the system will stop assigning tasks to the second data information and generate an exception identifier for its identifier. This exception identifier can be used for subsequent troubleshooting and data logging. For example, the identifier can be marked as "abnormal and pending processing," and related error information can be recorded for subsequent analysis and processing by the administrator.

[0116] In steps S610 to S630 above, the retry mechanism allows the task processing thread to attempt processing again if it fails the first time, increasing the likelihood of successful data processing. In complex system environments, various temporary problems may arise, such as network fluctuations or brief server failures. The retry mechanism effectively addresses these situations, ensuring that data is processed as correctly as possible and improving the reliability of the entire data processing flow. Setting an exception threshold also prevents unlimited retries, allowing for timely mitigation and enabling the system to focus resources on processing other processable data, thus improving overall resource utilization efficiency.

[0117] Please see Figure 7 It is understood that step S410 may include, but is not limited to, the following steps:

[0118] Step S710: Mark the identifiers corresponding to several first data information as test identifiers;

[0119] Step S720: Record the enqueue timestamp of the test identifier entering the message queue, and the dequeue timestamp of the test identifier in the message queue;

[0120] Step S730: Based on the enqueue timestamp and dequeue timestamp, obtain the first processing time of the first data information corresponding to the test marker number;

[0121] Step S740: Calculate the average processing time based on the first processing time corresponding to all test markers;

[0122] Step S750: Obtain the processing speed of the task thread pool based on the average processing time.

[0123] In step S710 of some embodiments, a portion or specific identifiers are selected from the identifiers corresponding to a plurality of first data information and marked as test identifiers. These test identifiers will be used to subsequently measure the processing speed of the task thread pool. For example, when processing bank outbound call record data, a certain number of outbound call record identifiers can be randomly selected as test identifiers.

[0124] In steps S720 to S730 of some embodiments, when an identifier with a test tag number enters the message queue, its enqueue timestamp is recorded. This timestamp can be accurate to milliseconds or even smaller time units to accurately measure processing time. Simultaneously, after these test tags are retrieved from the message queue by the task thread pool and processed, their dequeue timestamp is recorded. If the enqueue timestamp of a test tag number is 10:00:00.500 (hours:minutes:seconds.milliseconds), and the dequeue timestamp is 10:00:05.750, then the first processing time of the first data information corresponding to that test tag number is calculated based on its enqueue and dequeue timestamps. Specifically, the calculation method is to subtract the enqueue timestamp from the dequeue timestamp, resulting in a first processing time of 5.250 milliseconds.

[0125] In steps S740 to S750 of some embodiments, the average processing time is obtained based on the first processing time corresponding to all test markers. Specifically, the first processing time corresponding to all test markers is summed and then divided by the total number of test markers to obtain the average processing time. If there are 100 test markers with first processing times of 5 milliseconds, 6 milliseconds, 4 milliseconds, etc., and these times are summed up to 500 milliseconds, then the average processing time is 500 ÷ 100 = 5 milliseconds. The processing speed of the task thread pool is obtained based on the average processing time. The processing speed can be expressed as the amount of data processed per unit time. According to the above embodiment, if the average processing time is 5 milliseconds, and the task thread pool processes 100 data items within a certain time, then the processing speed can be expressed as 100 ÷ 0.005 = 20,000 data items per second.

[0126] In the steps described above, accurate processing speed information helps to better allocate resources. Based on the processing speed, it can be determined whether to increase the number of task processing threads, adjust server resource allocation, etc., to ensure that the system can run efficiently under different load conditions and improve resource utilization.

[0127] Please see Figure 8 This application also provides a massive data processing device 800, which can implement the above-described massive data processing method. The device includes:

[0128] The acquisition module 810 is used to acquire several pieces of first data information to be processed and the identifier number corresponding to each piece of first data information;

[0129] The first push module 820 is used to add the identifier corresponding to the first data information to the blocking queue in sequence according to the queue status in the preset blocking queue, and update the queue status.

[0130] The second push module 830 is used to push identifiers sequentially to the message queue of the preset message middleware according to the queue order of the blocking queue, and update the blocking queue and queue status.

[0131] The conversion module 840 is used to convert the first data information corresponding to the identifier in the message queue into the second data information.

[0132] The allocation and processing module 850 is used to push several second data information and their corresponding identifiers to the task thread pool for task allocation and processing; wherein, the task thread pool includes several task processing threads, and each task processing thread is responsible for processing one second data information.

[0133] The specific implementation of this massive data processing device is basically the same as the specific implementation of the massive data processing method described above, and will not be repeated here.

[0134] This application also provides an electronic device, which includes a memory and a processor. The memory stores a computer program, and the processor executes the computer program to implement the above-described massive data processing method. This electronic device can be any smart terminal, including tablet computers, in-vehicle computers, etc.

[0135] Please see Figure 9 , Figure 9 The hardware structure of an electronic device according to another embodiment is illustrated. The electronic device includes:

[0136] The processor 901 can be implemented using a general-purpose CPU (Central Processing Unit), microprocessor, application-specific integrated circuit (ASIC), or one or more integrated circuits, and is used to execute relevant programs to implement the technical solutions provided in the embodiments of this application.

[0137] The memory 902 can be implemented as a read-only memory (ROM), a static storage device, a dynamic storage device, or a random access memory (RAM). The memory 902 can store the operating system and other application programs. When the technical solutions provided in the embodiments of this specification are implemented through software or firmware, the relevant program code is stored in the memory 902 and is called and executed by the processor 901 using the massive data processing method of the embodiments of this application.

[0138] The input / output interface 903 is used to implement information input and output;

[0139] The communication interface 904 is used to enable communication and interaction between this device and other devices. Communication can be achieved through wired means (such as USB, Ethernet cable, etc.) or wireless means (such as mobile network, WIFI, Bluetooth, etc.).

[0140] Bus 905 transmits information between various components of the device (e.g., processor 901, memory 902, input / output interface 903, and communication interface 904);

[0141] The processor 901, memory 902, input / output interface 903, and communication interface 904 are connected to each other within the device via bus 905.

[0142] This application also provides a computer-readable storage medium storing a computer program that, when executed by a processor, implements the above-described massive data processing method.

[0143] Memory, as a non-transitory computer-readable storage medium, can be used to store non-transitory software programs and non-transitory computer-executable programs. Furthermore, memory may include high-speed random access memory, and may also include non-transitory memory, such as at least one disk storage device, flash memory device, or other non-transitory solid-state storage device. In some embodiments, memory may optionally include memory remotely located relative to the processor, and these remote memories can be connected to the processor via a network. Examples of such networks include, but are not limited to, the Internet, intranets, local area networks, mobile communication networks, and combinations thereof.

[0144] The massive data processing method, apparatus, electronic device, and storage medium provided in this application embodiment acquire several first data information to be processed and an identifier corresponding to each first data information. Based on the queue status in a preset blocking queue, the identifiers corresponding to the first data information are sequentially added to the blocking queue, and the queue status is updated. This ensures that data enters the processing flow in an orderly manner, avoiding chaos. Following the queue order of the blocking queue, identifiers are sequentially pushed to the message queue of a preset message middleware, and the blocking queue and queue status are updated. The message middleware acts as a buffer and coordinator, ensuring reliable data transmission. Based on the identifiers in the message queue, the first data information corresponding to the identifiers is converted into second data information. This conversion process can be performed according to specific business needs, such as data format conversion and cleaning operations. Several second data information and their corresponding identifiers are pushed to a task thread pool for task allocation and processing. The task thread pool includes several task processing threads, each task processing thread responsible for processing one second data information, thereby achieving parallel processing and improving processing efficiency. Distributing data processing tasks to multiple task processing threads achieves parallel processing. Meanwhile, using message middleware enables coordination of task allocation in a multi-host environment, avoiding the bottleneck of single-machine processing. When the data volume increases, processing capacity can be improved by increasing the number of task processing threads or hosts, thus effectively solving the problem of insufficient single-machine capacity. Furthermore, this method avoids the use of complex distributed scheduled task platforms. Through the combination of blocking queues and message middleware, it achieves ordered data transmission and processing without the need to build an additional complex platform. The implementation of this method is relatively simple, easy to maintain and extend, and reduces development and deployment costs.

[0145] The embodiments described in this application are for the purpose of more clearly illustrating the technical solutions of the embodiments of this application, and do not constitute a limitation on the technical solutions provided by the embodiments of this application. As those skilled in the art will know, with the evolution of technology and the emergence of new application scenarios, the technical solutions provided by the embodiments of this application are also applicable to similar technical problems.

[0146] Those skilled in the art will understand that the technical solutions shown in the figures do not constitute a limitation on the embodiments of this application, and may include more or fewer steps than shown, or combine certain steps, or different steps.

[0147] The device embodiments described above are merely illustrative. The units described as separate components may or may not be physically separate; that is, they may be located in one place or distributed across multiple network units. Some or all of the modules can be selected to achieve the purpose of this embodiment according to actual needs.

[0148] Those skilled in the art will understand that all or some of the steps in the methods disclosed above, as well as the functional modules / units in the systems and devices, can be implemented as software, firmware, hardware, or suitable combinations thereof.

[0149] The terms “first,” “second,” “third,” “fourth,” etc. (if present) in the specification and accompanying drawings of this application are used to distinguish similar objects and are not necessarily used to describe a specific order or sequence. It should be understood that such data can be interchanged where appropriate so that the embodiments of this application described herein can be implemented in orders other than those illustrated or described herein. Furthermore, the terms “comprising” and “having,” and any variations thereof, are intended to cover non-exclusive inclusion; for example, a process, method, system, product, or apparatus that comprises a series of steps or units is not necessarily limited to those steps or units explicitly listed, but may include other steps or units not explicitly listed or inherent to such processes, methods, products, or apparatus.

[0150] It should be understood that in this application, "at least one (item)" means one or more, and "more than" means two or more. "And / or" is used to describe the relationship between related objects, indicating that three relationships can exist. For example, "A and / or B" can represent three cases: only A exists, only B exists, and both A and B exist simultaneously, where A and B can be singular or plural. The character " / " generally indicates that the preceding and following related objects are in an "or" relationship. "At least one (item) of the following" or similar expressions refer to any combination of these items, including any combination of single or plural items. For example, at least one (item) of a, b, or c can represent: a, b, c, "a and b", "a and c", "b and c", or "a and b and c", where a, b, and c can be single or multiple.

[0151] In the several embodiments provided in this application, it should be understood that the disclosed apparatus and methods can be implemented in other ways. For example, the apparatus embodiments described above are merely illustrative; for instance, the division of the units described above is only a logical functional division, and in actual implementation, there may be other division methods. For example, multiple units or components may be combined or integrated into another system, or some features may be ignored or not executed. Furthermore, the coupling or direct coupling or communication connection shown or discussed may be through some interfaces; the indirect coupling or communication connection between apparatuses or units may be electrical, mechanical, or other forms.

[0152] The units described above as separate components may or may not be physically separate. The components shown as units may or may not be physical units; that is, they may be located in one place or distributed across multiple network units. Some or all of the units can be selected to achieve the purpose of this embodiment according to actual needs.

[0153] Furthermore, the functional units in the various embodiments of this application can be integrated into one processing unit, or each unit can exist physically separately, or two or more units can be integrated into one unit. The integrated unit can be implemented in hardware or as a software functional unit.

[0154] If the integrated unit is implemented as a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of this application, in essence, or the part that contributes to the prior art, or all or part of the technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes multiple instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute all or part of the steps of the methods of the various embodiments of this application. The aforementioned storage medium includes various media capable of storing programs, such as USB flash drives, portable hard drives, read-only memory (ROM), random access memory (RAM), magnetic disks, or optical disks.

[0155] The preferred embodiments of the present application have been described above with reference to the accompanying drawings, but this does not limit the scope of the claims of the present application. Any modifications, equivalent substitutions, and improvements made by those skilled in the art without departing from the scope and substance of the embodiments of the present application shall be within the scope of the claims of the present application.

Claims

1. A method for processing massive amounts of data, characterized in that, The method includes: Acquire several pieces of first data information to be processed and the identifier number corresponding to each piece of first data information; Based on the queue status in the preset blocking queue, the identifier corresponding to the first data information is sequentially added to the blocking queue, and the queue status is updated, including: Based on the changes in the message queue of the preset message middleware, the processing speed of the task thread pool is obtained; based on the processing speed, the enqueue speed for adding the identifier to the blocking queue is adjusted; based on the queue status in the preset blocking queue, the identifier corresponding to the first data information is sequentially added to the blocking queue according to the enqueue speed, and the queue status is updated, including: The identifier is added to a preset blocking queue, and the cumulative task value corresponding to the blocking queue is updated; when the cumulative task value is greater than or equal to a preset full load threshold, the queue status of the blocking queue is updated to busy. According to the queue order of the blocking queue, the identifier is pushed sequentially to the message queue of the preset message middleware, and the blocking queue and the queue status are updated. Based on the identifier in the message queue, the first data information corresponding to the identifier is converted into second data information; Several pieces of the second data information and the corresponding identifier are pushed to the task thread pool for task allocation and processing; wherein, the task thread pool includes several task processing threads, and each task processing thread is responsible for processing one piece of the second data information; When the task processing thread fails to complete the data processing of the current second data information, a retry feedback signal and retry count information are generated; wherein, the retry feedback signal includes the identifier corresponding to the current second data information; in response to the retry feedback signal, the second data information corresponding to the identifier is pushed back to the task thread pool for task allocation and processing; when the retry count information is greater than or equal to a preset abnormal threshold, task allocation for the second data information is stopped, and an abnormal identifier is generated for the identifier. The step of obtaining the processing speed of the task thread pool based on changes in the message queue includes: Mark several identifiers corresponding to the first data information as test identifiers; record the enqueue timestamp of the test identifier entering the message queue and the dequeue timestamp of the test identifier in the message queue; obtain the first processing time of the first data information corresponding to the test identifier based on the enqueue timestamp and the dequeue timestamp; obtain the average processing time based on the first processing time corresponding to all the test identifiers; obtain the processing speed of the task thread pool based on the average processing time.

2. The massive data processing method according to claim 1, characterized in that, The step of sequentially adding the identifier corresponding to the first data information to the blocking queue and updating the queue status further includes: When the cumulative value of the tasks is less than the preset full load threshold, the queue status of the blocked queue is updated to the smooth state.

3. The massive data processing method according to claim 2, characterized in that, The step of adding the identifier corresponding to the first data information to the blocking queue sequentially according to the queue status in the preset blocking queue, and updating the queue status, includes: When the queue status is in a smooth state, the identifier is added to the blocked queue, and the queue status is updated; When the queue status is busy, stop adding the identifier to the blocked queue until the queue status is updated to smooth.

4. The massive data processing method according to claim 1, after the step of pushing a plurality of second data information and the corresponding identifier to the task thread pool for task allocation and processing, further includes: When the task processing thread completes the data processing of the current second data information, it generates a completion feedback signal; wherein, the completion feedback signal includes the identifier corresponding to the current second data information; In response to the completion feedback signal, the corresponding identifier is deleted from the message queue.

5. A massive data processing device, characterized in that, The device includes: The acquisition module is used to acquire several pieces of first data information to be processed and the identifier number corresponding to each piece of first data information. The first push module is used to add the identifier corresponding to the first data information to the blocking queue sequentially according to the queue status in the preset blocking queue, and update the queue status, including: Based on the changes in the message queue of the preset message middleware, the processing speed of the task thread pool is obtained; based on the processing speed, the enqueue speed for adding the identifier to the blocking queue is adjusted; based on the queue status in the preset blocking queue, the identifier corresponding to the first data information is sequentially added to the blocking queue according to the enqueue speed, and the queue status is updated, including: The identifier is added to a preset blocking queue, and the cumulative task value corresponding to the blocking queue is updated; when the cumulative task value is greater than or equal to a preset full load threshold, the queue status of the blocking queue is updated to busy. The second push module is used to push the identifier number sequentially to the message queue of the preset message middleware according to the queue order of the blocking queue, and update the blocking queue and the queue status. The conversion module is used to convert the first data information corresponding to the identifier number in the message queue into second data information. The allocation and processing module is used to push several pieces of the second data information and their corresponding identifiers to a task thread pool for task allocation and processing; wherein, the task thread pool includes several task processing threads, each task processing thread is responsible for processing one piece of the second data information; when a task processing thread fails to complete the data processing of the current second data information, a retry feedback signal and retry count information are generated; wherein, the retry feedback signal includes the identifier corresponding to the current second data information; in response to the retry feedback signal, the second data information corresponding to the identifier is pushed back to the task thread pool for task allocation and processing; when the retry count information is greater than or equal to a preset abnormal threshold, task allocation for the second data information is stopped, and an abnormal identifier is generated for the identifier; The step of obtaining the processing speed of the task thread pool based on changes in the message queue includes: Mark several identifiers corresponding to the first data information as test identifiers; record the enqueue timestamp of the test identifier entering the message queue and the dequeue timestamp of the test identifier in the message queue; obtain the first processing time of the first data information corresponding to the test identifier based on the enqueue timestamp and the dequeue timestamp; obtain the average processing time based on the first processing time corresponding to all the test identifiers; obtain the processing speed of the task thread pool based on the average processing time.

6. An electronic device, characterized in that, The electronic device includes a memory and a processor. The memory stores a computer program, and the processor executes the computer program to implement the massive data processing method according to any one of claims 1 to 4.

7. A computer-readable storage medium storing a computer program, characterized in that, When the computer program is executed by the processor, it implements the massive data processing method according to any one of claims 1 to 4.

Images