Multi-thread-based data processing method and device, equipment, medium and product

Abstract

The embodiment of the invention provides a multi-thread-based data processing method and device, equipment, a medium and a product, and relates to the field of big data. The method comprises the following steps: acquiring a to-be-processed data stream from an external data source through a first thread, and sending the to-be-processed data stream to a second thread; writing the data stream to be processed into a priority queue through the second thread; and acquiring the data processing task from the priority queue through a thread pool, distributing the data processing task to an adaptive third thread, and executing data processing operation through the third thread. By means of the mode, the three-level thread architecture solves the single-thread bottleneck problem through layered decoupling of data collecting, scheduling and processing processes. According to the method, analysis, verification and lake entering of external mass data are completed within millisecond-level delay, throughput is increased from tens of thousands of pieces per second to millions of pieces per second, the whole process is free of disk falling and zero restarting, and the requirements of financial and telecommunication scenes for high-real-time, high-safety and high-compliance data entering into the lake are met.

Description

TECHNICAL FIELD

[0001] The present application relates to the field of big data, and in particular to a data processing method and device based on multi-threading, equipment, medium and product. BACKGROUND

[0002] In the financial, telecommunications and other industries that are extremely sensitive to data security and compliance, core business systems need to continuously obtain massive real-time data from external partners, the Internet and Internet of Things terminals every day, and efficiently import them into internal data lakes to support key applications such as risk control, marketing, regulatory reporting, etc. Such scenarios place extremely high demands on data security, processing timeliness and resource utilization: not only must sensitive information be prevented from being leaked during cross-domain transmission, but also millions of data streams per second must be completed within milliseconds of delay, and system resource usage must be controllable.

[0003] Existing technologies generally use two modes, "file ferry" or "API relay", to implement cross-domain data import into the lake. The file ferry scheme uses an FTP / SFTP intermediate server as a physically isolated node, and the external data source first uploads the complete file to the intermediate server, and then the internal scheduled task pulls and lands it to the data lake. The API relay scheme uses REST / HTTP gateways or message middleware to continuously call interfaces to push data from external systems.

[0004] However, whether it is file ferry or API relay, the core consumer generally uses a single-threaded serial processing model: data reception, parsing, verification, conversion, and lake import are executed in series, which cannot meet the processing needs of high throughput and low latency. SUMMARY

[0005] The present application provides a data processing method and device based on multi-threading, equipment, medium and product to solve the technical problem that whether it is file ferry or API relay, the core consumer generally uses a single-threaded serial processing model: data reception, parsing, verification, conversion, and lake import are executed in series, which cannot meet the processing needs of high throughput and low latency.

[0006] In a first aspect, the present application provides a data processing method based on multi-threading, the method comprising:

[0007] acquiring a to-be-processed data stream from an external data source through a first thread, and sending the to-be-processed data stream to a second thread;

[0008] writing the to-be-processed data stream into a priority queue through the second thread;

[0009] acquiring a data processing task from the priority queue through a thread pool and assigning it to an adapted third thread, and executing a data processing operation through the third thread.

[0010] In a possible implementation, the data stream to be processed comprises at least two data types of real-time type, batch processing type and service type, and the writing of the data stream to be processed into the priority queue according to the preset priority rule by the second thread comprises:

[0011] The data stream to be processed is parsed to determine the data type and corresponding time limit of each sub-data stream;

[0012] Based on the data type and corresponding time limit, a task priority label is generated for each sub-data stream;

[0013] Each sub-data stream is written into the priority queue according to the task priority label.

[0014] In a possible implementation, the writing of each sub-data stream into the priority queue according to the task priority label comprises:

[0015] For a sub-data stream with a task priority higher than that of the current head task of the priority queue, a head insertion operation is performed;

[0016] For sub-data streams with the same task priority, sequence sorting is performed according to the FIFO rule;

[0017] The task backlog state of the priority queue is monitored, and when the processing time length of a preset number of tasks does not meet the corresponding time limit, a capacity expansion operation of the thread pool is triggered.

[0018] In a possible implementation, the obtaining of the data processing task from the priority queue by the thread pool comprises:

[0019] The priority queue is traversed in descending order of task priority by the thread pool, and the data processing task at the head of the priority queue is sequentially assigned to a third thread in an idle state;

[0020] When it is detected that there is a third thread in the idle state after completing a local task and having no new task in the local area, a secondary task queue bound to the remaining third thread is scanned;

[0021] The data processing task at the head of the secondary task queue is transferred.

[0022] In a possible implementation, the triggering of the capacity expansion operation of the thread pool comprises:

[0023] The maximum allowed number of third threads in the thread pool and the current number of currently surviving third threads are obtained;

[0024] determine a maximum expansion quantity according to a difference between the maximum allowed quantity and the current quantity, and determine a required expansion quantity according to a preset ratio and the current quantity;

[0025] obtain a target expansion quantity according to a minimum value between the maximum expansion quantity and the required expansion quantity, and create a new third thread in the thread pool based on the target expansion quantity.

[0026] In a possible implementation, the performing, by the third thread, of the data processing operation includes:

[0027] The third thread performs a multi-level filtering operation on the data processing task to obtain a to-be-entered-lake data stream.

[0028] Performing a data entering-lake operation on the to-be-entered-lake data stream.

[0029] In a possible implementation, the multi-level filtering operation includes at least two of a data desensitization operation, a format verification operation, and a business filtering operation.

[0030] A filtering path of the multi-level filtering operation is determined according to a data feature of the data processing task, and a filtering rule used by the business filtering operation is updated through an API or a configuration file.

[0031] In a second aspect, the present application provides a multi-thread-based data processing apparatus, including a collection module, a scheduling module, and a processing module, wherein:

[0032] The collection module is configured to obtain a to-be-processed data stream from an external data source through a first thread, and send the to-be-processed data stream to a second thread.

[0033] The scheduling module is configured to write the to-be-processed data stream into a priority queue through the second thread.

[0034] The processing module is configured to obtain a data processing task from the priority queue through a thread pool and assign the data processing task to an adapted third thread, and perform a data processing operation through the third thread.

[0035] In a possible implementation, the scheduling module is further configured to:

[0036] analyze the to-be-processed data stream to determine a data type and a corresponding time requirement of each sub-data stream;

[0037] generate a task priority label for each sub-data stream based on the data type and the corresponding time requirement;

[0038] write each sub-data stream into the priority queue according to the task priority label.

[0039] In a possible implementation, the scheduling module is further configured to:

[0040] perform a head insertion operation for a sub-data stream whose task priority is higher than a task at a current head of the priority queue;

[0041] perform sequence sorting according to a FIFO rule for a sub-data stream whose task priority is the same;

[0042] monitor a task backlog state of the priority queue, and trigger a capacity expansion operation of the thread pool when a processing duration of a preset number of tasks does not meet a corresponding timeliness requirement.

[0043] In a possible implementation, the processing module is further configured to:

[0044] traverse the priority queue in a descending order of task priority through the thread pool, and sequentially assign data processing tasks at a head of the priority queue to a third thread in an idle state;

[0045] when it is detected that there is a third thread in an idle state after completing a local task and having no new task in the local area, scan a secondary task queue bound to a remaining third thread;

[0046] transfer a data processing task at a head of a secondary task queue.

[0047] In a possible implementation, the processing module is further configured to:

[0048] obtain a maximum allowed number of third threads in the thread pool and a current number of currently surviving third threads;

[0049] determine a maximum capacity expansion number according to a difference between the maximum allowed number and the current number, and determine a required capacity expansion number according to a preset proportion and the current number;

[0050] obtain a target capacity expansion number according to a minimum value of the maximum capacity expansion number and the required capacity expansion number, and create a new third thread in the thread pool based on the target capacity expansion number.

[0051] In a possible implementation, the processing module is further configured to:

[0052] perform a multi-level filtering operation on the data processing task by using the third thread, to obtain a to-be-entered-lake data stream;

[0053] perform a data entering operation on the to-be-entered-lake data stream.

[0054] In a possible implementation, the multi-stage filtering operation includes at least two of a data desensitization operation, a format verification operation, and a service filtering operation.

[0055] The filtering path of the multi-stage filtering operation is determined according to a data feature of the data processing task, and a filtering rule used by the service filtering operation is updated through an API or a configuration file.

[0056] In a third aspect, the present application further provides an electronic device, comprising a processor and a memory connected to the processor in communication;

[0057] The memory stores computer execution instructions.

[0058] The processor executes the computer execution instructions stored in the memory to implement the method according to any one of the first aspect.

[0059] In a fourth aspect, the present application further provides a computer readable storage medium, wherein the computer readable storage medium stores computer execution instructions, and the computer execution instructions are executed by a processor to implement the method according to any one of the first aspect.

[0060] In a fifth aspect, the present application further provides a computer program product, comprising a computer program, which is executed by a processor to implement the method according to the foregoing.

[0061] The multi-thread based data processing method, device, equipment, medium and product provided by the present application, the first thread obtains the to-be-processed data stream from the external data source and sends it to the second thread, the second thread writes the to-be-processed data stream into a priority queue, and the thread pool obtains the data processing task from the priority queue and executes it, so that the to-be-processed data stream is entered into the lake. Through the above-mentioned manner, the three-level thread architecture decouples the data collection, scheduling and processing processes through layering, solves the single-thread bottleneck problem: the first thread is responsible for high-concurrency pulling, the second thread uniformly distributes according to priority, and the thread pool processes in parallel, so that the external massive data is completed in milliseconds Delay, the throughput jumps from tens of thousands of pieces per second to millions of pieces, and the whole process is zero-falling disk and zero-restarting, which meets the data lake demand of high real-time, high security and high compliance in financial and telecommunication scenarios. BRIEF DESCRIPTION OF DRAWINGS

[0062] The accompanying drawings, which are incorporated into and form part of the specification, illustrate embodiments consistent with the present application and, together with the specification, serve to explain the principles of the application.

[0063] Figure 1 The flowchart of a multi-thread based data processing method provided by the embodiment of the present application Figure One ;

[0064] Figure 2 A flowchart of a data processing method based on multi-threading provided for an embodiment of the present application Figure Two

[0065] Figure 3 A structural diagram of a data processing device based on multi-threading provided for an embodiment of the present application

[0066] Figure 4 A structural diagram of an electronic device provided for an embodiment of the present application

[0067] The specific embodiments of the present application have been shown through the above-described drawings, and will be described in more detail hereinafter. These drawings and written descriptions are not intended to limit the scope of the concept of the present application in any way, but to illustrate the concept of the present application to those skilled in the art by referring to specific embodiments. DETAILED DESCRIPTION

[0068] Exemplary embodiments will be described in detail herein with reference to the attached drawings. The same reference numbers in different drawings represent the same or similar elements. The embodiments described in the following exemplary embodiments are not meant to represent all implementations consistent with the present application. Rather, they are merely examples of apparatuses and methods consistent with some aspects of the present application as detailed in the appended claims.

[0069] Financial, telecommunications, and other industries need to efficiently process massive external data into a lake to meet high real-time business needs such as risk control and marketing, while ensuring data security and compliance. Existing technologies mainly adopt two schemes: a file ferry scheme uses an FTP / SFTP intermediate server as a physical isolation node, and an external data source first uploads a complete file to the intermediate server, and then an internal scheduled task pulls and lands the file to the data lake. The API relay scheme pushes data by continuously calling interfaces from external systems through the construction of REST / HTTP gateways or message middleware. However, whether it is a file ferry or an API relay, the core consumer generally adopts a single-threaded serial processing model: data receiving, parsing, verification, conversion, and lake entry are executed in series, which cannot meet the processing needs of high throughput and low latency.

[0070] ​The application provides a multi-thread-based data processing method, a first thread acquires a to-be-processed data stream from an external data source and sends the to-be-processed data stream to a second thread, the second thread writes the to-be-processed data stream into a priority queue, and a thread pool acquires a data processing task from the priority queue and executes the data processing task, so that the to-be-processed data stream is flowed into a lake. In the above manner, the three-level thread architecture decouples the data acquisition, scheduling and processing processes through layering, solves the single-thread bottleneck problem, the first thread is responsible for high-concurrency pulling, the second thread uniformly distributes according to priority, and the thread pool processes in parallel, so that external massive data is completed in a millisecond-level delay, and the throughput is increased from tens of thousands of pieces per second to millions of pieces per second, and the whole process is zero-falling disk and zero-restarting, which meets the data flowing into the lake requirements of high real-time, high security and high compliance in the financial and telecommunication fields.

[0071] It should be noted that the user information (including but not limited to user equipment information, user personal information, etc.) and data (including but not limited to data for analysis, stored data, displayed data, etc.) involved in the application are all information and data authorized by the user or authorized by all parties, and the collection, storage, use, processing, transmission, provision, disclosure and application of related data comply with relevant laws, regulations and standards of relevant countries and regions, necessary security measures are taken, do not violate public order and good customs, and provide corresponding operation portal for user to choose authorization or refusal.

[0072] It should be noted that the multi-thread-based data processing method, device, equipment, medium and product provided by the application can be used in the field of big data, and can also be used in any field other than the field of big data, and the application of the multi-thread-based data processing method, device, equipment, medium and product is not limited.

[0073] Figure 1 A flowchart of a multi-thread-based data processing method provided by an embodiment of the application Figure One As shown in Figure 1 , the method comprises:

[0074] S101, acquiring a to-be-processed data stream from an external data source through a first thread, and sending the to-be-processed data stream to a second thread.

[0075] In this step, in the financial, telecommunications and other industries that are extremely sensitive to data security and compliance, core business systems need to continuously obtain massive real-time data from external partners, the Internet and Internet of Things terminals every day, and efficiently import them into internal data lakes to support risk control, marketing, regulatory reporting and other key applications. Such scenarios place extremely high demands on data security, processing timeliness and resource utilization: not only must sensitive information be prevented from being leaked during cross-domain transmission, but also millions of data streams per second must be completed within milliseconds of delay, and system resource usage must be controllable. Existing technologies generally use two modes, "file ferry" or "API transfer", to implement cross-domain data import into the lake. The file ferry scheme uses an FTP / SFTP intermediate server as a physical isolation node, and the external data source first uploads the complete file to the intermediate server, and then the internal scheduled task pulls and lands it to the data lake. The API transfer scheme uses REST / HTTP gateways or message middleware to continuously call interfaces to push data from external systems. However, whether it is file ferry or API transfer, the core consumer generally uses a single-threaded serial processing model: data reception, parsing, verification, conversion, and lake writing are all serially executed, which cannot meet the processing needs of high throughput and low delay.

[0076] And splitting the three-level responsibilities of "collection - scheduling - processing" into independent threads is to exchange throughput with parallelism and low delay with division of labor. In the single-threaded model, network packet reception, protocol parsing, security filtering, format conversion, and lake writing are all serial, and any link blockage will slow down the entire link; after putting these three things into three types of threads, each thread only focuses on its most skilled action, and any level can be horizontally expanded, so that the million-level per second of serial waiting is changed into a parallel pipeline, and the millisecond-level delay naturally converges.

[0077] The consumer thread, i.e., the first thread, is responsible for "taking in" cross-domain data from external data sources without loss. It maintains a single thread, avoiding context switching and lock conflicts caused by multi-threaded socket contention, but quickly pushes the original data stream to the main thread in a zero-copy manner, solving the bottleneck of "slow reception".

[0078] The main thread, the second thread, plays the role of "intelligent scheduling command center" and does not perform any heavy CPU processing, but only labels data packets with priority and puts them into a priority queue. Because heavy operations such as desensitization and verification are stripped, the main thread can complete distribution in microseconds, making high-priority tasks always visible to worker threads in the first place, thereby reducing the risk of "slow scheduling" to a very low level.

[0079] The number of third threads in the worker threads of the thread pool automatically scales with the load and is specifically responsible for time-consuming desensitization, format verification, business filtering, and lake writing. They take tasks from the head of the priority queue in parallel, ensuring that high-priority tasks are consumed first, and by dynamically expanding the CPU to more than 80%, completely eliminating the "processing block" of the single-thread era.

[0080] Taking a bank as an example, external data sources can include user credit reports, UnionPay real-time transaction streams, stock exchange market data and reconciliation files, and also cover merchant / user behavior logs returned by cooperative Internet platforms (e-commerce, payment, travel), as well as device sensing and transaction logs generated by Internet of Things terminals (ATM, POS, intelligent teller machines). These information often contains millions of pieces of data per second, including sensitive fields such as ID cards and card numbers, and must be desensitized and verified before being called by the risk control engine. The three-level thread architecture allows the bank to safely inject such complex data into the data lake within milliseconds, meeting the dual requirements of privacy and timeliness for supervision, and providing uninterrupted "clean" data for real-time risk control, precision marketing, and regulatory reporting.

[0081] Specifically, after the system is started and initialized, the first thread continuously monitors the external data source in single-thread mode. Once new data flow is detected, the first thread reads the to-be-processed data stream from the network buffer through zero-copy and pushes the data stream to the second thread in a lock-free queue after completing preliminary packaging, ensuring that the data does not land or copy during transmission, achieving efficient and low-latency cross-thread data transfer.

[0082] In another possible implementation, in financial, telecommunications, and other scenarios, external data sources are directly exposed to untrusted networks, and any reverse connection can become a penetration entry. At the same time, regulations require that "data can only enter, not exit", and any instructions or attack paths from the outside to the core production area must be blocked. Therefore, before the first thread establishes a connection with the external data source, the system first builds a "one-way out, no way in" physical isolation channel between the DMZ area and the production area, and deploys the first thread in a separate container in the DMZ area, so that it can only pull data in one direction and cannot access any internal management ports in the reverse direction, thereby compressing the potential attack surface to the edge of the DMZ. The DMZ area is a buffer isolation network segment between the external untrusted network and the internal production network.

[0083] Specifically, the first thread initiates a one-way pull request through a TLS 1.3 encrypted channel in the DMZ area: the handshake phase forces the 0-RTT key derivation of TLS 1.3 with two-way certificate verification, ensuring that the session key is not saved. After each batch of data arrives, the first thread immediately generates an SM2 / SM3 signature for the data packet using the digital signature generated by the external system private key, and encapsulates the original data and the signature into an immutable message block, and then pushes it into the production area through the kernel-level one-way transmission bus (based on PCIe or optical shutter). The second thread in the production area is preset with the corresponding public key, and the signature verification is completed before the message block lands. If the digest value does not match or the certificate chain verification fails, the message block is directly discarded and a security alert is triggered, otherwise it is allowed to enter the priority queue, and the subsequent process is completely decoupled from the internal thread pool, realizing the rigid control of "verification before transmission, discard if not allowed".

[0084] Through the security isolation mechanism, no matter what means the external network uses to tamper with the field, truncate the message or perform a replay attack, it will be instantly identified and intercepted at the signature verification link, reducing the risk of data leakage. At the same time, the one-way physical channel completely eliminates the possibility of reverse control messages, meeting the hard requirement of "only in and not out" for cross-domain data. In the actual test, the total time for encryption and decryption and signature verification is stable at the millisecond level under a high-throughput scenario of 1.2 million per second, with zero packet loss and zero retransmission, which not only guarantees the confidentiality, integrity and availability of data, but also does not have a perceptible impact on the overall lake performance.

[0085] S102, write the to-be-processed data stream into the priority queue through the second thread.

[0086] In this step, the second thread acts as an intelligent scheduling command center, receiving the to-be-processed data stream from the first thread; the to-be-processed data stream is usually composed of at least two types of sub-data streams. The second thread immediately assigns a task priority label to each type of sub-data stream. Then, the sub-data stream carrying the corresponding task priority label is written into the priority queue in turn, completing the microsecond-level distribution. By only performing the two light operations of "labeling + enqueuing", the main thread completely strips off the heavy CPU load of desensitization, verification, etc., ensuring that high-priority tasks are always at the head of the queue and can be acquired and processed by the worker thread in the first time, thereby compressing the risk of "slow scheduling" to the minimum. The specific operation will be described in detail in the subsequent embodiments, and will not be described here.

[0087] S103, acquire the data processing task from the priority queue through the thread pool and assign it to the adapted third thread, and execute the data processing operation through the third thread.

[0088] In this step, the thread pool obtains data processing tasks from the priority queue, and allocates a third thread in an idle state to each data processing task. After allocation, the third thread first reads the task content and sequentially performs multi-level filtering operations. Then, the data that passes the filtering is written by the third thread into the internal data lake. The entire "pick-up-filter-lake" process is completely parallelized, ensuring that high-priority tasks are completed first, while keeping CPU utilization above 80%, and completely eliminating the "processing block" phenomenon of the single-thread era.

[0089] Further, multi-level filtering operations are introduced at the cross-domain data lake link to prevent data carrying sensitive fields, format errors, or invalid business from directly entering the data lake, thereby avoiding linear expansion of subsequent storage, calculation, and compliance audit costs. By connecting at least two levels of "data desensitization, format verification, and business filtering" in series, the system can complete privacy protection, structure legality checking, and business semantic verification before the data is dropped into the lake, achieving "cleaning first, then landing" step-by-step quality assurance.

[0090] The filtering path of the multi-level filtering operation is not fixed, but is dynamically determined based on the data characteristics (such as field type, data volume, and source identifier) carried by the data processing task. When desensitized or non-sensitive field traffic is identified, the desensitization node can be automatically skipped. When complex nested structures or large field messages are detected, deep format verification nodes are added to reduce redundant calculations. The business filtering operation at the end of the chain supports online rule updates through APIs or configuration files. Operators can issue field whitelists, logical constraints, or risk labels at any time, and the changes take effect without the need to restart the service, meeting the requirements of financial and telecommunications scenarios for rapid iteration and zero downtime of rules.

[0091] In another possible implementation, traditional data desensitization uses "regular + dictionary" or "fixed rule script" methods, which can only perform character-level replacement based on field names or types in flat structures. Once complex structures such as nested JSON, array-in-object, and multi-table association are encountered, the rule engine cannot identify the semantic association between deep fields, resulting in missed sensitive nodes and repeated processing of non-sensitive fields, which reduces both desensitization coverage and execution efficiency.

[0092] To overcome the structural limitations of rule engines, the system introduces a graph neural network (GNN) to model data instances in real time: each field is treated as a node, and the nesting, foreign key, and reference relationships between fields are treated as edges, forming a dynamic data feature graph. After offline training, GNN can instantly output "node-sensitivity probability" and "edge-association strength" during inference, and output an optimal desensitization path: enabling multi-layer hashing or format-preserving encryption for highly sensitive and highly correlated nodes, and directly skipping low-sensitive or isolated nodes, thereby concentrating desensitization computing power on the "critical path".

[0093] After adopting GNN dynamic path optimization, the masking coverage of complex data structures is improved, the average masking latency per message is reduced by 20%, CPU usage is reduced by 15%, and the operational overhead caused by rule expansion is completely eliminated. At the same time, the model supports online incremental updates, and newly emerging sensitive field patterns can be learned and take effect within hours, significantly enhancing the adaptability and maintainability of the masking strategy.

[0094] This application provides a multi-threaded data processing method. A first thread retrieves the data stream to be processed from an external data source and sends it to a second thread. The second thread writes the data stream to be processed into a priority queue. A thread pool retrieves data processing tasks from the priority queue and executes them, allowing the data to flow into the data lake. Through this three-tiered thread architecture, the data acquisition, scheduling, and processing processes are decoupled in a layered manner, solving the single-threaded bottleneck problem: the first thread is dedicated to high-concurrency data retrieval, the second thread distributes data uniformly according to priority, and the thread pool processes data in parallel. This enables massive amounts of external data to complete parsing, verification, and data entry into the data lake within milliseconds of latency, increasing throughput from tens of thousands of records per second to millions of records per second, with zero disk writes and zero restarts throughout the process. This meets the high real-time, high security, and high compliance requirements for data entry into the data lake in financial and telecommunications scenarios. The security isolation mechanism uses a one-way channel from the DMZ zone to the production zone and end-to-end encryption with TLS 1.3 to block external attack surfaces at the entry point, ensuring that data can only enter and not exit. A multi-level filtering mechanism intercepts invalid, illegal, or spam data through a three-tiered "de-identification-verification-business" link, resulting in a 60% reduction in spam data entering the data lake and a simultaneous reduction in storage and computing costs. Graph neural network de-identification dynamically identifies complex structures such as nested JSON and multi-table joins at the head of the chain, planning the optimal de-identification path in real time to improve sensitive field coverage and achieve high-quality end-to-end data entry into the data lake that is "securely transmitted, cleanly filtered, and accurately de-identified."

[0095] Figure 2 A flowchart illustrating a multi-threaded data processing method provided in this application embodiment. Figure Two This embodiment provides a detailed explanation of the data processing steps of the second thread and the thread pool during the data ingestion process. For example... Figure 2 As shown, the method includes:

[0096] S201. Parse the data stream to be processed and determine the data type and corresponding timeliness requirements of each sub-data stream.

[0097] The data stream to be processed includes sub-data streams of at least two data types: real-time, batch processing, and service. For example, the same data stream to be processed may contain real-time transaction logs of 100,000 records per second (real-time), batch credit files arriving every five minutes (batch processing), and on-demand REST gateway user profile query messages (service).

[0098] In this step, task priorities must be strongly correlated with the data type and time limit requirements of each data stream. This is because: real-time data (such as transaction logs) is often used for risk control and fraud prevention; delays exceeding 100 milliseconds will cause the interception window to be missed. Service-oriented data (such as user profile queries) is geared towards online business; timeouts will impact customer experience. Batch-processing data (such as end-of-day reconciliation files) can tolerate minute-level delays, but if overwhelmed by high-concurrency real-time tasks for an extended period, it will slow down the generation of regulatory reports. Therefore, only by using "type-time limit" as the core priority factor can we ensure that critical business operations always receive sufficient computing resources, avoiding a disconnect between system throughput and business SLAs.

[0099] Therefore, after receiving the data stream to be processed, the second thread first parses the Topic name, Schema version, timestamp and business tag fields in the message header to determine the data type and timeliness requirements of each sub-data stream.

[0100] Specifically, if the difference between the timestamp and the current system time is less than 1 second and the Topic prefix is ​​"rt-trade", it is marked as real-time with a time limit of 50ms; if the message header contains "batch-no" and the time window is 5 minutes, it is classified as batch processing with a time limit of 300s; if the URI path contains " / service" and the waiting queue length suddenly increases, it is identified as service processing with a time limit of 200ms.

[0101] S202. Based on the data type and corresponding timeliness requirements, generate task priority labels for each sub-data stream.

[0102] In this step, after parsing the data type and timeliness requirements of each sub-data stream, the second thread obtains the appropriate task priority and generates the corresponding task priority label. This embodiment does not specifically limit the step of obtaining task priorities; it can be determined according to the actual situation.

[0103] In one possible implementation, the second thread pre-sets a static mapping table of "data type-timeliness-priority". When it is necessary to obtain the priority of a task with different data streams, the mapping table is queried according to the data type and timeliness requirements to obtain the corresponding task priority.

[0104] In another possible implementation, a gradient boosting tree model is trained offline, with input features being data type and timeliness requirements, and the output being task priority. A second thread directly calls the model to obtain the task priority for each data stream.

[0105] S203. Write each sub-data stream into the priority queue according to the task priority label.

[0106] In this step, in the high-concurrency lake entry scenario, if only relying on natural order addition, sudden high-priority tasks (such as real-time risk control logs) will be accumulated at the tail of the queue, causing the miss of the millisecond-level time window; at the same time, if the same priority tasks are not arranged in order, the data that arrives first will be processed later, which will cause the business side "delay jitter". Therefore, the second thread adopts a three-stage writing strategy of "high-priority head insertion + same priority FIFO + timeout expansion", which not only ensures zero waiting time for urgent tasks, but also maintains fairness among tasks of the same priority, and releases computing resources in advance when the queue depth and processing time deviate from the SLA, avoiding the blocking of the entire pipeline by a single point. Among them, the FIFO rule is that the element that enters the queue first must be taken out first, and the element that enters later can only be arranged behind to be dequeued in turn, so as to ensure that the processing order is completely consistent with the arrival order.

[0107] The specific steps are:

[0108] For the data stream of the task priority higher than the current head task of the priority queue, perform a head insertion operation;

[0109] For the data stream of the same priority task, perform sequence sorting according to the FIFO rule;

[0110] Monitor the task backlog state of the priority queue, and trigger the expansion operation of the thread pool when the processing time of a preset number of tasks does not meet the corresponding time limit requirement.

[0111] Through head insertion, the average waiting time of real-time tasks is greatly reduced; the same level FIFO reduces the order reversal rate of the same batch of transaction logs, ensuring that the regulatory flow and timestamp are strictly consistent. The backlog monitoring and expansion mechanism improves CPU utilization, reduces discarding and retransmission events, and improves overall lake entry throughput while meeting financial-level millisecond time limit requirements.

[0112] S204, traverse the priority queue in descending order of task priority through the thread pool, and sequentially distribute the head task of the priority queue to the third thread in an idle state.

[0113] In this step, in the high-throughput, multi-priority real-time lake entry scenario, if only the idle worker thread mechanically waits for the global lock, the high-priority task may still be preempted by the "slow thread" even if it is at the head of the queue, resulting in the failure of the millisecond-level time limit. At the same time, the task quantity among the worker threads is naturally unbalanced, and some worker threads enter idle state first while their local queues are empty. If they cannot actively "borrow work", CPU will be idle and overall throughput will decrease. Therefore, the thread pool adopts a "priority descending traversal + work transfer" double strategy: first, ensure that high-priority tasks are allocated to idle threads in the first place, and then transfer the load from busy threads to lightly loaded threads through the transfer mechanism, ensuring the time limit and improving parallelism.

[0114] The implementation process is:

[0115] Traverse the priority queue in descending order of task priority by the thread pool, and in turn assign the data processing task at the head of the priority queue to the third thread in an idle state;

[0116] When it is detected that there is a third thread in the thread pool that is idle after completing a local task and has no new task to be allocated, scan the secondary task queue bound to the remaining third threads;

[0117] Transfer the data processing task at the head of the secondary task queue.

[0118] Specifically, the thread pool maintains an index linked list sorted by priority. Each time the scheduler starts, it pops the head task in a lock-free CAS manner from the head of the linked list (the smallest priority value), and assigns it to any third thread currently in an idle state, completing the matching of "who is the most idle, who processes the most urgent task first". The allocated third thread executes the filter-in-lake logic locally. After the task is completed, the local queue of the third thread is immediately checked, and if there is no new task, the "idle" signal is broadcast to the pool and the third thread enters the transfer mode. The idle third thread uses a circular scanning algorithm to traverse the secondary task queue bound to the remaining third threads; through a CAS operation, the target queue head task can be "cut off" and migrated into the context of the third thread for execution, and after the transfer is successful, the third thread immediately enters the running state without returning to the global queue to compete for the lock.

[0119] In the above manner, the average waiting time of the task is reduced in descending order of priority, the work transfer mechanism improves the CPU utilization rate of the thread pool, the backlog length of the high-priority queue decreases, the overall throughput improves, and at the same time, the overhead of context switching caused by frequent expansion is avoided, achieving the comprehensive scheduling goal of low delay, high concurrency, and zero starvation.

[0120] In another possible implementation, data floods often increase sharply within seconds, and if the thread pool is filled at one time, it will instantly consume CPU and memory, causing the entire node to avalanche. If the expansion is too slow, the queue backlog will also break the SLA. Therefore, the system uses a "upper limit-difference-proportion" three-stage calculation: first, cap the maximum allowed number, then gradually increase the current number of survivors by proportion, and finally take the minimum value of the two as the target expansion number, which prevents over-expansion and ensures that high-priority tasks get thread resources in time. The specific process is as follows:

[0121] Obtain the maximum allowed number of third threads in the thread pool, and the current number of currently surviving third threads;

[0122] Determine the maximum expansion number according to the difference between the maximum allowed number and the current number, and determine the number of expansions needed according to the preset proportion and the current number;

[0123] According to the minimum of the maximum expansion quantity and the required expansion quantity, a target expansion quantity is obtained, and a new third thread is created in the thread pool based on the target expansion quantity.

[0124] The strategy changes the expansion action from "step type" to "gradient type", so that the CPU utilization rate is smoothly increased, and CPU peak alarm is avoided. At the same time, due to the avoidance of excessive threads, the memory additional occupation is reduced, the average waiting time of high priority tasks is shortened, and the "fast and stable" elastic expansion is realized.

[0125] Exemplarily, the maximum allowed quantity is 256, the current quantity is 32, and the preset proportion is 50%. Then the maximum expansion quantity = 256-32 = 224, the required expansion quantity = current quantity x preset proportion = 0.5 x 32 = 16. The target expansion quantity is the minimum of the required expansion quantity 16 and the maximum expansion quantity, which is 16.

[0126] The application provides a multi-thread-based data processing method, which reduces the average waiting time of tasks and the sequence reversal rate through a three-level scheduling strategy of "priority descending traversal + head insertion + same level FIFO". In combination with the working transfer and gradient expansion mechanism, the CPU utilization rate is smoothly increased, the high priority queue backlog is reduced, the memory additional occupation is reduced, the overall lake throughput is improved, there is no CPU peak and no packet retransmission, and the financial-level data scheduling goal of low delay, high concurrency and zero starvation is realized.

[0127] Figure 3 A structure schematic diagram of a multi-thread-based data processing device provided by an embodiment of the application is provided. As shown in the figure, Figure 3 The multi-thread-based data processing device 30 includes a collection module, a scheduling module and a processing module, wherein:

[0128] The collection module is configured to acquire a to-be-processed data stream from an external data source through a first thread, and send the to-be-processed data stream to a second thread;

[0129] The scheduling module is configured to write the to-be-processed data stream into a priority queue through the second thread;

[0130] The processing module is configured to acquire a data processing task from the priority queue through a thread pool and assign the data processing task to an adaptive third thread, and execute a data processing operation through the third thread.

[0131] In a possible implementation, the scheduling module is further configured to:

[0132] analyze the to-be-processed data stream, determine the data type and corresponding time limit requirement of each sub-data stream;

[0133] generate a task priority label for each sub-data stream based on the data type and corresponding time limit requirement;

[0134] According to the task priority label, each sub-data stream is written into the priority queue.

[0135] In a possible implementation, the scheduling module is further configured to:

[0136] For the sub-data stream with the task priority higher than the task at the current head of the priority queue, a head insertion operation is performed;

[0137] For the sub-data stream with the same task priority, sequence sorting is performed according to the FIFO rule;

[0138] The task backlog state of the priority queue is monitored, and when the processing time length of a preset number of tasks does not meet the corresponding timeliness requirement, a capacity expansion operation of the thread pool is triggered.

[0139] In a possible implementation, the processing module is further configured to:

[0140] The priority queue is traversed in descending order of task priority through the thread pool, and the data processing task at the head of the priority queue is sequentially assigned to the third thread in an idle state;

[0141] When it is detected that there is a third thread in the idle state after completing a local task and without a new task in the local area, the secondary task queue bound by the remaining third thread is scanned;

[0142] The data processing task at the head of the secondary task queue is transferred.

[0143] In a possible implementation, the processing module is further configured to:

[0144] The maximum allowed number of third threads in the thread pool and the current number of currently surviving third threads are obtained;

[0145] According to the difference between the maximum allowed number and the current number, the maximum expansion number is determined, and the number of threads that need to be expanded is determined according to a preset ratio and the current number;

[0146] According to the minimum value of the maximum expansion number and the number of threads that need to be expanded, the target expansion number is obtained, and a new third thread is created in the thread pool based on the target expansion number.

[0147] In a possible implementation, the processing module is further configured to:

[0148] The third thread performs a multi-level filtering operation on the data processing task to obtain a to-be-entered-lake data stream;

[0149] The data entered into the lake operation is performed on the to-be-entered-lake data stream.

[0150] In a possible implementation, the multi-stage filtering operation includes at least two of a data desensitization operation, a format verification operation, and a service filtering operation.

[0151] The filtering path of the multi-stage filtering operation is determined according to a data feature of the data processing task, and the filtering rule used by the service filtering operation is updated through an API or a configuration file.

[0152] The data processing apparatus based on the multi-thread provided in this embodiment can execute the data processing method based on the multi-thread provided in the method embodiments, and has similar implementation principles and technical effects, which will not be repeated here.

[0153] Figure 4 A structural schematic diagram of an electronic device provided in this embodiment is shown in FIG. 4. As shown in FIG. 4, the electronic device 40 includes a processor 401 and a memory 402 connected with the processor 401. Optionally, the electronic device 40 further includes a communication component 403. The processor 401, the memory 402, and the communication component 403 are connected through a bus 404. Wherein: Figure 4

[0154] The memory 402 stores computer execution instructions;

[0155] The processor 401 executes the computer execution instructions stored in the memory 402 to implement the method in any one of the preceding embodiments.

[0156] The at least one processor 401 can be a central processing unit (CPU), or an application specific integrated circuit (ASIC), or one or more integrated circuits configured to implement the embodiments of the present application.

[0157] Optionally, in specific implementation, the processor 401 and the memory 402 are independently implemented, and the processor 401 and the memory 402 can be connected with each other through a bus and complete communication between each other. The bus can be an industry standard architecture (ISA) bus, a peripheral component interconnect (PCI) bus, or an extended industry standard architecture (EISA) bus, etc. The bus can be divided into an address bus, a data bus, a control bus, etc., but it does not mean that there is only one bus or one type of bus.

[0158] ​Optionally, if the processor 401 and the memory 402 are integrated on a chip, the processor 401 and the memory 402 can complete the communication through an internal interface.

[0159] The application further provides a computer storage medium, and the computer storage medium stores computer execution instructions. When the processor executes the computer execution instructions, the technical solution of the foregoing data processing device based on multiple threads is implemented.

[0160] The foregoing computer readable storage medium can be implemented by any type of volatile or nonvolatile storage device or a combination thereof, such as a static random access memory (SRAM), an electrically erasable programmable read-only memory (EEPROM), an erasable programmable read-only memory (EPROM), a programmable read-only memory (PROM), a read-only memory (ROM), a magnetic storage, a flash memory, a magnetic disk, or an optical disk. The computer readable storage medium can be any available medium that can be accessed by a general-purpose or special-purpose computer.

[0161] An exemplary readable storage medium is coupled to the processor, so that the processor can read information from the readable storage medium and write information to the readable storage medium. Of course, the readable storage medium can also be a component of the processor. The processor and the readable storage medium can be located in an application specific integrated circuit (ASIC). Of course, the processor and the readable storage medium can also exist as discrete components in the control device of the clothes processing apparatus.

[0162] The division of the units is only a logical function division, and in actual implementation, there can be another division manner, for example, multiple units or components can be combined or integrated into another system, or some features can be ignored or not executed. In addition, the coupling or direct coupling or communication connection between the units shown or discussed can be indirect coupling or communication connection through some interfaces, devices or units, and can be electrical, mechanical or other forms.

[0163] The units described as separate components can or can not be physically separate, and the components shown as units can or can not be physical units, that is, can be located in one place, or can be distributed on multiple network units. According to actual needs, some or all of the units can be selected to achieve the purpose of the embodiment.

[0164] In addition, the functional units in each embodiment of the application can be integrated in one processing unit, or each unit can be physically present separately, or two or more units can be integrated in one unit.

[0165] If the functions are implemented in the form of software function units and sold or used as independent products, they can be stored in a computer readable storage medium. Based on this understanding, the technical solutions of the present application or the parts of the present application that essentially contribute to the prior art or the parts of the technical solutions can be embodied in the form of a software product. The computer software product is stored in a storage medium and includes a plurality of instructions for causing a computer device (which can be a personal computer, a server, or a network device, etc.) to execute all or part of the steps of the methods indicated by the embodiments of the present application. The aforementioned storage medium includes: a U disk, a mobile hard disk, a read-only memory (ROM, Read-Only Memory), a random access memory (RAM, Random Access Memory), a magnetic disk or an optical disk, and various media that can store program codes.

[0166] Those of ordinary skill in the art can understand that all or part of the steps of the above-mentioned method embodiments can be completed by program instruction-related hardware. The aforementioned program can be stored in a computer readable storage medium. When the program is executed, the steps of the above-mentioned method embodiments are executed; and the aforementioned storage medium includes: a ROM, a RAM, a magnetic disk or an optical disk, and various media that can store program codes.

[0167] So far, the technical solutions of the present application have been described in combination with the preferred embodiments shown in the drawings, but those skilled in the art can easily understand that the protection scope of the present application is obviously not limited to these specific embodiments, and the above embodiments are only used to illustrate the technical solutions of the present application, but not to limit them; although the present application has been described in detail with reference to the above embodiments, those of ordinary skill in the art should understand that they can still modify the technical solutions recorded in the above embodiments, or make equivalent replacement for part or all of the technical features; and these modifications or replacements do not make the essence of the corresponding technical solutions deviate from the scope of the technical solutions of the embodiments of the present application.

Claims

1. A data processing method based on multithreading, characterized in that, The method includes: The first thread obtains the data stream to be processed from an external data source and sends the data stream to the second thread. The second thread writes the data stream to be processed into a priority queue. Data processing tasks are retrieved from the priority queue using a thread pool and assigned to an appropriate third thread, which then performs the data processing operations.

2. The method according to claim 1, characterized in that, The data stream to be processed includes at least two data types: real-time, batch processing, and service. The step of writing the data stream to be processed into a priority queue via the second thread according to a preset priority rule includes: The data stream to be processed is parsed to determine the data type and corresponding timeliness requirements of each sub-data stream; Based on the data type and the corresponding timeliness requirements, task priority labels are generated for each sub-data stream; According to the task priority label, each sub-data stream is written into the priority queue.

3. The method according to claim 2, characterized in that, The step of writing each sub-data stream into the priority queue according to the task priority label includes: For the sub-data stream of the current queue head task whose task priority is higher than that of the priority queue, perform a head insertion operation; For data streams with the same task priority, sort them according to the FIFO rule; Monitor the task backlog status of the priority queue, and trigger the expansion operation of the thread pool when the processing time of a preset number of tasks does not meet the corresponding timeliness requirements.

4. The method according to claim 3, characterized in that, The step of retrieving data processing tasks from the priority queue via a thread pool includes: The thread pool iterates through the priority queue in descending order of task priority and assigns the data processing task at the head of the priority queue to the third thread that is in an idle state. When it is detected that there is a third thread in the thread pool that is idle after completing a local task and has no new local tasks to be assigned, the secondary task queue bound to the remaining third threads is scanned. Transfer any data processing task from the head of the primary task queue.

5. The method according to claim 3, characterized in that, The process of triggering the expansion operation of the thread pool includes: Get the maximum allowed number of third threads in the thread pool, and the current number of currently alive third threads; The maximum expansion quantity is determined based on the difference between the maximum allowed quantity and the current quantity, and the required expansion quantity is determined according to a preset ratio and the current quantity. Based on the minimum of the maximum expansion quantity and the required expansion quantity, obtain the target expansion quantity, and create a new third thread in the thread pool based on the target expansion quantity.

6. The method according to claim 1, characterized in that, The data processing operation performed through the third thread includes: The third thread performs multi-level filtering operations on the data processing task to obtain the data stream to be entered into the lake. Perform a data inflow operation on the data stream to be inflowed into the lake.

7. The method according to claim 6, characterized in that, The multi-level filtering operation includes at least two of the following: data desensitization operation, format validation operation, and business filtering operation; The filtering path of the multi-level filtering operation is determined based on the data characteristics of the data processing task, and the filtering rules used in the business filtering operation are updated through API or configuration file.

8. A data processing device based on multithreading, characterized in that, include: The module consists of a data acquisition module, a scheduling module, and a processing module, among which: The acquisition module is used to acquire a data stream to be processed from an external data source through a first thread, and send the data stream to be processed to a second thread; The scheduling module is used to write the data stream to be processed into a priority queue through the second thread; The processing module is used to obtain data processing tasks from the priority queue through the thread pool and assign them to the appropriate third thread, and then execute the data processing operation through the third thread.

9. An electronic device, characterized in that, include: A processor, and a memory communicatively connected to the processor; The memory stores computer-executed instructions; The processor executes computer execution instructions stored in the memory to implement the method as described in any one of claims 1 to 7.

10. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores computer-executable instructions, which, when executed by a processor, are used to implement the method as described in any one of claims 1 to 7.

11. A computer program product, characterized in that, Includes a computer program that, when executed by a processor, implements the method of any one of claims 1 to 7.

Images