Data processing method and device, computer equipment and storage medium

Abstract

The invention provides a data processing method and device, computer equipment and a storage medium. The data processing method provided by the invention can be applied to one of a plurality of sub-threads. The method comprises the following steps: receiving values of a head variable and a tail variable configured by a main thread; processing a data block corresponding to the sub-thread according to the head variable and the tail variable; wherein the head variable and the tail variable are both atomic variables, the head variable represents a starting position of a data block corresponding to the sub-thread in total data, and the tail variable represents an ending position of the data block corresponding to the sub-thread in the total data; the main thread is used for dividing the total data into a plurality of data blocks in one-to-one correspondence with the plurality of sub-threads.

Description

Technical Field

[0001] This disclosure relates to the field of data processing technology, and in particular to data processing methods, apparatus, computer equipment and storage media. Background Technology

[0002] Concurrent programming techniques have significant application value in improving CPU performance and scalability in multithreaded programs, enhancing thread efficiency, and reducing system resource consumption. Especially with the development of computer hardware and the widespread adoption of multi-core processors, the effectiveness of multithreaded data processing has become a crucial factor in determining the degree of software optimization.

[0003] Since multithreaded data processing involves multiple sub-threads accessing the same set of data, related technologies often require locking the entire set of data to prevent data corruption. However, this approach can lead to severe lock contention under thread contention, resulting in thread blocking and waiting states, leading to poor overall efficiency and unsatisfactory multithreaded data processing performance. Summary of the Invention

[0004] To overcome the problems existing in related technologies, this disclosure provides a data processing method, apparatus, computer equipment, and storage medium.

[0005] The first aspect of this disclosure provides a data processing method applied to one of multiple sub-threads, the method comprising:

[0006] Receive the values ​​of the header and tail variables configured by the main thread;

[0007] The data block corresponding to the sub-thread is processed according to the header variable and the tail variable;

[0008] The header variable and the tail variable are both atomic variables. The header variable represents the start position of the data block corresponding to the sub-thread in the total data, and the tail variable represents the end position of the data block corresponding to the sub-thread in the total data. The main thread is used to divide the total data into multiple data blocks that correspond one-to-one with the multiple sub-threads.

[0009] Optionally, processing the data block corresponding to the child thread based on the header variable and the tail variable includes:

[0010] The data unit indicated by the header variable is processed;

[0011] The head variable is updated in the direction of the tail variable so that the head variable indicates the next data unit, until the head variable coincides with the tail variable.

[0012] Optionally, the method further includes:

[0013] In the case where the head variable and the tail variable overlap, determine whether there is a sub-thread among the plurality of sub-threads whose head variable and tail variable do not overlap;

[0014] In the case where any of the multiple sub-threads has a non-overlapping head variable and tail variable, the data block determined by the head variable and tail variable of the non-overlapping sub-thread is divided into two sub-data blocks, and either of the sub-data blocks is transferred to the sub-thread for processing, and the head variable or tail variable of the non-overlapping sub-thread is updated.

[0015] Optionally, the step of transferring any of the sub-data blocks to the sub-thread for processing, and updating the head or tail variable of any sub-thread, includes:

[0016] The sub-data block closest to the tail variable of either of the two sub-data blocks is transferred to that sub-thread for processing, and the tail variable of that sub-thread is updated.

[0017] Optionally, dividing the data block determined by the head and tail variables of any of the child threads into two sub-data blocks includes:

[0018] Determine an intermediate variable between the head variable and the tail variable of any child thread, wherein the intermediate variable is the middle value between the head variable and the tail variable of any child thread;

[0019] The data block determined by the head variable of any child thread and the intermediate variable, and the data block determined by the intermediate variable and the tail variable of any child thread are respectively regarded as two sub-data blocks.

[0020] Optionally, the plurality of sub-threads have thread numbers;

[0021] Determining whether there are any sub-threads among the plurality of sub-threads whose head variables and tail variables do not overlap includes:

[0022] Based on the increasing or decreasing relationship of the thread numbers, determine in turn whether the head and tail variables of the multiple sub-threads overlap.

[0023] Optionally, the main thread is used for:

[0024] The plurality of sub-threads are determined, and the data volume of the data block corresponding to each sub-thread is determined based on the number of the plurality of sub-threads and the total data volume;

[0025] Based on the amount of data in the data block corresponding to each sub-thread, the total data is divided into multiple data blocks that correspond one-to-one with the multiple sub-threads.

[0026] A second aspect of this disclosure provides a data processing method, the method comprising:

[0027] The total data is divided into multiple data blocks, each corresponding to one of the multiple sub-threads;

[0028] Configure the values ​​of the header and tail variables for each data block corresponding to the sub-thread, so that the sub-thread processes the data block corresponding to the sub-thread according to the values ​​of the header and tail variables;

[0029] Both the header variable and the tail variable are atomic variables. The header variable represents the beginning position of the data block corresponding to the sub-thread in the total data, and the tail variable represents the end position of the data block corresponding to the sub-thread in the total data.

[0030] Optionally, dividing the total data into multiple data blocks corresponding one-to-one with the multiple sub-threads includes:

[0031] Multiple sub-threads are identified, and the data volume of the data block corresponding to each sub-thread is determined based on the number of the multiple sub-threads and the total data volume.

[0032] Based on the amount of data in the data block corresponding to each sub-thread, the total data is divided into multiple data blocks that correspond one-to-one with the multiple sub-threads.

[0033] Optionally, determining the data size of the data block corresponding to each sub-thread based on the number of the plurality of sub-threads and the total data size includes:

[0034] The data volume of each sub-thread is determined by the quotient of the total data volume and the number of sub-threads.

[0035] A third aspect of this disclosure provides a data processing apparatus, the apparatus corresponding to one of a plurality of sub-threads, the apparatus comprising:

[0036] Configure the receiving module to receive the values ​​of the header and tail variables configured by the main thread;

[0037] The data block processing module is used to process the data block corresponding to the sub-thread according to the header variable and the tail variable;

[0038] The header variable and the tail variable are both atomic variables. The header variable represents the start position of the data block corresponding to the sub-thread in the total data, and the tail variable represents the end position of the data block corresponding to the sub-thread in the total data. The main thread is used to divide the total data into multiple data blocks that correspond one-to-one with the multiple sub-threads.

[0039] Optionally, the data block processing module, when processing the data block corresponding to the sub-thread based on the header variable and the tail variable, is used for:

[0040] The data unit indicated by the header variable is processed;

[0041] The head variable is updated in the direction of the tail variable so that the head variable indicates the next data unit, until the head variable coincides with the tail variable.

[0042] Optionally, the device further includes:

[0043] The overlap determination module is used to determine whether there are any sub-threads among the plurality of sub-threads whose head variables and tail variables do not overlap when the head variable and the tail variable overlap.

[0044] The sub-data block partitioning module is used to divide the data block determined by the head variable and tail variable of any of the multiple sub-threads into two sub-data blocks, transfer any of the sub-data blocks to the sub-thread for processing, and update the head variable or tail variable of the sub-thread when there is any sub-thread whose head variable and tail variable do not overlap.

[0045] Optionally, the sub-data block partitioning module is used to, when transferring any of the sub-data blocks to the sub-thread for processing, and updating the head or tail variable of any sub-thread, perform the following:

[0046] The sub-data block closest to the tail variable of either of the two sub-data blocks is transferred to that sub-thread for processing, and the tail variable of that sub-thread is updated.

[0047] Optionally, the sub-data block partitioning module is used to partition the data block determined by the head variable and tail variable of any of the sub-threads into two sub-data blocks, for the following purposes:

[0048] Determine an intermediate variable between the head variable and the tail variable of any child thread, wherein the intermediate variable is the middle value between the head variable and the tail variable of any child thread;

[0049] The data block determined by the head variable of any child thread and the intermediate variable, and the data block determined by the intermediate variable and the tail variable of any child thread are respectively regarded as two sub-data blocks.

[0050] Optionally, the plurality of sub-threads have thread numbers; the overlap determination module is used to determine whether there are sub-threads among the plurality of sub-threads whose head variables and tail variables do not overlap, for the following purposes:

[0051] Based on the increasing or decreasing relationship of the thread numbers, determine in turn whether the head and tail variables of the multiple sub-threads overlap.

[0052] Optionally, the main thread is used for:

[0053] The plurality of sub-threads are determined, and the data volume of the data block corresponding to each sub-thread is determined based on the number of the plurality of sub-threads and the total data volume;

[0054] Based on the amount of data in the data block corresponding to each sub-thread, the total data is divided into multiple data blocks that correspond one-to-one with the multiple sub-threads.

[0055] A fourth aspect of this disclosure provides a data processing apparatus, the apparatus comprising:

[0056] The data block partitioning module is used to divide the total data into multiple data blocks that correspond one-to-one with multiple sub-threads;

[0057] A configuration sending module is used to configure the values ​​of the header and tail variables of the sub-thread corresponding to each data block, so that the sub-thread processes the data block corresponding to the sub-thread according to the values ​​of the header and tail variables;

[0058] Both the header variable and the tail variable are atomic variables. The header variable represents the beginning position of the data block corresponding to the sub-thread in the total data, and the tail variable represents the end position of the data block corresponding to the sub-thread in the total data.

[0059] Optionally, the data block partitioning module is used to divide the total data into multiple data blocks corresponding one-to-one with the multiple sub-threads, for the following purposes:

[0060] Multiple sub-threads are identified, and the data volume of the data block corresponding to each sub-thread is determined based on the number of the multiple sub-threads and the total data volume.

[0061] Based on the amount of data in the data block corresponding to each sub-thread, the total data is divided into multiple data blocks that correspond one-to-one with the multiple sub-threads.

[0062] Optionally, the data block partitioning module, when determining the data size of the data block corresponding to each sub-thread based on the number of the multiple sub-threads and the total data volume, is used for:

[0063] The data volume of each sub-thread is determined by the quotient of the total data volume and the number of sub-threads.

[0064] The fifth aspect of this disclosure provides a computer program product, including a computer program / instructions that, when executed by a processor, implement the method described in either the first or second aspect.

[0065] A sixth aspect of this disclosure provides a computer device including a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor, when executing the program, implements the method as described in either the first or second aspect.

[0066] The seventh aspect of this disclosure provides a computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements the method as described in either the first or second aspect.

[0067] The technical solutions provided by the embodiments of this disclosure may include the following beneficial effects:

[0068] In this embodiment of the disclosure, the sub-thread has a header variable and a tail variable to identify the position of the data block it is responsible for in the total data block, and both of these variables are atomic variables; while the main thread divides the total task into multiple complete data blocks that correspond one-to-one with multiple sub-threads, and then configures the header variable and tail variable of each sub-thread according to the data block corresponding to each sub-thread, so that each sub-thread processes the data block it is responsible for.

[0069] In the above process, firstly, the tasks executed by each child thread no longer overlap, but are each responsible for a complete data block divided from the total data. This greatly improves the continuity of the child thread task execution process. Especially when there is a correspondence between the child thread and the physical core, it can greatly improve the utilization of CPU and cache and avoid frequent and invalid data reading and erasing.

[0070] Secondly, the data blocks corresponding to the child threads are identified by head and tail variables. This effectively forms a private queue for each child thread without occupying additional memory space. This prevents contention during the processing of the total data by multiple child threads, eliminating the need to lock the total data. This not only solves the problem of thread blocking and waiting but also allows the main thread to add other data after the total data without affecting the processing of the total data by multiple child threads, greatly improving the freedom and efficiency of multi-threaded data processing.

[0071] Third, since atomic variables can ensure that read and write operations on the variable are atomic and will not be interrupted or interfered with by operations of other threads, the above implementation can ensure the safety of multi-threaded operations with almost no impact on task execution efficiency, thereby ensuring the availability and reliability of multi-threaded data processing.

[0072] It should be understood that the above general description and the following detailed description are exemplary and explanatory only, and are not intended to limit this disclosure. Attached Figure Description

[0073] The accompanying drawings, which are incorporated in and form part of this disclosure, illustrate embodiments consistent with this disclosure and, together with the description, serve to explain the principles of this disclosure.

[0074] Figure 1 These are schematic diagrams illustrating one method of overall data partitioning, as shown in some exemplary embodiments.

[0075] Figure 2 This is a flowchart illustrating a data processing method using some exemplary embodiments.

[0076] Figure 3 These are schematic diagrams illustrating another way of dividing the total data, as shown in some exemplary embodiments.

[0077] Figure 4 These are schematic diagrams illustrating computer architectures through exemplary embodiments.

[0078] Figure 5 This is a general schematic diagram illustrating a data processing method using some exemplary embodiments.

[0079] Figure 6 This is a flowchart illustrating another data processing method using some exemplary embodiments.

[0080] Figure 7 This is a block diagram illustrating a data processing apparatus according to some exemplary embodiments.

[0081] Figure 8 This is a block diagram illustrating another data processing apparatus, shown in some exemplary embodiments.

[0082] Figure 9 These are hardware structure diagrams of a computer device illustrating some exemplary embodiments. Detailed Implementation

[0083] Exemplary embodiments will now be described in detail, examples of which are illustrated in the accompanying drawings. When the following description relates to the drawings, unless otherwise indicated, the same numerals in different drawings denote the same or similar elements. The embodiments described in the following exemplary embodiments do not represent all embodiments consistent with this disclosure. Rather, they are merely examples of apparatuses and methods consistent with some aspects of this disclosure as detailed in the appended claims.

[0084] Concurrent programming techniques have significant application value in improving the performance and scalability of CPUs (Central Processing Units) in multithreaded programs, enhancing thread efficiency, and reducing system resource consumption. Especially with the development of computer hardware and the widespread adoption of multi-core processors, the effectiveness of multithreaded data processing has become a crucial factor in determining the degree of software optimization.

[0085] Since multithreaded data processing involves multiple sub-threads accessing the same set of data, related technologies often require locking the entire set of data to prevent data corruption. However, this approach can lead to severe lock contention under thread contention, resulting in thread blocking and waiting states, leading to poor overall efficiency and unsatisfactory multithreaded data processing performance.

[0086] For details, please see Figure 1 This diagram illustrates one possible method of data partitioning, as shown in some exemplary embodiments. Each small square in the diagram represents a data unit, i.e., a unit for processing data, such as a pixel in an image or a character in text. The specific partitioning method of the data unit can be determined according to the actual design in the engineering application, for example, flexibly determined by specific software code or specific hardware structure (e.g., a pixel in an image can be considered as a data unit, or a row or a predetermined number of rows of pixels in an image can be considered as a data unit). This disclosure does not limit this. For example, Figure 1 Each small square (i.e., data unit) in the image can be a pixel in an image. Figure 1 The image (i.e., the total data) has 8 rows and 10 columns of pixels, and the specific multi-threaded data processing task can be to use two sub-threads to perform 3×3 Gaussian filtering on the image. This scenario will be used as an example to illustrate the implementation method provided in this disclosure.

[0087] In some related technologies, the main thread might call existing functions and directly implement the data allocation process based on an existing stack (the size of which is preset). For example, it might divide the total data by using each row or a preset number of rows of pixels as a unit (i.e., the size of each data block), and then assign the divided data to different threads for processing. Taking a division unit of one row of pixels as an example, we can obtain the following: Figure 1 The processing flow is as follows: the first row of pixels is processed by thread 1, the second row by thread 2, and after thread 1 finishes processing the first row, it receives the third row and processes it, and so on. This processing method is currently quite common, and ideally, its processing speed should be twice that of single-threaded processing. However, actual testing has shown that this approach does not perform as expected.

[0088] For details, please see Figure 2 This is a common computer architecture where CPU1 and CPU2 are two cores in a multi-core CPU, each with its own private cache (Cache means high-speed buffer memory). Figure 2 In a CPU core, CPU1 corresponds to Cache1, CPU2 corresponds to Cache2, and each core can also access main memory (RAM), hard drives, network adapters, and other components. RAM capacity is typically much larger than the cache, but CPU access to main memory takes longer than accessing the cache. The cache is each CPU's own local cache; these caches are private and can only be accessed by that specific core, with very fast access speeds. When a core needs to access main memory, it first checks the local cache. If the data is not in the local cache, it then requests main memory.

[0089] The implementation of a 3×3 Gaussian filter algorithm requires obtaining the pixel data of the current row and the two rows above and below it. For example, when thread 2 uses CPU2 to calculate all pixels in the second row of the image, it loads the data of the first, second, and third rows of the image from memory into Cache2. In subsequent processes, if CPU2 needs the data of the first, second, and third rows, it doesn't need to retrieve it from memory again, but can directly retrieve it from Cache2. However, in... Figure 1 As shown in the diagram, the task of calculating the third line is assigned to another thread (i.e., thread 1) and performed using another core (e.g., CPU1). At this point, to complete the calculation, thread 1 needs to load the data from the second, third, and fourth lines from memory into CPU1's cache (Cache1), while the data stored in Cache2 is not used in subsequent calculations. In other words, in... Figure 1 As shown in the process, the tasks received by each thread are not continuous. A single thread frequently processes across lines, the tasks are trivial, and the principle of spatial locality is not well satisfied.

[0090] Furthermore, existing thread pool technologies use locking mechanisms to ensure thread synchronization. Locking is a synchronization mechanism in multithreaded programming used to protect access to shared resources. When multiple threads access shared resources simultaneously, without locking, inconsistencies or data loss may occur. Locking ensures that only one thread can access shared resources at a time, thus preventing these problems. Locks are typically implemented based on mutexes or read-write locks. A mutex is an exclusive lock; only one thread can acquire the lock at a time, and other threads must wait for the lock to be released. A read-write lock allows multiple threads to read shared resources simultaneously, but requires an exclusive lock during write operations; other threads must wait for the write operation to complete before continuing to read or write. However, regardless of the type of lock, locking introduces additional overhead, including operations such as acquiring, releasing, and waiting for the lock. These operations degrade thread pool performance and can become a bottleneck, limiting the scalability of the thread pool. Furthermore, if the current user-instructed task is to perform a Gaussian filtering operation on a portion of an image, the total data can be a specified part of that image. If the user subsequently instructs the same Gaussian filtering operation to be performed on other parts of the image, then when the main thread adds the aforementioned "other parts" to the total data, it must also request and hold a lock (whether it is a mutex lock or a read-write lock, writing data requires holding a lock), thereby causing a more serious thread blocking situation.

[0091] In view of the above, this disclosure provides a data processing method, apparatus, computer device, and storage medium. The embodiments of this disclosure will now be described in detail from both the sub-thread and main thread perspectives.

[0092] The first aspect of this disclosure provides a data processing method that can be applied to one of multiple child threads. See also... Figure 3 It may include steps S301 to S302.

[0093] Step S301: Receive the values ​​of the header and tail variables configured by the main thread.

[0094] The header variable and the tail variable are both atomic variables. The header variable represents the start position of the data block corresponding to the sub-thread in the total data, and the tail variable represents the end position of the data block corresponding to the sub-thread in the total data. The main thread is used to divide the total data into multiple data blocks that correspond one-to-one with the multiple sub-threads.

[0095] Atomic variables are a mechanism used in concurrent programming to achieve thread safety. They ensure that read and write operations on a variable are atomic, meaning they are not interrupted by operations from other threads. In a multithreaded environment, when multiple threads simultaneously read and write to the same variable, data inconsistency or race conditions may occur. Using atomic variables can avoid this. Atomic variables typically provide atomic reads, atomic writes, atomic additions, etc., operations that are not interrupted by other threads, guaranteeing the integrity and consistency of the operations. For example, in Java, the `java.util.concurrent.atomic` package provides several atomic variable classes, such as `atomicInteger`, `atomicLong`, and `atomicBoolean`. The C++11 standard introduced atomic variables...<stdatomi c.h> The header file provides a series of atomic variable types, such as std::atomic.c The atomic variables in this disclosure can be implemented based on the techniques described above.

[0096] Regarding the method of dividing data blocks by the main thread, for example, the main thread can be used to: determine the multiple sub-threads, and determine the data volume of the data block corresponding to each sub-thread based on the number of the multiple sub-threads and the total data volume; and divide the total data into multiple data blocks corresponding one-to-one with the multiple sub-threads based on the data volume of the data blocks corresponding to each sub-thread.

[0097] For details, please see Figure 4 The main thread can obtain the total data volume and the currently available child threads (e.g., the number of threads currently available in the thread pool) as the aforementioned multiple child threads. Based on the number of child threads and the total data volume, it determines the data volume of the data block corresponding to each child thread. Then, based on the data volume of the data block corresponding to each child thread, it divides the total data into multiple data blocks corresponding one-to-one with the multiple child threads. For example, if the total data has 8 rows, and there are currently 2 available child threads, then the size (i.e., data volume) of the data block corresponding to each child thread can be determined to be 4 pixel rows (i.e., the data volume of the data block corresponding to each child thread is determined based on the quotient of the total data volume and the number of child threads). The first 4 rows and the last 4 rows of the total data are divided into two data blocks and processed by the two child threads respectively. That is, the main thread can configure the header variable of thread 1 to point to the first row of the total data and the tail variable to point to the fourth row of the total data, while configuring the header variable of thread 2 to point to the fifth row of the total data and the tail variable to point to the eighth row of the total data (taking one pixel row as an example).

[0098] Based on the above partitioning method, each thread can have a complete and continuous data block that is adapted to the total task size, thereby basically avoiding problems such as cross-line operations and repeated data retrieval between threads, improving the efficiency of multi-threaded data processing, and also making better use of the principle of spatial locality.

[0099] Step S302: Process the data block corresponding to the sub-thread according to the head variable and the tail variable.

[0100] After the sub-thread starts running, for example, data units can be extracted from the total data based on the value of the header variable and processed until the data unit corresponding to the tail variable is completed. That is, processing the data block corresponding to the sub-thread based on the header variable and the tail variable includes: processing the data unit indicated by the header variable; updating the header variable in the direction of the tail variable so that the header variable indicates the next data unit, until the header variable and the tail variable coincide. In this process, the atomic variable characteristics of the header variable and the tail variable ensure that the execution process will not have errors; and retrieving data based on the value of the header variable and updating the value of the header variable ensures the continuity of the task, and makes the values ​​of the header variable and the tail variable directly represent the remaining tasks of the sub-thread, so that the thread execution progress of the sub-thread no longer needs to be recorded by setting additional variables. Of course, step S302 can also record the current execution position by setting additional variables or use other data reading methods to process the data block, which will not be elaborated here.

[0101] In this embodiment of the disclosure, the sub-thread has a header variable and a tail variable to identify the position of the data block it is responsible for in the total data block, and both of these variables are atomic variables; while the main thread divides the total task into multiple complete data blocks that correspond one-to-one with multiple sub-threads, and then configures the header variable and tail variable of each sub-thread according to the data block corresponding to each sub-thread, so that each sub-thread processes the data block it is responsible for.

[0102] In the above process, firstly, the tasks executed by each child thread no longer overlap, but are each responsible for a complete data block divided from the total data. This greatly improves the continuity of the child thread task execution process. Especially when there is a correspondence between the child thread and the physical core, it can greatly improve the utilization of CPU and cache and avoid frequent and invalid data reading and erasing.

[0103] Secondly, the data blocks corresponding to the child threads are identified by head and tail variables. This effectively forms a private queue for each child thread without occupying additional memory space. This prevents contention during the processing of the total data by multiple child threads, eliminating the need to lock the total data. This not only solves the problem of thread blocking and waiting but also allows the main thread to add other data after the total data without affecting the processing of the total data by multiple child threads, greatly improving the freedom and efficiency of multi-threaded data processing.

[0104] Third, since atomic variables can ensure that read and write operations on the variable are atomic and will not be interrupted or interfered with by operations of other threads, the above implementation can ensure the safety of multi-threaded operations with almost no impact on task execution efficiency, thereby ensuring the availability and reliability of multi-threaded data processing.

[0105] In some embodiments, methods are provided to further improve the efficiency of multi-threaded data processing. Specifically, since the performance of different cores may vary, and the time consumed by a core in executing a task may be affected by various uncertainties or other hardware bottlenecks, the above method may further include:

[0106] In the case where the head variable and the tail variable overlap, determine whether there is a sub-thread among the plurality of sub-threads whose head variable and tail variable do not overlap;

[0107] In the case where any of the multiple sub-threads has a non-overlapping head variable and tail variable, the data block determined by the head variable and tail variable of the non-overlapping sub-thread is divided into two sub-data blocks, and either of the sub-data blocks is transferred to the sub-thread for processing, and the head variable or tail variable of the non-overlapping sub-thread is updated.

[0108] When the head and tail variables of a child thread overlap, the last data unit that the child thread is responsible for is also submitted for processing, at which point the thread's task will be completed. To maximize the utilization of the thread's processing capacity, the child thread can check if there are any other child threads with unfinished tasks. Upon finding any such child thread, the unfinished task is divided into two sub-task blocks. This child thread then processes only one sub-task block and transfers the other sub-task block to itself for processing (hereinafter referred to as "task stealing"), thereby distributing the workload among other child threads. Based on this, the implementation method provided in this disclosure can dynamically adapt to the performance of different child threads and dynamically balance the task processing status among child threads. This not only significantly improves the flexibility and data processing efficiency of the above method but also ensures that the main thread, when allocating data blocks, will not waste core computing power even without a detailed analysis of the data processing capabilities between cores. This also greatly improves the usability and universality of the above method.

[0109] Furthermore, the step of transferring any of the sub-data blocks to the sub-thread for processing and updating the head variable or tail variable of the any sub-thread may include: transferring the sub-data block that is closer to the tail variable of the any sub-thread from the two sub-data blocks to the sub-thread for processing, and updating the tail variable of the any sub-thread.

[0110] For example, a child thread can first identify any child thread whose task has not yet been completed. The following explanation uses the example of thread 1's task being completed while thread 2's task is still half-finished. That is, thread 1's header and tail variables already point to the same data unit (i.e., they overlap). At this point, thread 1, by calling thread 2's header and tail variables, discovers that thread 2's header and tail variables have not yet overlapped (a threshold can be set here; for example, task stealing is only performed on thread 2 when the difference between thread 2's header and tail variables exceeds a certain threshold; that is, the condition "not yet overlapping" can also be "the difference between the header and tail variables is less than or equal to the threshold"). Next, thread 1 can determine an intermediate variable between the head variable and the tail variable of thread 2, and then use the data block determined by the head variable and the intermediate variable of any sub-thread, and the data block determined by the intermediate variable and the tail variable of any sub-thread as two sub-data blocks respectively; that is, the data between the head variable and the intermediate variable of thread 2 is divided into one sub-data block, and the data between the intermediate variable and the tail variable of thread 2 is divided into another sub-data block (i.e., "the sub-data block closer to the tail variable of any sub-thread").

[0111] In this preferred approach, thread 2 can execute the sub-data block between the header variable and the intermediate variable, while thread 1 executes the sub-data block between the intermediate variable and the tail variable of thread 2. Specifically, the tail variable of thread 2 is modified to become the intermediate variable, the header variable of thread 1 is modified to point to the next data unit of the intermediate variable, and the tail variable of thread 1 is modified to become the original tail variable of thread 2 (atomic variables ensure that this process will not lead to data execution errors). Based on this, without interrupting the task execution of thread 2, a portion of the data that thread 2 has not yet finished processing can be seamlessly transferred to thread 1 for processing, further ensuring the continuity of data during multi-threaded task processing, thereby ensuring the efficiency of multi-threaded data processing.

[0112] In the above process, optionally, the intermediate variable can be the median value of the head and tail variables of any of the sub-threads, i.e., the mean of the head and tail variables (if the head and tail variables are not divisible, the intermediate variable can be determined by rounding in one direction). Based on this, the problem of poor task stealing effect caused by too much or too little data transferred at one time can be avoided, the thread processing progress between threads can be balanced to the greatest extent, and the efficiency of multi-threaded data processing can be improved.

[0113] Optionally, the multiple sub-threads have thread numbers. Determining whether any of the multiple sub-threads have overlapping head and tail variables may include: determining whether the head and tail variables of the multiple sub-threads overlap based on the increasing or decreasing relationship of the thread numbers. For example, after being identified as "multiple sub-threads," thread 1 and thread 2 may have numbers "1" and "2" respectively. This number can be known by the sub-thread itself, or by other sub-threads or the main thread. During the task stealing process described above, the sub-threads can check whether the head and tail variables of other sub-threads overlap based on the increasing or decreasing relationship of the thread numbers (of course, the thread number is not necessarily a number; when it is not a number, it can be adaptively sorted according to its corresponding code or alphabetical order). The checking process stops when a sub-thread with non-overlapping head and tail variables is found.

[0114] The advantage of this approach is that when there are many threads (e.g., more than 3), multiple child threads may have completed their assigned tasks simultaneously, while multiple child threads may not have finished their tasks. If each completed thread checks and steals tasks from other threads in an unordered manner, multiple child threads might steal the same task from the same thread, leading to decreased efficiency. By having each child thread check the values ​​of the head and tail variables of other child threads sequentially before stealing tasks, the checking and stealing processes of each child thread can be staggered as much as possible, avoiding contention and maximizing the efficiency of multi-threaded data processing.

[0115] The following combination Figure 5 The present disclosure further elaborates on the data processing method provided in at least one embodiment. For example, if there are four available child threads in the current thread pool, the main thread can divide the total data size by four to obtain the size of the data block that should be processed by each child thread, divide the total data into blocks according to this size, and configure the header variable (i.e., ...) of each child thread. Figure 5 (head) and tail variables (i.e. Figure 5 (tai l in the middle); Meanwhile, after confirming the invocation of these four child threads, the main thread or thread pool can configure thread numbers "1", "2", "3", and "4" for these four threads respectively. Next, each child thread can retrieve data from the total data based on the values ​​of the head and tail variables (for the kernel, this process can involve retrieving data from the cache, while the total data stored in memory is also continuously read into the cache) and process it sequentially (i.e., Figure 5 (POP in the middle).

[0116] And then, when there are child threads (e.g.) Figure 5 When thread 2 in the process finishes processing its task, it can sequentially read the values ​​of the head and tail variables of the other child threads. Figure 5 In this context, child threads sequentially check and steal the remaining tasks of other child threads in ascending order of thread number; for example, in... Figure 5 If thread 3's task is not yet completed, thread 2 can modify thread 3's tail variable to the intermediate value between thread 3's head variable and the original tail variable, and use this intermediate value as thread 2's head variable, and the original tail variable of thread 3 as thread 2's new tail variable (in this implementation, the way the child thread processes the task can be to first update the head variable towards the tail variable so that it points to the next data unit, and then deliver the data unit pointed to by the head variable to the relevant algorithm for processing). The specific implementation details and beneficial effects of the above process have been explained above and will not be repeated here.

[0117] The second aspect of this disclosure provides a data processing method that can be applied to the main thread. Please refer to [link to relevant documentation]. Figure 6 It may include steps S601 to S602.

[0118] Step S601: Divide the total data into multiple data blocks that correspond one-to-one with multiple sub-threads;

[0119] Step S602: Configure the values ​​of the header and tail variables of the sub-thread corresponding to each data block, so that the sub-thread processes the data block corresponding to the sub-thread according to the values ​​of the header and tail variables;

[0120] Both the header variable and the tail variable are atomic variables. The header variable represents the beginning position of the data block corresponding to the sub-thread in the total data, and the tail variable represents the end position of the data block corresponding to the sub-thread in the total data.

[0121] Optionally, dividing the total data into multiple data blocks corresponding one-to-one with the multiple sub-threads may include:

[0122] Multiple sub-threads are identified, and the data volume of the data block corresponding to each sub-thread is determined based on the number of the multiple sub-threads and the total data volume.

[0123] Based on the amount of data in the data block corresponding to each sub-thread, the total data is divided into multiple data blocks that correspond one-to-one with the multiple sub-threads.

[0124] Optionally, determining the data size of the data block corresponding to each sub-thread based on the number of the plurality of sub-threads and the total data size may include: determining the data size of the data block corresponding to each sub-thread based on the quotient of the total data size and the number of the plurality of sub-threads.

[0125] The second aspect embodiment described above corresponds to the first aspect embodiment. Specific details and optional implementation methods can be found in the relevant description of the first aspect embodiment, and will not be repeated here. It is worth noting that in this embodiment, the main thread's method of dividing the total data fundamentally changes the data division approach in multi-threaded data processing, and it has a cooperative relationship with some logical details of the child threads (such as task stealing).

[0126] Corresponding to the embodiments of the foregoing methods, this disclosure also provides embodiments of the apparatus and the terminal to which it is applied.

[0127] A third aspect of this disclosure provides a data processing apparatus corresponding to one of a plurality of sub-threads; see [link to relevant documentation]. Figure 7 The device may include:

[0128] Configure receiving module 701 to receive the values ​​of the header and tail variables configured in the main thread;

[0129] The data block processing module 702 is used to process the data block corresponding to the sub-thread according to the header variable and the tail variable;

[0130] The header variable and the tail variable are both atomic variables. The header variable represents the start position of the data block corresponding to the sub-thread in the total data, and the tail variable represents the end position of the data block corresponding to the sub-thread in the total data. The main thread is used to divide the total data into multiple data blocks that correspond one-to-one with the multiple sub-threads.

[0131] Optionally, the data block processing module, when processing the data block corresponding to the sub-thread based on the header variable and the tail variable, is used for:

[0132] The data unit indicated by the header variable is processed;

[0133] The head variable is updated in the direction of the tail variable so that the head variable indicates the next data unit, until the head variable coincides with the tail variable.

[0134] Optionally, the device further includes:

[0135] The overlap determination module is used to determine whether there are any sub-threads among the plurality of sub-threads whose head variables and tail variables do not overlap when the head variable and the tail variable overlap.

[0136] The sub-data block partitioning module is used to divide the data block determined by the head variable and tail variable of any of the multiple sub-threads into two sub-data blocks, transfer any of the sub-data blocks to the sub-thread for processing, and update the head variable or tail variable of the sub-thread when there is any sub-thread whose head variable and tail variable do not overlap.

[0137] Optionally, the sub-data block partitioning module is used to, when transferring any of the sub-data blocks to the sub-thread for processing, and updating the head or tail variable of any sub-thread, perform the following:

[0138] The sub-data block closest to the tail variable of either of the two sub-data blocks is transferred to that sub-thread for processing, and the tail variable of that sub-thread is updated.

[0139] Optionally, the sub-data block partitioning module is used to partition the data block determined by the head variable and tail variable of any of the sub-threads into two sub-data blocks, for the following purposes:

[0140] Determine an intermediate variable between the head variable and the tail variable of any child thread, wherein the intermediate variable is the middle value between the head variable and the tail variable of any child thread;

[0141] The data block determined by the head variable of any child thread and the intermediate variable, and the data block determined by the intermediate variable and the tail variable of any child thread are respectively regarded as two sub-data blocks.

[0142] Optionally, the plurality of sub-threads have thread numbers; the overlap determination module is used to determine whether there are sub-threads among the plurality of sub-threads whose head variables and tail variables do not overlap, for the following purposes:

[0143] Based on the increasing or decreasing relationship of the thread numbers, determine in turn whether the head and tail variables of the multiple sub-threads overlap.

[0144] Optionally, the main thread is used for:

[0145] The plurality of sub-threads are determined, and the data volume of the data block corresponding to each sub-thread is determined based on the number of the plurality of sub-threads and the total data volume;

[0146] Based on the amount of data in the data block corresponding to each sub-thread, the total data is divided into multiple data blocks that correspond one-to-one with the multiple sub-threads.

[0147] This disclosure provides a data processing apparatus in a fourth aspect; please refer to [link to relevant documentation]. Figure 8 The device may include:

[0148] The data block partitioning module 801 is used to divide the total data into multiple data blocks that correspond one-to-one with multiple sub-threads;

[0149] The configuration sending module 802 is used to configure the values ​​of the header and tail variables of the sub-thread corresponding to each data block, so that the sub-thread processes the data block corresponding to the sub-thread according to the values ​​of the header and tail variables;

[0150] Both the header variable and the tail variable are atomic variables. The header variable represents the beginning position of the data block corresponding to the sub-thread in the total data, and the tail variable represents the end position of the data block corresponding to the sub-thread in the total data.

[0151] Optionally, the data block partitioning module is used to divide the total data into multiple data blocks corresponding one-to-one with the multiple sub-threads, for the following purposes:

[0152] Multiple sub-threads are identified, and the data volume of the data block corresponding to each sub-thread is determined based on the number of the multiple sub-threads and the total data volume.

[0153] Based on the amount of data in the data block corresponding to each sub-thread, the total data is divided into multiple data blocks that correspond one-to-one with the multiple sub-threads.

[0154] Optionally, the data block partitioning module, when determining the data size of the data block corresponding to each sub-thread based on the number of the multiple sub-threads and the total data volume, is used for:

[0155] The data volume of each sub-thread is determined by the quotient of the total data volume and the number of sub-threads.

[0156] The specific implementation process of the functions and roles of each module in the above device can be found in the implementation process of the corresponding steps in the above method, and will not be repeated here.

[0157] The fifth aspect of this disclosure provides a computer program product including a computer program / instructions that, when executed by a processor, implement the method as described in the first or second aspect.

[0158] For the device embodiments and computer program product embodiments, since they basically correspond to the method embodiments, the relevant parts can be referred to in the description of the method embodiments. Furthermore, the device embodiments described above are merely illustrative; the modules described as separate components may or may not be physically separate, and the components shown as modules may or may not be physical modules, i.e., they may be located in one place or distributed across multiple network modules. Some or all of the modules can be selected to achieve the purpose of this disclosure according to actual needs. Those skilled in the art can understand and implement this without any inventive effort.

[0159] Sixthly, embodiments of the data processing apparatus provided in this disclosure can be applied to computer equipment. See also... Figure 9 The illustration exemplifies a hardware schematic of a computer device. For example, device 900 could be a mobile phone, computer, digital broadcasting terminal, messaging device, game console, tablet device, medical device, fitness equipment, personal digital assistant, etc.

[0160] Device 900 may include one or more of the following components: processing component 901, memory 902, power supply component 903, multimedia component 904, audio component 905, input / output (I / O) interface 906, sensor component 907, and communication component 908.

[0161] Processing component 901 typically controls the overall operation of device 900, such as operations associated with display, telephone calls, data communication, camera operation, and recording. Processing component 901 may include one or more processors 909 to execute instructions to complete all or part of the steps of the methods described above. Furthermore, processing component 901 may include one or more modules to facilitate interaction between processing component 901 and other components. For example, processing component 901 may include a multimedia module to facilitate interaction between multimedia component 904 and processing component 901.

[0162] Memory 902 is configured to store various types of data to support the operation of device 900. Examples of this data include instructions for any application or method operating on device 900, contact data, phonebook data, messages, pictures, videos, etc. Memory 902 can be implemented by any type of volatile or non-volatile storage device or a combination thereof, such as static random access memory (SRAM), electrically erasable programmable read-only memory (EEPROM), erasable programmable read-only memory (EPROM), programmable read-only memory (PROM), read-only memory (ROM), magnetic storage, flash memory, magnetic disk, or optical disk.

[0163] The power supply component 903 provides power to the various components of the device 900. The power supply component 903 may include a power management system, one or more power sources, and other components associated with generating, managing, and distributing power to the device 900.

[0164] Multimedia component 904 includes a screen that provides an output interface between the device 900 and the user. In some embodiments, the screen may include a liquid crystal display (LCD) and a touch panel (TP). If the screen includes a touch panel, the screen may be implemented as a touchscreen to receive input signals from the user. The touch panel includes one or more touch sensors to sense touch, swipe, and gestures on the touch panel. The touch sensors may sense not only the boundaries of the touch or swipe action but also the duration and pressure associated with the touch or swipe operation. In some embodiments, multimedia component 904 includes a front-facing camera and / or a rear-facing camera. When the device 900 is in an operating mode, such as a shooting mode or a video mode, the front-facing camera and / or the rear-facing camera may receive external multimedia data. Each front-facing camera and rear-facing camera may be a fixed optical lens system or have focal length and optical zoom capabilities.

[0165] Audio component 905 is configured to output and / or input audio signals. For example, audio component 905 includes a microphone (MIC) configured to receive external audio signals when device 900 is in an operating mode, such as call mode, recording mode, and voice recognition mode. The received audio signals may be further stored in memory 902 or transmitted via communication component 908. In some embodiments, audio component 905 also includes a speaker for outputting audio signals.

[0166] I / O interface 906 provides an interface between processing component 901 and peripheral interface modules, such as keyboards, click wheels, buttons, etc. These buttons may include, but are not limited to, home buttons, volume buttons, power buttons, and lock buttons.

[0167] Sensor assembly 907 includes one or more sensors for providing status assessments of various aspects of device 900. For example, sensor assembly 907 can detect the on / off state of device 900, the relative positioning of components such as the display and keypad of device 900, changes in the position of device 900 or a component of device 900, the presence or absence of user contact with device 900, the orientation or acceleration / deceleration of device 900, and temperature changes of device 900. Sensor assembly 907 may also include a proximity sensor configured to detect the presence of nearby objects without any physical contact. Sensor assembly 907 may also include a light sensor, such as a CMOS or CCD image sensor, for use in imaging applications. In some embodiments, sensor assembly 907 may also include an accelerometer, a gyroscope, a magnetometer, a pressure sensor, or a temperature sensor.

[0168] Communication component 908 is configured to facilitate wired or wireless communication between device 900 and other devices. Device 900 can access wireless networks based on communication standards, such as Wi-Fi, 2G or 3G, 4G or 5G, or combinations thereof. In one exemplary embodiment, communication component 908 receives broadcast signals or broadcast-related information from an external broadcast management system via a broadcast channel. In one exemplary embodiment, communication component 908 also includes a near-field communication (NFC) module to facilitate short-range communication. For example, the NFC module may be implemented based on radio frequency identification (RFID) technology, Infrared Data Association (IRDA) technology, ultra-wideband (UWB) technology, Bluetooth (BT) technology, and other technologies.

[0169] In an exemplary embodiment, device 900 may be implemented by one or more application-specific integrated circuits (ASICs), digital signal processors (DSPs), digital signal processing devices (DSPDs), programmable logic devices (PLDs), field-programmable gate arrays (FPGAs), controllers, microcontrollers, microprocessors, or other electronic components to perform the data processing method of the computer device described above.

[0170] In a seventh aspect, this disclosure also provides a non-transitory computer-readable storage medium including instructions, such as a memory 902 including instructions, which can be executed by a processor 909 of device 900 to complete the data processing method of the computer device described above. For example, the non-transitory computer-readable storage medium may be a ROM, random access memory (RAM), CD-ROM, magnetic tape, floppy disk, and optical data storage device, etc.

[0171] The foregoing has described specific embodiments of this disclosure. Other embodiments are within the scope of the appended claims. In some cases, the actions or steps recited in the claims may be performed in a different order than that shown in the embodiments and may still achieve the desired results. Furthermore, the processes depicted in the drawings do not necessarily require the specific or sequential order shown to achieve the desired results. In some embodiments, multiple data processing and parallel processing are also possible or may be advantageous.

[0172] Other embodiments of this disclosure will readily occur to those skilled in the art upon consideration of the specification and practice of the invention applied herein. This disclosure is intended to cover any variations, uses, or adaptations of this disclosure that follow the general principles of this disclosure and include common knowledge or customary techniques in the art not claimed herein. The specification and examples are to be considered exemplary only, and the true scope and spirit of this disclosure are indicated by the following claims.

[0173] It should be understood that this disclosure is not limited to the precise structures described above and shown in the accompanying drawings, and various modifications and changes can be made without departing from its scope. The scope of this disclosure is limited only by the appended claims.

[0174] The above description is merely a preferred embodiment of this disclosure and is not intended to limit this disclosure. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of this disclosure should be included within the scope of protection of this disclosure.

Claims

1. A data processing method, characterized in that, The method, applied to one of multiple child threads, includes: Receive the values ​​of the header and tail variables configured by the main thread; The data block corresponding to the sub-thread is processed according to the header variable and the tail variable; The header variable and the tail variable are both atomic variables. The header variable represents the start position of the data block corresponding to the sub-thread in the total data, and the tail variable represents the end position of the data block corresponding to the sub-thread in the total data. The main thread is used to divide the total data into multiple data blocks that correspond one-to-one with the multiple sub-threads.

2. The data processing method according to claim 1, characterized in that, The step of processing the data block corresponding to the child thread based on the header variable and the tail variable includes: The data unit indicated by the header variable is processed; The head variable is updated in the direction of the tail variable so that the head variable indicates the next data unit, until the head variable coincides with the tail variable.

3. The data processing method according to claim 2, characterized in that, The method further includes: In the case where the head variable and the tail variable overlap, determine whether there is a sub-thread among the plurality of sub-threads whose head variable and tail variable do not overlap; In the case where any of the multiple sub-threads has a non-overlapping head variable and tail variable, the data block determined by the head variable and tail variable of the non-overlapping sub-thread is divided into two sub-data blocks, and either of the sub-data blocks is transferred to the sub-thread for processing, and the head variable or tail variable of the non-overlapping sub-thread is updated.

4. The data processing method according to claim 3, characterized in that, The step of transferring any of the sub-data blocks to the sub-thread for processing, and updating the head or tail variable of any sub-thread, includes: The sub-data block closest to the tail variable of either of the two sub-data blocks is transferred to that sub-thread for processing, and the tail variable of that sub-thread is updated.

5. The data processing method according to claim 3, characterized in that, The process of dividing the data block determined by the head and tail variables of any of the child threads into two sub-data blocks includes: Determine an intermediate variable between the head variable and the tail variable of any child thread, wherein the intermediate variable is the middle value between the head variable and the tail variable of any child thread; The data block determined by the head variable of any child thread and the intermediate variable, and the data block determined by the intermediate variable and the tail variable of any child thread are respectively regarded as two sub-data blocks.

6. The data processing method according to claim 3, characterized in that, The multiple sub-threads have thread numbers; Determining whether there is a sub-thread among the plurality of sub-threads whose head variable and tail variable do not overlap includes: Based on the increasing or decreasing relationship of the thread numbers, determine in turn whether the head and tail variables of the multiple sub-threads overlap.

7. The data processing method according to claim 1, characterized in that, The main thread is used for: The plurality of sub-threads are determined, and the data volume of the data block corresponding to each sub-thread is determined based on the number of the plurality of sub-threads and the total data volume; Based on the amount of data in the data block corresponding to each sub-thread, the total data is divided into multiple data blocks that correspond one-to-one with the multiple sub-threads.

8. A data processing method, characterized in that, The method includes: The total data is divided into multiple data blocks, each corresponding to one of the multiple sub-threads; Configure the values ​​of the header and tail variables for each data block corresponding to the sub-thread, so that the sub-thread processes the data block corresponding to the sub-thread according to the values ​​of the header and tail variables; Both the header variable and the tail variable are atomic variables. The header variable represents the beginning position of the data block corresponding to the sub-thread in the total data, and the tail variable represents the end position of the data block corresponding to the sub-thread in the total data.

9. The data processing method according to claim 8, characterized in that, The process of dividing the total data into multiple data blocks, each corresponding to one of the multiple sub-threads, includes: Multiple sub-threads are identified, and the data volume of the data block corresponding to each sub-thread is determined based on the number of the multiple sub-threads and the total data volume. Based on the amount of data in the data block corresponding to each sub-thread, the total data is divided into multiple data blocks that correspond one-to-one with the multiple sub-threads.

10. The data processing method according to claim 9, characterized in that, The step of determining the data size of each data block corresponding to each sub-thread based on the number of the multiple sub-threads and the total data size includes: The data volume of each sub-thread is determined by the quotient of the total data volume and the number of sub-threads.

11. A data processing apparatus, characterized in that, The device, corresponding to one of a plurality of child threads, includes: Configure the receiving module to receive the values ​​of the header and tail variables configured by the main thread; The data block processing module is used to process the data block corresponding to the sub-thread according to the header variable and the tail variable; The header variable and the tail variable are both atomic variables. The header variable represents the start position of the data block corresponding to the sub-thread in the total data, and the tail variable represents the end position of the data block corresponding to the sub-thread in the total data. The main thread is used to divide the total data into multiple data blocks that correspond one-to-one with the multiple sub-threads.

12. A data processing apparatus, characterized in that, The device includes: The data block partitioning module is used to divide the total data into multiple data blocks that correspond one-to-one with multiple sub-threads; A configuration sending module is used to configure the values ​​of the header and tail variables of the sub-thread corresponding to each data block, so that the sub-thread processes the data block corresponding to the sub-thread according to the values ​​of the header and tail variables; Both the header variable and the tail variable are atomic variables. The header variable represents the beginning position of the data block corresponding to the sub-thread in the total data, and the tail variable represents the end position of the data block corresponding to the sub-thread in the total data.

13. A computer program product comprising a computer program / instructions, characterized in that, When the computer program / instructions are executed by the processor, they implement the method described in any one of claims 1 to 10.

14. A computer device, characterized in that, It includes a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor, when executing the program, implements the method as described in any one of claims 1 to 10.

15. A computer-readable storage medium having a computer program stored thereon, characterized in that, When the program is executed by the processor, it implements the method as described in any one of claims 1 to 10.

Images