Data processing method and device, equipment and storage medium

Abstract

The application discloses a data processing method and device, equipment and a storage medium, and relates to the technical field of data processing. The method comprises the following steps: creating a collection coroutine pool corresponding to each query time slice in N query time slices of a query task according to the N query time slices, wherein the collection coroutine pool comprises M first coroutines; reading query data related to each query time slice from a database through the M first coroutines in the collection coroutine pool corresponding to each query time slice; and when a target collection coroutine pool exists and the time of a target query time slice corresponding to the target collection coroutine pool matches the time of a first query time slice carried by a notification pipeline, reading the query data related to the target query time slice to a data pipeline to send the data in the data pipeline to a transfer service. In this way, the efficiency of querying data from the database is improved, and the energy consumption of querying data from the database is reduced by concurrently processing the data of the N query time slices.

Description

Technical Field

[0001] This application belongs to the field of data processing technology, and in particular relates to a data processing method, apparatus, device and storage medium. Background Technology

[0002] Data Transfer Express (DTEx) is a platform for real-time data distribution and batch data transfer for service systems. This platform needs to be able to query data from the database according to the time slice dimension and push the queried data to the transfer service.

[0003] In related technologies, the process of querying data from a database according to a time slice dimension involves reading data corresponding to each time slice sequentially from the database according to the preset start time and time slice size. For example, if the preset start time of the time slice is 00:00 on January 1st and the time slice size is 10 minutes, then the data from 00:00 to 00:10 is read from the database first. Only after the data from 00:10 to 00:20 can the data from 00:10 to 00:20 be read. This increases the time and energy consumption of data querying and reduces the efficiency of data querying from the database. Summary of the Invention

[0004] This application provides a data processing method, apparatus, device, and storage medium that can solve the problems of high energy consumption and low efficiency in querying data from databases in related technologies.

[0005] In a first aspect, embodiments of this application provide a data processing method, which may include:

[0006] Obtain the N query time slices for the query task;

[0007] Based on N query time slices, create N collection coroutine pools. The N collection coroutine pools include the collection coroutine pool corresponding to each query time slice in the N query time slices. The collection coroutine pool includes M first coroutines.

[0008] By collecting M first coroutines from the Ctrip pool corresponding to each query time slice, query data related to each query time slice is read from the database.

[0009] If a target collection goroutine pool exists among the N collection goroutine pools and the time of the target query time slice corresponding to the target collection goroutine pool matches the time of the first query time slice carried by the notification pipeline, the query data related to the target query time slice is read into the data pipeline to send the data in the data pipeline to the relay service. Each first goroutine in the target collection goroutine pool is in the first state.

[0010] Secondly, embodiments of this application provide a data processing apparatus, which may include:

[0011] The acquisition module is used to acquire N query time slices for a query task;

[0012] A creation module is used to create N collection coroutine pools based on N query time slices. The N collection coroutine pools include the collection coroutine pool corresponding to each query time slice in the N query time slices, and the collection coroutine pool includes M first coroutines.

[0013] The reading module is used to read query data related to each query time slice from the database by collecting M first coroutines in the Ctrip pool corresponding to each query time slice;

[0014] The reading module can also be used to read query data related to the target query time slice into the data pipeline when there is a target collection goroutine pool among the N collection goroutine pools and the time of the target query time slice corresponding to the target collection goroutine pool matches the time of the first query time slice carried by the notification pipeline, so as to send the data in the data pipeline to the relay service. Each first goroutine in the target collection goroutine pool is in the first state.

[0015] Thirdly, embodiments of this application provide a computer device, which includes: a processor and a memory storing computer program instructions;

[0016] When the processor executes computer program instructions, it implements the data processing method as described in the first aspect.

[0017] Fourthly, embodiments of this application provide a computer storage medium storing computer program instructions, which, when executed by a processor, implement the data processing method as described in the first aspect.

[0018] Fifthly, embodiments of this application provide a chip, which includes a processor and a communication interface. The communication interface and the processor are coupled, and the processor is used to run programs or instructions to implement the data processing method as shown in the first aspect.

[0019] In a sixth aspect, embodiments of this application provide a computer program product stored in a storage medium, which is executed by at least one processor to implement the data processing method as described in the first aspect.

[0020] The data processing method, apparatus, device, and storage medium of this application embodiment obtain N query time slices for a query task; based on the N query time slices, N collection coroutine pools are created, each collection coroutine pool including a collection coroutine pool corresponding to each of the N query time slices, and each collection coroutine pool including M first coroutines; through the M first coroutines in the collection coroutine pool corresponding to each query time slice, query data related to each query time slice is read from the database. This allows for concurrent data processing through the M first coroutines in the collection coroutine pool corresponding to each query time slice, significantly reducing waiting time during time slice processing, improving the efficiency of querying data from the database, and reducing the energy consumption of querying data from the database. Then, if a target collection coroutine pool exists among the N collection coroutine pools and the time of the target query time slice corresponding to the target collection coroutine pool matches the time of the first query time slice carried by the notification pipe, the query data related to the target query time slice is read into the data pipe to send the data in the data pipe to the relay service. Each first coroutine in the target collection coroutine pool is in a first state. Therefore, by leveraging the characteristics of single-threaded serial processing, the correctness of packet assembly is achieved without affecting the efficiency of querying data from the database, thus avoiding the problem of out-of-order data in time slices. Attached Figure Description

[0021] To more clearly illustrate the technical solutions of the embodiments of this application, the accompanying drawings used in the embodiments of this application will be briefly introduced below. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0022] Figure 1 A flowchart illustrating a data processing method provided in an embodiment of this application;

[0023] Figure 2 This is a schematic diagram of the data processing method of the go program in a data processing method provided in an embodiment of this application;

[0024] Figure 3 This is a schematic diagram of the structure of a data processing apparatus provided in one embodiment of this application;

[0025] Figure 4 This is a schematic diagram of the structure of a computer device provided in one embodiment of this application. Detailed Implementation

[0026] The features and exemplary embodiments of various aspects of this application will be described in detail below. To make the objectives, technical solutions, and advantages of this application clearer, the application will be further described in detail below with reference to the accompanying drawings and specific embodiments. It should be understood that the specific embodiments described herein are only intended to explain this application and not to limit it. For those skilled in the art, this application can be implemented without some of these specific details. The following description of the embodiments is merely to provide a better understanding of this application by illustrating examples.

[0027] It should be noted that, in this document, relational terms such as "first" and "second" are used merely to distinguish one entity or operation from another, and do not necessarily require or imply any such actual relationship or order between these entities or operations. Furthermore, the terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such a process, method, article, or apparatus. Without further limitations, an element defined by the phrase "comprising..." does not exclude the presence of additional identical elements in the process, method, article, or apparatus that includes the element.

[0028] The acquisition, storage, use, and processing of data (including but not limited to features and information mentioned in this document) in the technical solution of this application all comply with the relevant provisions of national laws and regulations.

[0029] To address the problems encountered in related technologies, this application proposes a data synchronization scheme with preprocessable time slices. This scheme can efficiently extract and send database data. It can reduce the waiting time between time slices by preprocessing different time slices in parallel, ensure the correctness of packet assembly by relying on the serial processing characteristics of a single coroutine, and enable the sequential packetization and transmission of data between time slices. This ensures that the data between time slices is sent in sequence and avoids out-of-order processing.

[0030] The following will be combined with the appendix Figures 1 to 4 This application describes in detail the methods, apparatus, computer devices, and storage media of the embodiments. It should be noted that these embodiments are not intended to limit the scope of this application.

[0031] The following is combined with Figure 1 The data processing method shown will be explained in detail.

[0032] Figure 1 This is a flowchart illustrating a data processing method provided in an embodiment of this application.

[0033] like Figure 1 As shown, this data processing method can be applied to computer equipment, and the data processing method may specifically include the following steps:

[0034] Step 110: Obtain N query time slices for the query task; Step 120: Create N collection coroutine pools based on the N query time slices. Each collection coroutine pool includes a collection coroutine pool corresponding to each of the N query time slices, and each collection coroutine pool includes M first coroutines, where M is a positive integer; Step 130: Read query data related to each query time slice from the database using the M first coroutines in the collection coroutine pool corresponding to each query time slice; Step 140: If a target collection coroutine pool exists among the N collection coroutine pools and the time of the target query time slice corresponding to the target collection coroutine pool matches the time of the first query time slice carried by the notification pipeline, read the query data related to the target query time slice into the data pipeline to send the data in the data pipeline to the relay service. Each first coroutine in the target collection coroutine pool is in a first state.

[0035] In this way, N collection coroutine pools can be created based on the N query time slices of the query task. Each of the N collection coroutine pools includes a collection coroutine pool corresponding to each of the N query time slices, and each collection coroutine pool includes M first coroutines. Through the M first coroutines in the collection coroutine pool corresponding to each query time slice, query data related to each query time slice is read from the database. This allows for concurrent data processing using the M first coroutines in the collection coroutine pool corresponding to each query time slice, significantly reducing waiting time during time slice processing, improving the efficiency of data retrieval from the database, and reducing the energy consumption of data retrieval from the database. Then, if a target collection coroutine pool exists among the N collection coroutine pools and the time of the target query time slice corresponding to the target collection coroutine pool matches the time of the first query time slice carried by the notification pipeline, the query data related to the target query time slice is read into the data pipeline and sent to the relay service. Each first coroutine in the target collection coroutine pool is in a first state. Therefore, by leveraging the characteristics of single-threaded serial processing, the correctness of packet assembly is achieved without affecting the efficiency of querying data from the database, thus avoiding the problem of out-of-order data in time slices.

[0036] The steps described above are explained in detail below.

[0037] First, regarding step 110, in some embodiments of this application, the number of query time slices N that need to be preprocessed is set, where N is a positive integer.

[0038] It should be noted that database data query is the most time-consuming part in this application embodiment. Therefore, if the machine memory allows, the number of concurrent query time slices can be increased, that is, N can be appropriately increased.

[0039] Next, regarding step 120, in some embodiments of this application, step 120 may specifically include steps 1201 and 1202.

[0040] Step 1201: Based on the N query time slices, create a second coroutine corresponding to each of the N query time slices.

[0041] Specifically, by querying the main coroutine of the task, a corresponding second coroutine can be created for each of the N query time slices.

[0042] For example, if N is 3, that is, the 3 query time slices can be query time slice 1, query time slice 2 and query time slice 3, then a corresponding second coroutine 1 can be created for query time slice 1, a corresponding second coroutine 2 can be created for query time slice 2, and a corresponding second coroutine 3 can be created for query time slice 3.

[0043] Step 1202: Create a collection goroutine pool corresponding to each query time slice by using the second goroutine corresponding to each query time slice.

[0044] For example, still using the above embodiment as an example, querying time slice 1 creates a corresponding second coroutine 1, which can be used to create a collection coroutine pool 1 corresponding to query time slice 1. Similarly, querying time slice 2 creates a corresponding second coroutine 2, which can be used to create a collection coroutine pool 2 corresponding to query time slice 2. And querying time slice 3 creates a corresponding second coroutine 3, which can be used to create a collection coroutine pool 3 corresponding to query time slice 3.

[0045] Furthermore, regarding step 130, in some embodiments of this application, step 130 may specifically include:

[0046] By using each first coroutine in the collection coroutine pool corresponding to each query time slice, data related to 1 / M query time slices in each query time slice is read from the database.

[0047] For example, taking the above embodiment as an example, the collection goroutine pool 1 corresponding to the query time slice 1 may include M first goroutines. If M is 3, the 3 first goroutines in the collection goroutine pool 1 may include first goroutine 1, first goroutine 2 and first goroutine 3. In this way, first goroutine 1 can read the relevant data in 1 / 3 of the query time slice in the query time slice 1.

[0048] It should be noted that other query time slices refer to each first coroutine in the collection coroutine pool 1 corresponding to the aforementioned query time slice 1, and read the data related to the 1 / M query time slice in query time slice 1 from the database, which will not be elaborated here.

[0049] Furthermore, in some embodiments of this application, prior to step 140, a process of creating a notification pipeline is included to read the time of the first query time slice in the notification pipeline after all the aforementioned arbitrary acquisition coroutine pools have finished processing.

[0050] Based on this, the notification pipeline includes N notification pipelines, and the N notification pipelines include the notification pipeline corresponding to each query time slice in the N query time slices. The data processing method may also include steps 2101 to 2103.

[0051] Step 2101: Based on the N query time slices, create an initial notification pipeline corresponding to each of the N query time slices.

[0052] For example, taking the above embodiment as an example, if N is 3, that is, the 3 query time slices can be query time slice 1, query time slice 2 and query time slice 3, then the initial notification pipeline 1 corresponding to query time slice 1, the initial notification pipeline 2 corresponding to query time slice 2 and the initial notification pipeline 3 corresponding to query time slice 3 can be created.

[0053] Step 2102: Select the first query time slice from N query time slices.

[0054] For example, taking the above embodiment as an example, the first query time slice, such as query time slice 1, is selected from the three query time slices.

[0055] Step 2103: Write the start time of the first query time slice and the size of the first query time slice into the initial notification pipe corresponding to each query time slice, to obtain N notification pipes.

[0056] For example, the start time of query time slice 1, such as 2021-10-19 10:24:30.000, and the size of the first query time slice, such as 10 minutes, can be written into initial notification pipe 1, initial notification pipe 2, and initial notification pipe 3 respectively, so that each initial notification pipe can calculate the time (pub_ts) of the first query time slice. In this way, three notification pipes are obtained, namely notification pipe 1, notification pipe 2, and notification pipe 3.

[0057] Here, in this embodiment of the application, the notification pipe has read and write functions. If no data, such as the start time of the query time slice and the size of the first query time slice, is written into the notification pipe, the second coroutine corresponding to each query time slice will be blocked and unable to perform subsequent steps until the start time of the query time slice and the size of the first query time slice are written into the pipe. If this step is not performed, all second coroutines will be blocked on the notification pipe and unable to perform subsequent steps.

[0058] Therefore, after this step, each second coroutine can check whether the time of the first query time slice carried in the notification pipe is the same as the time of the query time slice currently being processed by the second coroutine. If they are the same, step 140 is executed; otherwise, the coroutine continues to block. It can be seen that only one second coroutine can enter step 140, while the other N-1 second coroutines are blocked again on the notification pipe.

[0059] Since the most time-consuming data query part in this embodiment has been preprocessed, other coroutines are only waiting for the final packaging and sending steps, which significantly improves the parallel processing capability, significantly reduces the waiting time during time slice processing, improves the efficiency of querying data from the database, and reduces the energy consumption of querying data from the database. For example, if there are 3 query time slices, and the processing time of each query time slice is 1 second, 2 seconds, and 3 seconds, if a serial processing method is used, without considering the subsequent packet sending time, the data query time of the system will be 6 seconds. However, after using M first coroutines in the acquisition process pool corresponding to each query time slice to process the data concurrently, the data query time is reduced to 3 seconds.

[0060] Then, regarding step 140, in some embodiments of this application, before step 140, the data processing method may further include step 150, temporarily storing the second coroutine corresponding to each query time slice into the notification pipeline.

[0061] Based on this, step 140 may specifically include:

[0062] If a target collection coroutine pool exists among the N collection coroutine pools, and the time of the target query time slice corresponding to the target collection coroutine pool matches the time of the first query time slice carried by the notification pipeline, the query data related to the target query time slice is read into the data pipeline by creating a second coroutine of the target collection coroutine pool.

[0063] For example, when the second coroutine that creates the target acquisition coroutine pool determines that the time of the first query time slice carried in the notification pipeline is the same as the time of the query time slice currently being processed by the second coroutine, the second coroutine that creates the target acquisition coroutine pool can read the query data related to the target query time slice into the data pipeline.

[0064] In some other embodiments of this application, the first state is the state in which the first coroutine has completed reading. Before reading the query data related to the target query time slice into the data pipeline in step 140, the data processing method further includes steps 3101 to 3103.

[0065] Step 3101: Monitor the running status of each first coroutine in the collection coroutine pool corresponding to each query time slice through the second coroutine corresponding to each query time slice.

[0066] For example, still using the above embodiment as an example, if query time slice 1 corresponds to second coroutine 1, query time slice 2 corresponds to second coroutine 2, and query time slice 3 corresponds to second coroutine 3. Second coroutine 1 monitors the running status of each first coroutine in the collection coroutine pool 1 corresponding to query time slice 1, second coroutine 2 monitors the running status of each first coroutine in the collection coroutine pool 2 corresponding to query time slice 2, and second coroutine 3 monitors the running status of each first coroutine in the collection coroutine pool 3 corresponding to query time slice 3.

[0067] Step 3102: When the running state of each first coroutine in the acquisition coroutine pool is in the first state, the acquisition coroutine pool is determined as the target acquisition coroutine pool.

[0068] For example, if the second coroutine 3 detects that the running status of each first coroutine in the collection coroutine pool 3 corresponding to the query time slice 3 has read all the relevant data within the query time slice 3, then the collection coroutine pool 3 can be determined as the target collection coroutine pool.

[0069] Step 3103: By creating a second coroutine in the target acquisition coroutine pool, the time of the first query time slice carried by the notification pipeline is read.

[0070] For example, a second coroutine 3 can be created in the acquisition coroutine pool 3 to read the time of the first query time slice carried by N notification pipelines. The second coroutine 3 then determines whether the time of the first query time slice carried in the notification pipeline is the same as the time of the query time slice currently being processed by the second coroutine 3. If they are the same, step 140 is executed; otherwise, the process continues to block, i.e., the following steps are executed. In some embodiments of this application, after step 130, the data processing method may further include:

[0071] If a target collection coroutine pool exists among the N collection coroutine pools, and the time of the target query time slice corresponding to the target collection coroutine pool does not match the time of the first query time slice carried by the notification pipeline, the second coroutine corresponding to the target query time slice will be temporarily stored in the notification pipeline; wherein, the second coroutine corresponding to the target query time slice is used to create the target collection coroutine pool.

[0072] Furthermore, based on the above steps, after step 140, the data processing method may further include the process of sending data to the relay service. Based on this, the data processing method may further include steps 4101 to 4103.

[0073] Step 4101: Using a packet assembly coroutine, the data in the data pipeline is encapsulated according to a preset quantity to obtain a query data packet corresponding to the preset quantity.

[0074] For example, the coroutine entering this step writes all the query data retrieved in step 130 above into the data pipeline, and then starts a packet assembly coroutine. The purpose of the packet assembly coroutine is to combine the read query data into one or more query data packets according to the required number of packets. For example, if the current query time slice 1 retrieves 89 database records, and each query data packet is required to contain 10 records, then the packet assembly coroutine will split the 89 database records into 9 data packets and write them into the data pipeline storing the packet data. Among them, 8 query data packets contain 10 database records each, and 1 query data packet contains 9 database records.

[0075] Therefore, the acquired data can support multiple packaged transmission relay services, shortening the encapsulation time and improving the efficiency of sending data in the data pipeline to the relay service.

[0076] Step 4102: Read query data packets from the data pipeline by sending a third coroutine from the coroutine pool.

[0077] Step 4103: Send a query data packet to the relay service.

[0078] Therefore, this application adopts a single-coroutine serial processing method, which can ensure that the number of data records in multiple data packets is less than the required number of data records in the packet during the packet assembly process. That is, at most, there will be only one data packet with fewer records than the required number of records in the packet. The reason why single-coroutine serial processing can be used here is that it only involves memory copying and does not involve complex central processing unit operations, so it will not affect the processing performance.

[0079] In addition, after step 4102, the data processing method may also include steps 4104 and 4105.

[0080] Step 4104: If the query data packet is not included in the detection data pipeline, select the second query time slice from N query time slices.

[0081] Step 4105: Adjust the time of the first query time slice carried by the notification pipeline to match the time of the second query time slice.

[0082] For example, when the query data packet in the data pipeline is consumed and sent, the main coroutine writes the start time of the next query time slice to be processed, i.e., the second query time slice (e.g., query time slice 2), and the size of the first query time slice into the notification pipeline corresponding to each query time slice. At this time, the second coroutine blocked in the notification pipeline is reactivated, reads the time of the second query time slice in the notification pipeline, and then the coroutine with the same time slice time as the current second coroutine 2 continues to execute the above step 140, while other coroutines continue to be blocked, thereby realizing the continuous advancement of time slices.

[0083] This ensures that data is sent sequentially between time slices, preventing out-of-order processing.

[0084] The following uses the Go language as an example to illustrate the data processing method provided in the embodiments of this application, as shown below.

[0085] like Figure 2 As shown, the data processing method can be divided into steps 1 to 7.

[0086] Step 1: Set the number of query time slices N, where N>=1. Database data query is the most time-consuming part in this embodiment. Therefore, if the machine memory allows, N can be appropriately increased to increase the number of concurrent query time slices.

[0087] Step 2: Open N notification pipes to notify each collection goroutine pool of the time of the current query time slice (pub_ts), and write the start time of the time slice, for example, the time of query time slice 1: 2021-10-19 10:24:30.000, into the N notification pipes. The pipes have read and write functions. If there is no data in the pipe, the goroutine reading the pipe will be blocked until data is written into the pipe.

[0088] Step 3: Start N second coroutines. Each second coroutine starts a collection coroutine pool. Each collection coroutine pool has M first coroutines. Each first coroutine in the collection coroutine pool processes 1 / M of the data within the current time slice. Once all first coroutines in the collection coroutine pool have finished processing, they read the time of the first query time slice from the notification pipe. Without step 2, all N second coroutines would be blocked on the notification pipe and unable to proceed with subsequent steps. The N second coroutines read the time slice information (pub_ts) written in step 2.

[0089] Step 4: Each of the N second coroutines checks whether the read time slice information (pub_ts) is the same as the time slice of the query it is currently processing. If they are the same, proceed to step 5; otherwise, continue blocking. Here, only one second coroutine can enter step 5; the other N-1 second coroutines are blocked again on the notification pipe.

[0090] Step 5: The second coroutine in this step writes all the query data from Step 2 into the data storage pipeline, and then starts a packet assembly coroutine. The purpose of the packet assembly coroutine is to combine the queried data into one or more request packets according to the required number of packets. For example, if the current time slice query returns 89 database records, and each packet is required to contain 10 records, then the packet assembly coroutine will split the 89 database records into 9 data packets and write them into the data storage pipeline. Eight packets contain 10 database records each, and one packet contains 9 database records.

[0091] Step 6: Start a sending goroutine pool containing P third goroutines. The P third goroutines concurrently send data to the data pipes storing packet data. When the data in the packet data pipes is consumed, the main goroutine writes the next time slice information (pub_ts) to be processed, such as the time of querying time slice 2, into the N notification pipes created in step 2.

[0092] Step 7: Once data is written to the notification pipe, the second coroutine that was blocked in Step 3 is reactivated and reads the time slice information (pub_ts) from the notification pipe.

[0093] Then, the coroutine with the same time slice as the current coroutine continues to execute steps 4 to 7, while other coroutines continue to be blocked, thus enabling the query time slice to be continuously advanced.

[0094] As can be seen, in this embodiment of the application, concurrent data preprocessing can be achieved through the above steps 2, 3 and 4, which can significantly reduce the system's time slice processing waiting time. For example, if three time slices need to be processed, and the processing time for each time slice is 1 second, 2 seconds and 3 seconds, if serial processing is used, without considering the subsequent packet sending time, the system's data query time will take 6 seconds. However, after adopting the above scheme, the system's data query time is reduced to 3 seconds.

[0095] Steps 4 and 5 employ a single-coroutine serial processing method, ensuring that the packet assembly process will not result in multiple packets containing fewer data records than the required number of records. In other words, at most only one packet will have fewer records than the required number. The reason for using single-coroutine serial processing here is that it only involves memory copying and does not involve complex CPU operations, thus not impacting processing performance.

[0096] The design of the notification pipeline and notification mechanism involved in steps 2, 3, 4, 5, 6 and 7 ensures that data between time slices is sent sequentially and that out-of-order processing will not occur.

[0097] In summary, the embodiments of this application provide a data synchronization scheme that can preprocess time slices, which can support concurrent preprocessing of multiple time slices of data to improve the overall synchronization performance of the system; and, by utilizing the characteristics of single-threaded serial processing, the correctness of packet assembly is achieved without affecting processing performance; and, the notification mechanism ensures that there will be no out-of-order problem of time slice data when preprocessing multiple time slices, thereby improving the processing efficiency of multiple time slices.

[0098] This application also provides a data processing apparatus, specifically combined with... Figure 3 Please provide a detailed explanation.

[0099] Figure 3 This is a schematic diagram of the structure of a data processing apparatus provided in one embodiment of this application.

[0100] In some embodiments of this application, Figure 3 The data processing device shown can be installed in a computer device.

[0101] like Figure 3 As shown, the data processing device 30 may specifically include:

[0102] The acquisition module 301 is used to acquire N query time slices of the query task;

[0103] Module 302 is used to create N collection coroutine pools based on N query time slices. The N collection coroutine pools include collection coroutine pools corresponding to each query time slice in the N query time slices, and each collection coroutine pool includes M first coroutines.

[0104] The reading module 303 is used to read query data related to each query time slice from the database by collecting M first coroutines in the Ctrip pool corresponding to each query time slice;

[0105] The reading module 303 is also used to read the query data related to the target query time slice into the data pipeline when there is a target collection goroutine pool among the N collection goroutine pools and the time of the target query time slice corresponding to the target collection goroutine pool matches the time of the first query time slice carried by the notification pipeline, so as to send the data in the data pipeline to the relay service, and each first goroutine in the target collection goroutine pool is in the first state.

[0106] Thus, the data processing apparatus provided in this application embodiment can create N collection coroutine pools based on the N query time slices of the query task. Each of the N collection coroutine pools includes a collection coroutine pool corresponding to each of the N query time slices, and each collection coroutine pool includes M first coroutines. By using the M first coroutines in the collection coroutine pool corresponding to each query time slice, query data related to each query time slice is read from the database. This allows for concurrent data processing using the M first coroutines in the collection coroutine pool corresponding to each query time slice, significantly reducing waiting time during time slice processing, improving the efficiency of querying data from the database, and reducing the energy consumption of querying data from the database. Then, if a target collection coroutine pool exists among the N collection coroutine pools and the time of the target query time slice corresponding to the target collection coroutine pool matches the time of the first query time slice carried by the notification pipe, the query data related to the target query time slice is read into the data pipe to send the data in the data pipe to the relay service. Each first coroutine in the target collection coroutine pool is in a first state. Therefore, by leveraging the characteristics of single-threaded serial processing, the correctness of packet assembly is achieved without affecting the efficiency of querying data from the database, thus avoiding the problem of out-of-order data in time slices.

[0107] The data processing device 70 in the embodiments of this application will be described in detail below.

[0108] In some embodiments of this application, the reading module 303 may be specifically used to read data related to 1 / M query time slices in each query time slice from the database through each first coroutine in the collection coroutine pool corresponding to each query time slice.

[0109] In some embodiments of this application, the data processing device 70 may further include a selection module and a writing module; wherein,

[0110] The creation module 302 can also be used to create an initial notification pipeline corresponding to each of the N query time slices, based on the N query time slices, when the notification pipeline includes N notification pipelines and the N notification pipelines include a notification pipeline corresponding to each query time slice in the N query time slices.

[0111] The selection module is used to select the first query time slice from N query time slices;

[0112] The write module is used to write the start time point and size of the first query time slice to the initial notification pipe corresponding to each query time slice, resulting in N notification pipes.

[0113] In some embodiments of this application, the creation module 302 may be specifically used to create a second coroutine corresponding to each of the N query time slices based on the N query time slices;

[0114] A pool of collection coroutines is created for each query time slice using a second coroutine corresponding to each query time slice.

[0115] In some embodiments of this application, the data processing device 70 may further include a monitoring module and a determination module; wherein,

[0116] The monitoring module is used to monitor the running status of each first coroutine in the collection coroutine pool corresponding to each query time slice by means of the second coroutine corresponding to each query time slice when the first state is that the first coroutine has completed the reading.

[0117] The determination module is used to determine the acquisition process pool as the target acquisition process pool when the running state of each first process in the acquisition process pool is in the first state.

[0118] Specifically, the reading module 303 can be used to read the time of the first query time slice carried by the notification pipeline by creating a second coroutine in the target acquisition coroutine pool.

[0119] In some embodiments of this application, the data processing device 70 may further include a temporary storage module; wherein,

[0120] The temporary storage module is used to temporarily store the second coroutine corresponding to each query time slice into the notification pipeline;

[0121] The reading module 303 is specifically used to read query data related to the target query time slice into the data pipeline when there is a target collection coroutine pool among the N collection coroutine pools and the time of the target query time slice corresponding to the target collection coroutine pool matches the time of the first query time slice carried by the notification pipeline.

[0122] In some embodiments of this application, the data processing device 70 may further include an encapsulation module and a transmission module; wherein,

[0123] The encapsulation module is used to encapsulate the data in the data pipeline according to a preset quantity through the package assembly coroutine, so as to obtain a query data package corresponding to the preset quantity;

[0124] The reading module 303 is also used to read query data packets from the data pipeline by sending a third coroutine from the coroutine pool;

[0125] The sending module is used to send query data packets to the relay service.

[0126] In some embodiments of this application, the data processing device 70 may further include a selection module and an adjustment module; wherein,

[0127] The selection module is used to select a second query time slice from N query time slices when the query data packet is not included in the detection data pipeline;

[0128] The adjustment module is used to adjust the time of the first query time slice carried by the notification pipeline to match the time of the second query time slice.

[0129] In some embodiments of this application, the data processing device 70 may further include a temporary storage module, which is used to temporarily store the second coroutine corresponding to the target query time slice in the notification pipe when there is a target acquisition coroutine pool among the N acquisition coroutine pools and the time of the target query time slice corresponding to the target acquisition coroutine pool does not match the time of the first query time slice carried by the notification pipe; wherein, the second coroutine corresponding to the target query time slice is used to create the target acquisition coroutine pool.

[0130] Based on the same inventive concept, this application also provides a computer device. (Specifically combined with...) Figure 4 Please provide a detailed explanation.

[0131] Figure 4 This is a schematic diagram of the structure of a computer device provided in one embodiment of this application.

[0132] like Figure 4As shown, the computer device may include at least one of the following as described in the embodiments of this application: a user terminal device and a server terminal device. The computer device may include a processor 401 and a memory 402 storing computer program instructions.

[0133] Specifically, the processor 401 may include a central processing unit (CPU), an application-specific integrated circuit (ASTC), or one or more integrated circuits that can be configured to implement the embodiments of this application.

[0134] Memory 402 may include a large-capacity memory for data or instructions. For example, and not limitingly, memory 402 may include a hard disk drive (HDD), a floppy disk drive, flash memory, optical disk, magneto-optical disk, magnetic tape, or a Universal Serial Bus (USB) drive, or a combination of two or more of these. Where appropriate, memory 402 may include removable or non-removable (or fixed) media. Where appropriate, memory 402 may be internal or external to the integrated gateway disaster recovery device. In a particular embodiment, memory 402 is non-volatile solid-state memory. In a particular embodiment, memory 402 includes solid-state storage (ROM). Where appropriate, the ROM may be a mask-programmed ROM, a programmable ROM (PROM), an erasable PROM (EPROM), an electrically erasable PROM (EEPROM), an electrically rewritable ROM (EAROM), or flash memory, or a combination of two or more of these.

[0135] The processor 401 implements any of the data processing methods described in the above embodiments by reading and executing computer program instructions stored in the memory 402.

[0136] In one example, the computer device may also include a communication interface 403 and a bus 410. Wherein, as... Figure 4 As shown, the processor 401, memory 402, and communication interface 403 are connected through bus 410 and complete communication with each other.

[0137] The communication interface 403 is mainly used to realize communication between various modules, devices, units and / or equipment in the embodiments of this application.

[0138] Bus 410 includes hardware, software, or both, that couples components of a flow control device together. For example, and not limitingly, the bus may include an Accelerated Graphics Port (AGP) or other graphics bus, an Enhanced Industry Standard System (ETSA) bus, a Front Side Bus (FSB), an HyperTransport (HT) interconnect, an Industry Standard System (TSA) bus, an Infinite Bandwidth Interconnect, a Low Pin Count (LPC) bus, a memory bus, a Microchannel System (MCA) bus, a Peripheral Component Interconnect (PCT) bus, a PCT-Express (PCT-X) bus, a Serial Advanced Technology Attachment (SATA) bus, a Video Electronics Standards Association Local (VLB) bus, or other suitable buses, or combinations of two or more of these. Where appropriate, bus 410 may include one or more buses. Although specific buses are described and illustrated in embodiments of this application, any suitable bus or interconnect is contemplated herein.

[0139] The data processing device can execute the data processing method described in the embodiments of this application, thereby achieving the combination Figures 2 to 4 The data processing methods and apparatus described.

[0140] Furthermore, in conjunction with the data processing methods in the above embodiments, this application embodiment can provide a computer-readable storage medium for implementation. This computer-readable storage medium stores computer program instructions; when executed by a processor, these computer program instructions implement any of the data processing methods in the above embodiments.

[0141] It should be clarified that this application is not limited to the specific configurations and processes described above and shown in the figures. For the sake of brevity, detailed descriptions of known methods are omitted here. In the above embodiments, several specific steps are described and shown as examples. However, the method process of this application is not limited to the specific steps described and shown. Those skilled in the art can make various changes, modifications, and additions, or change the order of steps, after understanding the spirit of this application.

[0142] The functional blocks shown in the above block diagram can be implemented as hardware, software, firmware, or a combination thereof. When implemented in hardware, they can be, for example, electronic circuits, application-specific integrated circuits (ASTCs), appropriate firmware, plug-ins, function cards, etc. When implemented in software, the elements of this application are programs or code segments used to perform the required tasks. Programs or code segments can be stored on a machine-readable medium or transmitted over a transmission medium or communication link via data signals carried on a carrier wave. "Machine-readable medium" can include any medium capable of storing or transmitting information. Examples of machine-readable media include electronic circuits, semiconductor memory devices, ROM, flash memory, erasable ROM (EROM), floppy disks, CD-ROMs, optical disks, hard disks, fiber optic media, radio frequency (RF) links, etc. Code segments can be downloaded via computer networks such as the Internet, intranets, etc.

[0143] It should also be noted that the exemplary embodiments mentioned in this application describe methods or systems based on a series of steps or apparatus. However, this application is not limited to the order of the above steps; that is, the steps can be performed in the order mentioned in the embodiments, or in a different order, or several steps can be performed simultaneously.

[0144] The above are merely specific embodiments of this application. Those skilled in the art will clearly understand that, for the sake of convenience and brevity, the specific working processes of the systems, modules, and units described above can be referred to the corresponding processes in the foregoing method embodiments, and will not be repeated here. It should be understood that the protection scope of this application is not limited thereto. Any person skilled in the art can easily conceive of various equivalent modifications or substitutions within the technical scope disclosed in this application, and these modifications or substitutions should all be covered within the protection scope of this application.

Claims

1. A data processing method, comprising: Obtain the N query time slices for the query task; Based on the N query time slices, N collection coroutine pools are created. The N collection coroutine pools include a collection coroutine pool corresponding to each of the N query time slices, and the collection coroutine pool includes M first coroutines. By collecting M first coroutines from the Ctrip pool corresponding to each query time slice, query data related to each query time slice is read from the database; If a target collection coroutine pool exists among the N collection coroutine pools and the time of the target query time slice corresponding to the target collection coroutine pool matches the time of the first query time slice carried by the notification pipeline, the query data related to the target query time slice is read into the data pipeline to send the data in the data pipeline to the relay service, and each first coroutine in the target collection coroutine pool is in a first state; if a target collection coroutine pool exists among the N collection coroutine pools and the time of the target query time slice corresponding to the target collection coroutine pool does not match the time of the first query time slice carried by the notification pipeline, the second coroutine corresponding to the target query time slice is temporarily stored in the notification pipeline; wherein, the second coroutine corresponding to the target query time slice is used to create the target collection coroutine pool.

2. The method according to claim 1, wherein, The step of reading query data related to each query time slice from the database through M first coroutines in the Ctrip pool corresponding to each query time slice includes: Data related to 1 / M query time slices in each query time slice is read from the database using each first coroutine in the collection coroutine pool corresponding to each query time slice.

3. The method according to claim 1, wherein, The notification pipeline includes N notification pipelines, each of which corresponds to a notification pipeline for each of the N query time slices; before reading the query data related to the target query time slice into the data pipeline, the method further includes: Based on the N query time slices, create an initial notification pipeline corresponding to each of the N query time slices; Select the first query time slice from the N query time slices; The start time and size of the first query time slice are written into the initial notification channel corresponding to each query time slice, thus obtaining the N notification channels.

4. The method according to claim 1, wherein, The step of creating N data collection coroutine pools based on the N query time slices includes: Based on the N query time slices, create a second coroutine corresponding to each of the N query time slices; A pool of acquisition coroutines corresponding to each query time slice is created using a second coroutine corresponding to each query time slice.

5. The method according to claim 4, wherein, The first state is the state in which the first coroutine has completed the reading process; Before reading the query data related to the target query time slice into the data pipeline, the method further includes: The running status of each first coroutine in the collection coroutine pool corresponding to each query time slice is monitored by the second coroutine corresponding to each query time slice; When each first coroutine in the acquisition coroutine pool is in the first state, the acquisition coroutine pool is determined as the target acquisition coroutine pool. By creating a second coroutine in the target acquisition coroutine pool, the time of the first query time slice carried by the notification pipeline is read.

6. The method according to claim 4, wherein, After reading the query data related to each query time slice from the database, the method further includes: The second coroutine corresponding to each query time slice is temporarily stored in the notification pipeline; When a target acquisition coroutine pool exists among the N acquisition coroutine pools and the time of the target query time slice corresponding to the target acquisition coroutine pool matches the time of the first query time slice carried by the notification pipeline, the query data related to the target query time slice is read into the data pipeline, including: If a target acquisition coroutine pool exists among the N acquisition coroutine pools and the time of the target query time slice corresponding to the target acquisition coroutine pool matches the time of the first query time slice carried by the notification pipeline, the query data related to the target query time slice is read into the data pipeline by creating a second coroutine of the target acquisition coroutine pool.

7. The method according to claim 1, wherein, After reading the query data related to the target query time slice into the data pipeline, the method further includes: By using a packet-assembly coroutine, the data in the data pipeline is encapsulated according to a preset quantity to obtain a query data packet corresponding to the preset quantity; The query data packet is read from the data pipeline by sending a third coroutine from the coroutine pool; Send the query data packet to the relay service.

8. The method according to claim 7, wherein, The method further includes: If the data pipeline does not contain a query data packet, a second query time slice is selected from the N query time slices; The time of the first query time slice carried by the notification pipeline is adjusted to match the time of the second query time slice.

9. A data processing apparatus, comprising: The acquisition module is used to acquire N query time slices for a query task; A creation module is used to create N collection coroutine pools based on the N query time slices. The N collection coroutine pools include a collection coroutine pool corresponding to each query time slice in the N query time slices, and the collection coroutine pool includes M first coroutines. The reading module is used to read query data related to each query time slice from the database through M first coroutines in the Ctrip pool corresponding to each query time slice; The reading module is further configured to, when a target collection coroutine pool exists among the N collection coroutine pools and the time of the target query time slice corresponding to the target collection coroutine pool matches the time of the first query time slice carried by the notification pipe, read the query data related to the target query time slice into the data pipe to send the data in the data pipe to the relay service, wherein each first coroutine in the target collection coroutine pool is in a first state; when a target collection coroutine pool exists among the N collection coroutine pools and the time of the target query time slice corresponding to the target collection coroutine pool does not match the time of the first query time slice carried by the notification pipe, temporarily store the second coroutine corresponding to the target query time slice into the notification pipe; wherein the second coroutine corresponding to the target query time slice is used to create the target collection coroutine pool.

10. A computer device, the device comprising: Processor and memory storing computer program instructions; When the processor executes the computer program instructions, it implements the steps of the data processing method as described in any one of claims 1-8.

11. A storage medium storing computer program instructions, which, when executed by a processor, implement the steps of the data processing method as described in any one of claims 1-8.

12. A computer program product, characterized in that, The program product is stored in a storage medium, and the program product is executed by at least one processor to implement the steps of the data processing method as described in any one of claims 1-8.

Images