Data processing method and device, equipment and storage medium

Abstract

The invention discloses a data processing method and device, equipment and a storage medium, and relates to the technical field of data processing. The method comprises the steps that according to N query time slices of a query task, a collection coroutine pool corresponding to each query time slice in the N query time slices is created, and each collection coroutine pool comprises M first coroutines; reading query data related to each query time slice from a database through M first coroutines in an acquisition stroke pool corresponding to each query time slice; and when the target acquisition coroutine pool exists and the time of the target query time slice corresponding to the target acquisition coroutine pool is matched with the time of the first query time slice carried by the notification pipeline, reading query data related to the target query time slice to the data pipeline so as to send data in the data pipeline to the transfer service. In this way, by conducting concurrent processing on the data of the N query time slices, the efficiency of querying the data from the database is improved, and the energy consumption of querying the data from the database is reduced.

Description

Technical Field

[0001] The present application belongs to the field of data processing technology, and in particular, relates to a data processing method, device, equipment and storage medium. Background Art

[0002] Data Transfer Express (DTEx) is a platform for real-time data distribution and batch data transfer for service systems. The platform needs to query data from the database according to the time slice dimension and push the queried data to the transfer service.

[0003] In the related art, the process of querying data from a database according to a time slice dimension is to read the data corresponding to each time slice from the database serially according to the time slice size by presetting the start time and size of the time slice. For example, the start time of the preset time slice is 0:00 on January 1, and the time slice size is 10 minutes. Based on this, the data from 0:00 to 0:10 is read from the database first. After the extraction is completed, the data from 0:10 to 20 can be read. This will increase the duration and energy consumption of data query and reduce the efficiency of querying data from the database. Summary of the invention

[0004] The embodiments of the present application provide a data processing method, apparatus, device and storage medium, which can solve the problems of high energy consumption and low efficiency in querying data from a database in the related art.

[0005] In a first aspect, an embodiment of the present application provides a data processing method, which may include:

[0006] Get N query time slices of the query task;

[0007] According to the N query time slices, N collection coroutine pools are created, the N collection coroutine pools include a collection coroutine pool corresponding to each query time slice in the N query time slices, and the collection coroutine pool includes M first coroutines;

[0008] Read the query data related to each query time slice from the database through the M first goroutines in the collection Ctrip pool corresponding to each query time slice;

[0009] When there is a target collection coroutine pool among the N collection coroutine pools and the time of the target query time slice corresponding to the target collection coroutine pool matches the time of the first query time slice carried by the notification pipeline, the query data related to the target query time slice is read into the data pipeline to send the data in the data pipeline to the transit service, and each first coroutine in the target collection coroutine pool is in the first state.

[0010] In a second aspect, an embodiment of the present application provides a data processing device, which may include:

[0011] The acquisition module is used to obtain N query time slices of the query task;

[0012] A creation module, used to create N collection coroutine pools according to the N query time slices, wherein the N collection coroutine pools include a collection coroutine pool corresponding to each query time slice in the N query time slices, and the collection coroutine pool includes M first coroutines;

[0013] A reading module, used to read query data related to each query time slice from the database through M first synergies in the collection Ctrip pool corresponding to each query time slice;

[0014] The reading module can also be used to read the query data related to the target query time slice into the data pipeline to send the data in the data pipeline to the transit service when there is a target collection coroutine pool among the N collection coroutine pools and the time of the target query time slice corresponding to the target collection coroutine pool matches the time of the first query time slice carried by the notification pipeline, and each first Ctrip in the target collection coroutine pool is in the first state.

[0015] In a third aspect, an embodiment of the present application provides a computer device, the computer device comprising: a processor and a memory storing computer program instructions;

[0016] When the processor executes the computer program instructions, the data processing method shown in the first aspect is implemented.

[0017] In a fourth aspect, an embodiment of the present application provides a computer storage medium having computer program instructions stored thereon, and when the computer program instructions are executed by a processor, the data processing method shown in the first aspect is implemented.

[0018] In a fifth aspect, an embodiment of the present application provides a chip, the chip including a processor and a communication interface, the communication interface and the processor are coupled, and the processor is used to run programs or instructions to implement the data processing method shown in the first aspect.

[0019] In a sixth aspect, an embodiment of the present application provides a computer program product, which is stored in a storage medium and is executed by at least one processor to implement the data processing method shown in the first aspect.

[0020] The data processing method, device, equipment and storage medium of the embodiment of the present application obtain N query time slices of the query task; according to the N query time slices, create N collection coroutine pools, the N collection coroutine pools include a collection coroutine pool corresponding to each query time slice in the N query time slices, and the collection coroutine pool includes M first coroutines; through the M first coroutines in the collection coroutine pool corresponding to each query time slice, read the query data related to each query time slice from the database, so that the data can be processed concurrently by the M first coroutines in the collection coroutine pool corresponding to each query time slice, which can significantly reduce the waiting time during time slice processing, improve the efficiency of querying data from the database, and reduce the energy consumption of querying data from the database. Then, when there is a target collection coroutine pool in the N collection coroutine pools and the time of the target query time slice corresponding to the target collection coroutine pool matches the time of the first query time slice carried by the notification pipeline, the query data related to the target query time slice is read into the data pipeline to send the data in the data pipeline to the transit service, and each first coroutine in the target collection coroutine pool is in the first state. Therefore, by utilizing the characteristics of single-threaded serial processing, the correctness of the package assembly is achieved without affecting the efficiency of querying data from the database, thus avoiding the problem of data disorder in the time slice. BRIEF DESCRIPTION OF THE DRAWINGS

[0021] In order to more clearly illustrate the technical solution of the embodiments of the present application, the following is a brief introduction to the drawings required for use in the embodiments of the present application. For ordinary technicians in this field, other drawings can be obtained based on these drawings without any creative work.

[0022] Figure 1 A flowchart of a data processing method provided in an embodiment of the present application;

[0023] Figure 2 A schematic diagram of a data processing method flow of a go program in a data processing method provided in an embodiment of the present application;

[0024] Figure 3 is a structural schematic diagram of a data processing device provided by an embodiment of the present application;

[0025] Figure 4 It is a schematic diagram of the structure of a computer device provided by an embodiment of the present application. DETAILED DESCRIPTION

[0026] The features and exemplary embodiments of various aspects of the present application will be described in detail below. In order to make the purpose, technical solutions and advantages of the present application clearer, the present application will be further described in detail below in conjunction with the accompanying drawings and specific embodiments. It should be understood that the specific embodiments described herein are only intended to explain the present application, rather than to limit the present application. For those skilled in the art, the present application can be implemented without the need for some of these specific details. The following description of the embodiments is only to provide a better understanding of the present application by illustrating the examples of the present application.

[0027] It should be noted that, in this article, relational terms such as first and second, etc. are only used to distinguish one entity or operation from another entity or operation, and do not necessarily require or imply any such actual relationship or order between these entities or operations. Moreover, the terms "include", "comprise" or any other variants thereof are intended to cover non-exclusive inclusion, so that a process, method, article or device including a series of elements includes not only those elements, but also other elements not explicitly listed, or also includes elements inherent to such process, method, article or device. In the absence of further restrictions, the elements defined by the statement "include..." do not exclude the existence of other identical elements in the process, method, article or device including the elements.

[0028] The acquisition, storage, use, and processing of data (including but not limited to the features and information in the text) in the technical solution of this application comply with the relevant provisions of national laws and regulations.

[0029] In order to solve the problems arising in the related technology, the embodiment of the present application proposes a data synchronization solution that can pre-process time slices, which can efficiently complete database data extraction and transmission. It can not only reduce the waiting time during processing between time slices by pre-processing different time slices in parallel, but also rely on the single-coroutine serial processing characteristics to achieve the correctness of package assembly, and can also realize the sequential packaging and sending of data between time slices, thereby ensuring that the data between time slices is sent sequentially and there will be no out-of-order processing.

[0030] The following will be combined with the attached Figures 1 to 4 , the methods, devices, computer equipment and storage media of the embodiments of the present application are described in detail. It should be noted that these embodiments are not intended to limit the scope of the disclosure of the present application.

[0031] Combine the following Figure 1 The data processing method shown is described in detail.

[0032] Figure 1 A flowchart of a data processing method provided in an embodiment of the present application.

[0033] like Figure 1 As shown, the data processing method can be applied to a computer device, and the data processing method can specifically include the following steps:

[0034] Step 110, obtaining N query time slices of the query task; Step 120, creating N collection coroutine pools according to the N query time slices, the N collection coroutine pools including a collection coroutine pool corresponding to each query time slice in the N query time slices, the collection coroutine pool including M first coroutines, M being a positive integer; Step 130, reading query data related to each query time slice from the database through the M first coroutines in the collection coroutine pool corresponding to each query time slice; Step 140, when there is a target collection coroutine pool in the N collection coroutine pools and the time of the target query time slice corresponding to the target collection coroutine pool matches the time of the first query time slice carried by the notification pipeline, reading the query data related to the target query time slice into the data pipeline to send the data in the data pipeline to the transit service, and each first coroutine in the target collection coroutine pool is in the first state.

[0035] In this way, N collection coroutine pools can be created according to the N query time slices of the query task, and the N collection coroutine pools include a collection coroutine pool corresponding to each query time slice in the N query time slices, and the collection coroutine pool includes M first coroutines; through the M first coroutines in the collection coroutine pool corresponding to each query time slice, the query data related to each query time slice is read from the database, so that the data can be processed concurrently by the M first coroutines in the collection coroutine pool corresponding to each query time slice, which can significantly reduce the waiting time during time slice processing, improve the efficiency of querying data from the database, and reduce the energy consumption of querying data from the database. Then, when there is a target collection coroutine pool in the N collection coroutine pools and the time of the target query time slice corresponding to the target collection coroutine pool matches the time of the first query time slice carried by the notification pipeline, the query data related to the target query time slice is read into the data pipeline to send the data in the data pipeline to the transit service, and each first coroutine in the target collection coroutine pool is in the first state. Therefore, by utilizing the characteristics of single-threaded serial processing, the correctness of the package assembly is achieved without affecting the efficiency of querying data from the database, thus avoiding the problem of data disorder in the time slice.

[0036] The above steps are described in detail below, as shown below.

[0037] First, referring to step 110, in some embodiments of the present application, the number N of query time slices that need to be preprocessed is set, where N is a positive integer.

[0038] It should be noted that database data query is the most time-consuming part in the embodiment of the present application. Therefore, if the machine memory allows, the number of time slices for concurrent queries can be increased, that is, N can be appropriately increased.

[0039] Next, step 120 is referred to. In some embodiments of the present application, step 120 may specifically include step 1201 and step 1202.

[0040] Step 1201: Create a second coroutine corresponding to each query time slice in the N query time slices according to the N query time slices.

[0041] Specifically, a second coroutine corresponding to each query time slice in the N query time slices may be created through the main coroutine of the query task.

[0042] Exemplarily, if N is 3, that is, the three query time slices can be query time slice 1, query time slice 2 and query time slice 3, then a corresponding second coroutine 1 can be created for query time slice 1, a corresponding second coroutine 2 can be created for query time slice 2, and a corresponding second coroutine 3 can be created for query time slice 3.

[0043] Step 1202: Create a collection coroutine pool corresponding to each query time slice through the second coroutine corresponding to each query time slice.

[0044] Exemplarily, still taking the above embodiment as an example, query time slice 1 creates a corresponding second coroutine 1, and through the second coroutine 1, a collection coroutine pool 1 corresponding to the query time slice 1 can be created. Also, query time slice 2 creates a corresponding second coroutine 2, and through the second coroutine 2, a collection coroutine pool 2 corresponding to the query time slice 2 can be created. Also, query time slice 3 creates a corresponding second coroutine 3, and through the second coroutine 3, a collection coroutine pool 3 corresponding to the query time slice 3 can be created.

[0045] Furthermore, step 130 may include:

[0046] Data related to 1 / M query time slices in each query time slice is read from the database through each first coroutine in the collection coroutine pool corresponding to each query time slice.

[0047] Exemplarily, still taking the above embodiment as an example, the collection coroutine pool 1 corresponding to the query time slice 1 may include M first coroutines. If M is 3, the 3 first coroutines in the collection coroutine pool 1 may include the first coroutine 1, the first coroutine 2 and the first coroutine 3. In this way, the first coroutine 1 can read the relevant data within 1 / 3 of the query time slices in the query time slice 1.

[0048] It should be noted that other query time slices all refer to each first coroutine in the acquisition coroutine pool 1 corresponding to the aforementioned query time slice 1, and read data related to 1 / M query time slices in query time slice 1 from the database, which will not be repeated here.

[0049] And, in some embodiments of the present application, before step 140, a process of creating a notification pipeline is also included to read the time of the first query time slice in the notification pipeline after all the coroutines in any of the aforementioned acquisition coroutine pools are processed.

[0050] Based on this, the notification pipeline includes N notification pipelines, and the N notification pipelines include a notification pipeline corresponding to each query time slice in the N query time slices. The data processing method can also include steps 2101 to 2103.

[0051] Step 2101: Create an initial notification channel corresponding to each query time slice in the N query time slices according to the N query time slices.

[0052] Exemplarily, still taking the above embodiment as an example, if N is 3, that is, the three query time slices can be query time slice 1, query time slice 2 and query time slice 3, then the initial notification pipeline 1 corresponding to query time slice 1, the initial notification pipeline 2 corresponding to query time slice 2 and the initial notification pipeline 3 corresponding to query time slice 3 can be created.

[0053] Step 2102: Select the first query time slice from the N query time slices.

[0054] Exemplarily, still taking the above embodiment as an example, the first query time slice, such as query time slice 1, is selected from the three query time slices.

[0055] Step 2103: Write the starting time point of the first query time slice and the size of the first query time slice into the initial notification channel corresponding to each query time slice, and obtain N notification channels.

[0056] Exemplarily, the starting time point of query time slice 1, such as: 2021-10-1910:24:30.000 and the size of the first query time slice, such as 10 minutes, can be written into initial notification pipeline 1, initial notification pipeline 2 and initial notification pipeline 3 respectively, so that each initial notification pipeline calculates the time of the first query time slice (pub_ts). In this way, three notification pipelines are obtained, namely notification pipeline 1, notification pipeline 2 and notification pipeline 3.

[0057] Here, in the embodiment of the present application, the notification pipeline has read and write functions. If there is no data written into the notification pipeline, such as the starting time point of the query time slice and the size of the first query time slice, the second coroutine corresponding to each query time slice will be blocked and unable to perform subsequent steps until the starting time point of the query time slice and the size of the first query time slice are written into the pipeline. If this step is not taken, all second coroutines are blocked on the notification pipeline and unable to perform subsequent steps.

[0058] Therefore, after this step, each second coroutine can read the time of the first query time slice carried in the notification pipeline to see if it is the same as the time of the query time slice processed by the current second coroutine. If they are the same, step 140 is executed, otherwise, the block continues. It can be seen that only one second coroutine can enter step 140, and the other N-1 second coroutines are all blocked on the notification pipeline again.

[0059] Since the most time-consuming data query part in the embodiment of the present application has been preprocessed, other coroutines are only waiting for the final packaging and sending steps, which significantly improves the parallel processing capability, can significantly reduce the waiting time during time slice processing, improve the efficiency of querying data from the database, and reduce the energy consumption of querying data from the database. For example: if 3 query time slices are required, the processing time of each query time slice is 1 second, 2 seconds, and 3 seconds. If serial processing is used, without considering the subsequent package sending time, the system data query takes 6 seconds. After the M first coroutines in the collection Ctrip pool corresponding to each query time slice are used to concurrently process the data, the data query time is reduced to 3 seconds.

[0060] Then, referring to step 140, in some embodiments of the present application, before step 140, the data processing method may further include step 150, temporarily storing the second coroutine corresponding to each query time slice in the notification pipeline.

[0061] Based on this, step 140 may specifically include:

[0062] When there is a target collection coroutine pool among the N collection coroutine pools and the time of the target query time slice corresponding to the target collection coroutine pool matches the time of the first query time slice carried by the notification pipeline, the query data related to the target query time slice is read into the data pipeline by creating a second coroutine in the target collection coroutine pool.

[0063] Exemplarily, when the second coroutine of creating the target acquisition coroutine pool reads the time of the first query time slice carried in the notification pipeline and determines that the time of the first query time slice carried in the notification pipeline is the same as the time of the query time slice currently processed by the second coroutine, the query data related to the target query time slice can be read into the data pipeline by creating the second coroutine of the target acquisition coroutine pool.

[0064] In other embodiments of the present application, the first state is a state in which the first coroutine has completed reading. Before reading the query data related to the target query time slice into the data pipeline in step 140, the data processing method also includes steps 3101 to 3103.

[0065] Step 3101, monitoring the running status of each first coroutine in the collection coroutine pool corresponding to each query time slice through the second coroutine corresponding to each query time slice.

[0066] Exemplarily, still taking the above embodiment as an example, if query time slice 1 corresponds to second coroutine 1, query time slice 2 corresponds to second coroutine 2, and query time slice 3 corresponds to second coroutine 3. Second coroutine 1 monitors the running status of each first coroutine in collection coroutine pool 1 corresponding to query time slice 1, second coroutine 2 monitors the running status of each first coroutine in collection coroutine pool 2 corresponding to query time slice 2, and second coroutine 3 monitors the running status of each first coroutine in collection coroutine pool 3 corresponding to query time slice 3.

[0067] Step 3102: When the running state of each first coroutine in the collection coroutine pool is the first state, the collection coroutine pool is determined as the target collection coroutine pool.

[0068] Exemplarily, if the second coroutine 3 detects that the running status of each first coroutine in the collection coroutine pool 3 corresponding to the query time slice 3 has read all the relevant data in the query time slice 3, the collection coroutine pool 3 can be determined as the target collection coroutine pool.

[0069] Step 3103: Create a second coroutine in the target acquisition coroutine pool and read the time of the first query time slice carried by the notification pipeline.

[0070] Exemplarily, the second coroutine 3 of the acquisition coroutine pool 3 can be created to read the time of the first query time slice carried by N notification pipelines, so that the second coroutine 3 determines whether the time of the first query time slice carried in the notification pipeline is the same as the time of the query time slice currently processed by the second coroutine 3. If they are the same, step 140 is executed; if they are not the same, the block continues, that is, the following steps are executed. That is, in some embodiments of the present application, after step 130, the data processing method may also include:

[0071] When there is a target collection coroutine pool among N collection coroutine pools and the time of the target query time slice corresponding to the target collection coroutine pool does not match the time of the first query time slice carried by the notification pipeline, the second coroutine corresponding to the target query time slice is temporarily stored in the notification pipeline; wherein the second coroutine corresponding to the target query time slice is used to create the target collection coroutine pool.

[0072] In addition, based on the above steps, after step 140, the data processing method may further include a process of sending data to a transit service. Based on this, the data processing method may further include steps 4101 to 4103.

[0073] Step 4101, through the package assembly coroutine, the data in the data pipeline is packaged according to the preset number to obtain the query data packet corresponding to the preset number.

[0074] Exemplarily, the coroutine entering this step writes all query data queried in the above step 130 into the data pipeline, and then starts a packet assembly coroutine. The purpose of the packet assembly coroutine is to combine the read query data into one or more query data packets according to the required number of packets. For example, there are 89 database records queried in the current query time slice 1. If each query data packet is required to contain 10 data, then the packet assembly coroutine will split the 89 database records into 9 data packets and write them into the data pipeline where the packet data is stored, of which 8 query data packets contain 10 database records and 1 query data packet contains 9 database records.

[0075] As a result, the acquired data can support multiple packaged and sent transit services, shorten the encapsulation time, and improve the efficiency of sending data in the data pipeline to the transit service.

[0076] Step 4102, read the query data packet from the data pipeline by sending the third coroutine in the coroutine pool.

[0077] Step 4103, sending a query data packet to the transit service.

[0078] Therefore, the embodiment of the present application adopts a single-coroutine serial processing method, which can ensure that in the packet assembly process, the number of data records in multiple data packets will not be less than the required number of data records in the packet, that is, at most there will be only one data packet with a number of records less than the required number of records in the data packet. The reason why single-coroutine serial processing can be used here is that it only involves memory copying and does not involve complex CPU operations, so it will not affect the processing performance.

[0079] In addition, after step 4102, the data processing method may further include step 4104 and step 4105.

[0080] Step 4104, when the query data packet is not included in the detection data pipeline, select a second query time slice from the N query time slices.

[0081] Step 4105: adjust the time of the first query time slice carried by the notification channel to the time of the second query time slice.

[0082] Exemplarily, when the query data packet in the data pipeline is consumed, i.e., sent, the main coroutine will write the next query time slice to be processed, i.e., the second query time slice, such as the starting time point of query time slice 2, and the size of the first query time slice into the notification pipeline corresponding to each query time slice. At this time, the second coroutine blocked in the notification pipeline is reactivated, and the time of the second query time slice in the notification pipeline is read. Then, the coroutine with the same time slice as the current second coroutine 2 continues to execute the above step 140, and other coroutines continue to be blocked, thereby realizing the continuous advancement of the time slice.

[0083] In this way, it is ensured that data is sent sequentially between time slices and out-of-order processing will not occur.

[0084] The following uses the go language as an example to illustrate the data processing method provided in the embodiment of the present application, as shown below.

[0085] like Figure 2 As shown, the data processing method can include steps 1 to 7.

[0086] Step 1, set the number of query time slices N, N> = 1. Database data query is the most time-consuming part in the embodiment of the present application. Therefore, if the machine memory allows, N can be appropriately increased to increase the number of concurrent query time slices.

[0087] Step 2: Open N notification channels to notify each collection coroutine pool of the time of the query time slice currently being processed (pub_ts), and write the start time of the time slice, such as the time of query time slice 1: 2021-10-19 10:24:30.000, into the N notification channels. The channel has read and write functions. If there is no data in the channel, the coroutine that reads the channel will be blocked until data is written into the channel.

[0088] Step 3, start N second coroutines, each second coroutine starts a collection coroutine pool, each collection coroutine pool has M first coroutines, each first coroutine in the collection coroutine pool processes 1 / M of the data in the current time slice, and when all the first coroutines in this collection coroutine pool are processed, read the time of the first query time slice in the notification pipeline. If there is no step 2, all N second coroutines are blocked on the notification pipeline and cannot proceed to subsequent steps. N second coroutines read the time slice information (pub_ts) written in step 2.

[0089] Step 4: Each of the N second goroutines determines whether the read time slice information (pub_ts) is the same as the query time slice it is currently processing. If they are the same, it executes step 5, otherwise it continues to block. Here, only one second goroutine can enter step 5, and the other N-1 second goroutines are all blocked on the notification pipeline again.

[0090] Step 5: The second coroutine in this step writes all query data queried in step 2 into the data storage pipeline, and then starts a package assembly coroutine. The purpose of the package assembly coroutine is to combine the query data into one or more request packages according to the required number of packages. For example, if 89 database records are queried in the current time slice, and each package is required to contain 10 data, then the package assembly coroutine will split the 89 database records into 9 data packets and write them into the pipeline for storing package data, of which 8 packets contain 10 database records and 1 packet contains 9 database records.

[0091] Step 6, start a sending coroutine pool, which contains P third coroutines. The P third coroutines concurrently send data pipelines that store package data. When the data in the package data pipeline is consumed, the main coroutine writes the next time slice information (pub_ts) to be processed, such as the time of querying time slice 2, into the N notification pipelines created in step 2.

[0092] Step 7: Once data is written to the notification pipeline, the second coroutine blocked in step 3 is reactivated to read the time slice information (pub_ts) in the notification pipeline.

[0093] Then, the coroutine with the same time slice as the current coroutine continues to execute steps 4 to 7, and other coroutines continue to block, thereby achieving continuous advancement of the query time slice.

[0094] It can be seen that in the embodiment of the present application, concurrent pre-processing of data can be achieved through the above steps 2, 3 and 4, which can significantly reduce the system's time slice processing waiting time. For example: if three time slices are to be processed, the processing time for each time slice is 1 second, 2 seconds, and 3 seconds. If serial processing is used, without considering the subsequent packet sending time, the system data query time will take 6 seconds. After adopting the above solution, the system data query time is reduced to 3 seconds.

[0095] Because single-coroutine serial processing is used in steps 4 and 5, it can be ensured that the number of data records in multiple packages will not be less than the required number of data records in the package during the package assembly process, that is, at most there will be a situation where the number of records in a package is less than the required number of records in the package. The reason why single-coroutine serial processing can be used here is that it only involves memory copying and does not involve complex CPU operations, so it will not affect processing performance.

[0096] The design of the notification pipeline and notification mechanism involved in step 2, step 3, step 4, step 5, step 6 and step 7 ensures that data between time slices are sent sequentially and out-of-order processing will not occur.

[0097] In summary, the embodiments of the present application provide a data synchronization solution that can pre-process time slices, which can support concurrent pre-processing of multiple time slice data to improve the overall synchronization performance of the system; and, by utilizing the characteristics of single-threaded serial processing, the correctness of packet assembly is achieved without affecting the processing performance; and the notification mechanism ensures that when pre-processing multiple time slices, there will be no time slice data disorder problems, thereby improving the processing efficiency of multiple time slices.

[0098] The present application also provides a data processing device, specifically in combination with Figure 3 Provide detailed explanation.

[0099] Figure 3 It is a structural diagram of a data processing device provided by an embodiment of the present application.

[0100] In some embodiments of the present application, Figure 3 The data processing device shown may be arranged in a computer device.

[0101] like Figure 3 As shown, the data processing device 30 may specifically include:

[0102] An acquisition module 301 is used to acquire N query time slices of a query task;

[0103] A creation module 302 is used to create N collection coroutine pools according to the N query time slices, wherein the N collection coroutine pools include a collection coroutine pool corresponding to each query time slice in the N query time slices, and the collection coroutine pool includes M first coroutines;

[0104] A reading module 303 is used to read query data related to each query time slice from the database by collecting M first synergies in the Ctrip pool corresponding to each query time slice;

[0105] The reading module 303 is also used to read the query data related to the target query time slice into the data pipeline to send the data in the data pipeline to the transit service when there is a target collection coroutine pool in the N collection coroutine pools and the time of the target query time slice corresponding to the target collection coroutine pool matches the time of the first query time slice carried by the notification pipeline, and each first Ctrip in the target collection coroutine pool is in the first state.

[0106] In this way, the data processing device provided by the embodiment of the present application can create N collection coroutine pools according to the N query time slices of the query task, and the N collection coroutine pools include a collection coroutine pool corresponding to each query time slice in the N query time slices, and the collection coroutine pool includes M first coroutines; through the M first coroutines in the collection coroutine pool corresponding to each query time slice, the query data related to each query time slice is read from the database, so that the data can be processed concurrently by the M first coroutines in the collection coroutine pool corresponding to each query time slice, which can significantly reduce the waiting time during time slice processing, improve the efficiency of querying data from the database, and reduce the energy consumption of querying data from the database. Then, when there is a target collection coroutine pool in the N collection coroutine pools and the time of the target query time slice corresponding to the target collection coroutine pool matches the time of the first query time slice carried by the notification pipeline, the query data related to the target query time slice is read into the data pipeline to send the data in the data pipeline to the transit service, and each first coroutine in the target collection coroutine pool is in the first state. Therefore, by utilizing the characteristics of single-threaded serial processing, the correctness of the package assembly is achieved without affecting the efficiency of querying data from the database, thus avoiding the problem of data disorder in the time slice.

[0107] The data processing device 70 in the embodiment of the present application is described in detail below.

[0108] In some embodiments of the present application, the reading module 303 can be specifically used to read data related to 1 / M query time slices in each query time slice from the database through each first coroutine in the acquisition coroutine pool corresponding to each query time slice.

[0109] In some embodiments of the present application, the data processing device 70 in the embodiment of the present application may further include a selection module and a writing module; wherein,

[0110] The creation module 302 may also be used to create an initial notification channel corresponding to each query time slice in the N query time slices according to the N query time slices, when the notification channel includes N notification channels and the N notification channels include a notification channel corresponding to each query time slice in the N query time slices;

[0111] A selection module, used for selecting a first query time slice from N query time slices;

[0112] The writing module is used to write the starting time point of the first query time slice and the size of the first query time slice into the initial notification channel corresponding to each query time slice respectively, so as to obtain N notification channels.

[0113] In some embodiments of the present application, the creation module 302 may be specifically configured to create, according to the N query time slices, a second coroutine corresponding to each query time slice in the N query time slices;

[0114] Through the second coroutine corresponding to each query time slice, a collection coroutine pool corresponding to each query time slice is created.

[0115] In some embodiments of the present application, the data processing device 70 in the embodiments of the present application may also include a monitoring module and a determination module; wherein,

[0116] A monitoring module, used for monitoring the running state of each first coroutine in the collection coroutine pool corresponding to each query time slice through the second coroutine corresponding to each query time slice when the first state is a state in which the first coroutine has completed reading;

[0117] A determination module, configured to determine the collection Ctrip pool as a target collection coroutine pool when the running state of each first coroutine in the collection coroutine pool is the first state;

[0118] The reading module 303 can be specifically used to read the time of the first query time slice carried by the notification pipeline by creating a second coroutine in the target acquisition coroutine pool.

[0119] In some embodiments of the present application, the data processing device 70 in the embodiment of the present application may further include a temporary storage module; wherein,

[0120] A temporary storage module is used to temporarily store the second coroutine corresponding to each query time slice into the notification pipeline;

[0121] The reading module 303 is specifically used to read the query data related to the target query time slice into the data pipeline by creating a second coroutine in the target collection coroutine pool when there is a target collection coroutine pool among the N collection coroutine pools and the time of the target query time slice corresponding to the target collection coroutine pool matches the time of the first query time slice carried by the notification pipeline.

[0122] In some embodiments of the present application, the data processing device 70 in the embodiments of the present application may further include a packaging module and a sending module; wherein,

[0123] The encapsulation module is used to encapsulate the data in the data pipeline according to the preset number through the package assembly coroutine to obtain the query data packets corresponding to the preset number;

[0124] The reading module 303 is further used to read the query data packet from the data pipeline by sending the third coroutine in the coroutine pool;

[0125] The sending module is used to send a query data packet to the transit service.

[0126] In some embodiments of the present application, the data processing device 70 in the embodiments of the present application may further include a selection module and an adjustment module; wherein,

[0127] A selection module, configured to select a second query time slice from the N query time slices when the query data packet is not included in the detection data pipeline;

[0128] The adjustment module is used to adjust the time of the first query time slice carried by the notification channel to the time of the second query time slice.

[0129] In some embodiments of the present application, the data processing device 70 in the embodiments of the present application may also include a temporary storage module, which is used to temporarily store the second coroutine corresponding to the target query time slice to the notification pipeline when there is a target acquisition coroutine pool in the N acquisition coroutine pools and the time of the target query time slice corresponding to the target acquisition coroutine pool does not match the time of the first query time slice carried by the notification pipeline; wherein the second coroutine corresponding to the target query time slice is used to create the target acquisition coroutine pool.

[0130] Based on the same inventive concept, the present application also provides a computer device. Figure 4 Provide detailed explanation.

[0131] Figure 4 It is a schematic diagram of the structure of a computer device provided by an embodiment of the present application.

[0132] like Figure 4As shown, the computer device may include at least one of the following involved in the embodiments of the present application: a user terminal device, a server terminal device. The computer device may include a processor 401 and a memory 402 storing computer program instructions.

[0133] Specifically, the processor 401 may include a central processing unit (CPU), or an application specific integrated circuit (Application Specific Integrated Circuit (ASTC), or may be configured to implement one or more integrated circuits of the embodiments of the present application.

[0134] The memory 402 may include a large capacity memory for data or instructions. By way of example and not limitation, the memory 402 may include a hard disk drive (HDD), a floppy disk drive, a flash memory, an optical disk, a magneto-optical disk, a magnetic tape, or a universal serial bus (USB) drive or a combination of two or more of these. In appropriate cases, the memory 402 may include a removable or non-removable (or fixed) medium. In appropriate cases, the memory 402 may be inside or outside the integrated gateway disaster recovery device. In a specific embodiment, the memory 402 is a non-volatile solid-state memory. In a specific embodiment, the memory 402 includes a solid-state storage (ROM). In appropriate cases, the ROM may be a mask-programmed ROM, a programmable ROM (PROM), an erasable PROM (EPROM), an electrically erasable PROM (EEPROM), an electrically rewritable ROM (EAROM) or a flash memory or a combination of two or more of these.

[0135] The processor 401 implements any one of the data processing methods in the above embodiments by reading and executing computer program instructions stored in the memory 402 .

[0136] In one example, the computer device may further include a communication interface 403 and a bus 410. Figure 4 As shown, the processor 401, the memory 402, and the communication interface 403 are connected via a bus 410 and communicate with each other.

[0137] The communication interface 403 is mainly used to implement communication between various modules, devices, units and / or equipment in the embodiments of the present application.

[0138] Bus 410 includes hardware, software or both, and the parts of flow control device are coupled to each other.For example, but not limitation, bus may include accelerated graphics port (AGP) or other graphics bus, enhanced industry standard system (ETSA) bus, front side bus (FSB), hypertransport (HT) interconnection, industry standard system (TSA) bus, infinite bandwidth interconnection, low pin count (LPC) bus, memory bus, micro channel system (MCA) bus, peripheral component interconnection (PCT) bus, PCT-Express (PCT-X) bus, serial advanced technology attachment (SATA) bus, video electronics standard association local (VLB) bus or other suitable bus or two or more of these combinations. In appropriate cases, bus 410 may include one or more buses. Although the present application embodiment describes and shows a specific bus, the application considers any suitable bus or interconnection.

[0139] The data processing device can execute the data processing method in the embodiment of the present application, thereby realizing the combination Figures 2 to 4 The data processing method and device described.

[0140] In addition, in combination with the data processing method in the above embodiments, the present application embodiment can provide a computer-readable storage medium for implementation. The computer-readable storage medium stores computer program instructions; when the computer program instructions are executed by a processor, any one of the data processing methods in the above embodiments is implemented.

[0141] It should be clear that the present application is not limited to the specific configuration and processing described above and shown in the figures. For the sake of simplicity, a detailed description of the known method is omitted here. In the above embodiments, several specific steps are described and shown as examples. However, the method process of the present application is not limited to the specific steps described and shown, and those skilled in the art can make various changes, modifications and additions, or change the order between the steps after understanding the spirit of the present application.

[0142] The functional blocks shown in the above block diagram can be implemented as hardware, software, firmware or a combination thereof. When implemented in hardware, it can be, for example, an electronic circuit, an application specific integrated circuit (ASIC), appropriate firmware, a plug-in, a function card, etc. When implemented in software, the elements of the present application are programs or code segments that are used to perform the required tasks. The program or code segment can be stored in a machine-readable medium, or transmitted on a transmission medium or a communication link by a data signal carried in a carrier wave. "Machine-readable medium" can include any medium capable of storing or transmitting information. Examples of machine-readable media include electronic circuits, semiconductor memory devices, ROM, flash memory, erasable ROM (EROM), floppy disks, CD-ROMs, optical disks, hard disks, optical fiber media, radio frequency (RF) links, etc. The code segment can be downloaded via a computer network such as the Internet, an intranet, etc.

[0143] It should also be noted that the exemplary embodiments mentioned in this application describe some methods or systems based on a series of steps or devices. However, this application is not limited to the order of the above steps, that is, the steps can be performed in the order mentioned in the embodiment, or in a different order from the embodiment, or several steps can be performed simultaneously.

[0144] The above are only specific implementation methods of the present application. Those skilled in the art can clearly understand that for the convenience and simplicity of description, the specific working processes of the systems, modules and units described above can refer to the corresponding processes in the aforementioned method embodiments, and will not be repeated here. It should be understood that the protection scope of the present application is not limited to this. Any technician familiar with the technical field can easily think of various equivalent modifications or replacements within the technical scope disclosed in this application, and these modifications or replacements should be included in the protection scope of this application.

Claims

1. A data processing method, comprising: Get N query time slices of the query task; According to the N query time slices, N collection coroutine pools are created, wherein the N collection coroutine pools include a collection coroutine pool corresponding to each query time slice in the N query time slices, and the collection coroutine pool includes M first coroutines; Reading query data related to each query time slice from the database through M first synergies in the collection Ctrip pool corresponding to each query time slice; When there is a target collection coroutine pool among the N collection coroutine pools and the time of the target query time slice corresponding to the target collection coroutine pool matches the time of the first query time slice carried by the notification pipeline, the query data related to the target query time slice is read into the data pipeline to send the data in the data pipeline to the transit service, and each first coroutine in the target collection coroutine pool is in the first state.

2. The method according to claim 1, wherein: The step of reading query data related to each query time slice from a database by collecting M first synergies in a Ctrip pool corresponding to each query time slice includes: Data related to 1 / M query time slices in each query time slice is read from a database through each first coroutine in the collection coroutine pool corresponding to each query time slice.

3. The method according to claim 1, wherein: The notification pipeline includes N notification pipelines, and the N notification pipelines include a notification pipeline corresponding to each query time slice in the N query time slices; before reading the query data related to the target query time slice into the data pipeline, the method further includes: According to the N query time slices, creating an initial notification channel corresponding to each query time slice in the N query time slices; Selecting a first query time slice from the N query time slices; The starting time point of the first query time slice and the size of the first query time slice are respectively written into the initial notification pipeline corresponding to each query time slice to obtain the N notification pipelines.

4. The method according to claim 1, wherein: The step of creating N collection coroutine pools according to the N query time slices includes: According to the N query time slices, create a second coroutine corresponding to each query time slice in the N query time slices; A collection coroutine pool corresponding to each query time slice is created through the second coroutine corresponding to each query time slice.

5. The method according to claim 4, wherein: The first state is a state in which the first coroutine has completed reading; Before reading the query data related to the target query time slice into the data pipeline, the method further includes: Monitoring the running status of each first coroutine in the collection coroutine pool corresponding to each query time slice through the second coroutine corresponding to each query time slice; When the running state of each first coroutine in the collection coroutine pool is the first state, determining the collection coroutine pool as the target collection coroutine pool; The time of the first query time slice carried by the notification pipeline is read by creating a second coroutine in the target acquisition coroutine pool.

6. The method according to claim 4, wherein: After reading the query data related to each query time slice from the database, the method further includes: Temporarily storing the second coroutine corresponding to each query time slice in the notification pipeline; When there is a target acquisition coroutine pool in the N acquisition coroutine pools and the time of the target query time slice corresponding to the target acquisition coroutine pool matches the time of the first query time slice carried by the notification pipeline, query data related to the target query time slice is read into the data pipeline, including: When there is a target acquisition coroutine pool among the N acquisition coroutine pools and the time of the target query time slice corresponding to the target acquisition coroutine pool matches the time of the first query time slice carried by the notification pipeline, the query data related to the target query time slice is read into the data pipeline by creating a second coroutine of the target acquisition coroutine pool.

7. The method according to claim 1, wherein: After reading the query data related to the target query time slice into the data pipeline, the method further includes: By means of a package assembly coroutine, the data in the data pipeline is packaged according to a preset number to obtain a query data packet corresponding to the preset number; Reading the query data packet from the data pipeline by sending a third coroutine in the coroutine pool; The query data packet is sent to the transit service.

8. The method according to claim 7, wherein: The method further comprises: In the case of detecting that the data pipeline does not include a query data packet, selecting a second query time slice from the N query time slices; The time of the first query time slice carried by the notification channel is adjusted to the time of the second query time slice.

9. The method according to claim 1, wherein: The method further comprises: When there is a target collection coroutine pool among the N collection coroutine pools and the time of the target query time slice corresponding to the target collection coroutine pool does not match the time of the first query time slice carried by the notification pipeline, the second coroutine corresponding to the target query time slice is temporarily stored in the notification pipeline; wherein the second coroutine corresponding to the target query time slice is used to create the target collection coroutine pool.

10. A data processing device, comprising: The acquisition module is used to obtain N query time slices of the query task; A creation module, configured to create N collection coroutine pools according to the N query time slices, wherein the N collection coroutine pools include a collection coroutine pool corresponding to each query time slice in the N query time slices, and the collection coroutine pool includes M first coroutines; A reading module, configured to read query data related to each query time slice from a database through M first synergies in a collection Ctrip pool corresponding to each query time slice; The reading module is also used to read the query data related to the target query time slice into the data pipeline to send the data in the data pipeline to the transit service when there is a target collection coroutine pool in the N collection coroutine pools and the time of the target query time slice corresponding to the target collection coroutine pool matches the time of the first query time slice carried by the notification pipeline, and each first coroutine in the target collection coroutine pool is in the first state.

11. A computer device, comprising: a processor and a memory storing computer program instructions; When the processor executes the computer program instructions, the steps of the data processing method according to any one of claims 1 to 9 are implemented.

12. A storage medium having computer program instructions stored thereon, wherein the computer program instructions, when executed by a processor, implement the steps of the data processing method according to any one of claims 1 to 9.

13. A computer program product, characterized in that The program product is stored in a storage medium, and the program product is executed by at least one processor to implement the steps of the data processing method according to any one of claims 1 to 9.

Images