Scheduling method, device, computer storage medium and electronic device

Abstract

The present invention provides a scheduling method, which includes the following steps: pre-decomposing the download and storage business of report data into three processes: downloading file data, parsing file data and persisting file data into a warehouse; creating a first message queue, a second message queue and a third message queue, wherein the first message queue contains downloaded file data, the second message queue contains parsed file data, the third message queue contains persisted file data into a warehouse, and the first message queue, the second message queue and the third message queue run in parallel; the ratio of any two values ​​of the computing resources occupied by the first message queue, the computing resources occupied by the second message queue and the computing resources occupied by the third message queue is equal to the inverse ratio of the time required to spend with the same computing resources of the corresponding message queues.

Description

Background Art

[0002] In the Internet era, a large number of data requests often appear in a certain time period. The existing scheduling method for high-frequency requests is to split the process of specific business steps, and each process node is distributed and scheduled in the form of message queues. For example, the process of downloading and saving report data is broken down into: 1. Download file data, 2. Parse file data, 3. Persist file data into the database.

[0003] In conventional processing methods, the process is in one thread, and the function process of downloading file data, parsing file data, and persisting file data into the database is called in sequence. When encountering frequent data requests, the whole thread can only be parallelized to improve efficiency, but the efficiency improved by this method is still difficult to meet high-frequency requests. Therefore, a scheduling method, device, computer storage medium and electronic device that can be applied to frequent data requests are needed. Summary of the invention

[0004] The present invention provides a scheduling method, device, computer storage medium and electronic device, aiming to solve the technical problems mentioned in the background technology.

[0005] The present invention first provides a scheduling method, which comprises the following steps:

[0006] The download and storage of report data is pre-divided into three processes: downloading file data, parsing file data, and persisting file data into the database.

[0007] Create a first message queue, a second message queue and a third message queue, wherein the first message queue contains downloaded file data, the second message queue contains parsed file data, the third message queue contains persistent storage of file data, and the first message queue, the second message queue and the third message queue run in parallel.

[0008] Furthermore, the ratio of any two values ​​of the computing resources occupied by the first message queue, the computing resources occupied by the second message queue, and the computing resources occupied by the third message queue is equal to the ratio of the time required for the corresponding message queues with the same computing resources, so that the time spent by the first message queue, the second message queue and the third message queue in the single-threaded case is the same.

[0009] Furthermore, the data request service is deployed across regions in advance, and a method for preventing network fluctuations is also included, including:

[0010] When an abnormality is detected in the network line of a certain area, the tasks corresponding to that area will be distributed to other areas for deployment and processing.

[0011] Furthermore, the speed of connecting to the server in different regions is compared, and the CDN node of the fastest regional network is used as the response node.

[0012] Furthermore, a deduplication method of request is included, and a distributed lock is set for any interface request service, and the deduplication method includes:

[0013] When a request for a second interface request service is detected when one interface data is already occupied by a first interface request service, a first distributed lock corresponding to the first interface request service and a second distributed lock corresponding to the second interface request are acquired;

[0014] If it is detected that the second distributed lock and the first distributed lock are mutually exclusive, the second interface request service is put into a waiting queue;

[0015] If no mutual exclusion is detected between the two distributed locks, the second interface request service is regarded as a duplicate request and is removed.

[0016] The present invention also provides a scheduling device, comprising:

[0017] The disassembly module is used to disassemble the report data download and storage business into three processes in advance: downloading file data, parsing file data, and persisting file data into the database;

[0018] A multi-message queue creation module is used to create a first message queue, a second message queue and a third message queue, wherein the first message queue contains downloaded file data, the second message queue contains parsed file data, the third message queue contains persistent storage of file data, and the first message queue, the second message queue and the third message queue run in parallel.

[0019] Further, a distributed lock is set for any interface request service, and the device includes:

[0020] A deduplication detection module, configured to, when a request for a second interface request service is detected while one type of interface data is already occupied by a first interface request service, obtain a first distributed lock corresponding to the first interface request service and a second distributed lock corresponding to the second interface request service;

[0021] A first execution module, configured to put a second interface request service into a waiting queue if it is detected that the second distributed lock and the first distributed lock are mutually exclusive;

[0022] The second execution module is configured to regard the second interface request service as a duplicate request and remove it if no mutual exclusion is detected between the two distributed locks.

[0023] The present invention also provides a computer-readable medium on which a computer program is stored, characterized in that the above-mentioned scheduling method is implemented when the program is processed and executed.

[0024] The present invention also provides an electronic device, comprising:

[0025] one or more processors;

[0026] The storage device is used to store one or more programs, and when the one or more programs are executed by the one or more processors, the one or more processors implement the above-mentioned scheduling method.

[0027] In the present invention, the report data download and save service is divided into three processes corresponding to three message queue services: 1. Download file data in batches to produce files to be parsed; 2. Parse file data in batches to produce data persistence tasks; 3. Persistent storage of batch data. In this way, the efficiency is improved both horizontally and vertically. This embodiment improves the operating rate of the report data download and save service by improving both horizontally and vertically.

[0028] It is to be understood that both the foregoing general description and the following detailed description are exemplary and explanatory only and are not restrictive of the disclosure. BRIEF DESCRIPTION OF THE DRAWINGS

[0029] Figure 1 It is a flowchart of the steps of a scheduling method according to an embodiment of the present invention.

[0030] Figure 2 It is a module diagram of a scheduling method according to an embodiment of the present invention.

[0031] Figure 3 It is a step flow chart of a deduplication method according to one embodiment of the present invention.

[0032] Figure 4 It is a modular schematic diagram of a scheduling device according to an embodiment of the present invention. DETAILED DESCRIPTION

[0033] Embodiments of the present invention are described in detail below, examples of which are shown in the accompanying drawings, wherein the same or similar reference numerals throughout represent the same or similar elements or elements having the same or similar functions. The embodiments described below with reference to the accompanying drawings are exemplary and are only used to explain the present invention, and cannot be interpreted as limiting the present invention.

[0034] Those skilled in the art will appreciate that, unless otherwise stated, the singular forms "a", "an", "said" and "the" used herein may also include plural forms. It should be further understood that the term "comprising" used in the specification of the present invention refers to the presence of the features, integers, steps, operations, but does not exclude the presence or addition of one or more other features, integers, steps, operations.

[0035] It will be understood by those skilled in the art that, unless otherwise defined, all terms (including technical and scientific terms) used herein have the same meaning as generally understood by those skilled in the art in the art to which the present invention belongs. It should also be understood that terms such as those defined in general dictionaries should be understood to have meanings consistent with the meanings in the context of the prior art, and will not be interpreted with idealized or overly formal meanings unless specifically defined as herein.

[0036] See also Figure 1-2 The present invention first provides a scheduling method, which comprises the following steps:

[0037] S10: The report data download and storage business is pre-divided into three processes: downloading file data, parsing file data and persisting file data into the database.

[0038] S20: Create a first message queue, a second message queue and a third message queue, wherein the first message queue contains downloaded file data, the second message queue contains parsed file data, the third message queue contains persistent storage of file data, and the first message queue, the second message queue and the third message queue run in parallel.

[0039] The existing scheduling method is to set the process in one thread and call the function process of downloading file data, parsing file data, and persisting file data in the database in sequence. However, if you want to improve efficiency, you can only parallelize the entire process, which is equivalent to only improving horizontally.

[0040] In this embodiment, the download and save service of report data is actually divided into three processes corresponding to three message queue services: 1. Download file data in batches and produce files to be parsed; 2. Parse file data in batches and produce data persistence tasks; 3. Persistent storage of batch data. In this way, the efficiency is improved both horizontally and vertically. This embodiment improves the operating rate of the download and save service of report data by improving it both horizontally and vertically, ensuring the high availability and uninterruptedness of the interface request service and improving the system throughput.

[0041] In a specific implementation provided by the present invention, assuming that the first message queue takes 1 second, the second message queue takes 2 seconds, and the third message queue takes 1 second, a process takes a total of 4 seconds to process. If there are a total of 100,000 task data to be processed, the solution adopted by the prior art in the single-threaded case takes a total of 100,000*4=400,000 seconds. If multi-threaded processing is used, 10 threads are opened under certain computing resources, then ideally it takes 400,000 / 10=40,000 seconds.

[0042] In the technical solution provided by the present invention, when the amount of data is large, the time required for n data is (1+2+1)+(n-1)max(1,2,1). Because the process is dependent, the data execution time of the first task is still serial, so it takes 4 seconds. The subsequent data processes are theoretically parallel, so only the maximum time-consuming process node time is required. Therefore, assuming that each process node is single-threaded, the time required to process 100,000 tasks is 4+(99999*2)=200002 seconds, and then 10 threads are opened under the same certain resources, then the ideal time is 200002 / 10=20000.2 seconds.

[0043] Preferably, in one embodiment of the present invention, the ratio of any two values ​​of the computing resources occupied by the first message queue, the computing resources occupied by the second message queue, and the computing resources occupied by the third message queue is equal to the inverse ratio of the time required for the corresponding message queues with the same computing resources, so that the time spent by the first message queue, the second message queue and the third message queue in the single-threaded case is the same.

[0044] Assume that the first message queue takes 1 second to execute, the second message queue takes 2 seconds to execute, and the third message queue takes 1 second to execute, that is, the time spent by the first message queue, the second message queue and the third message queue with the same computing resources is in a ratio of 1:2:1. At this time, the ratio of the computing resources occupied by the first message queue, the computing resources occupied by the second message queue and the computing resources occupied by the third message queue is set to 1:2:1. It can be concluded that the time spent by the first message queue, the second message queue and the third message queue in the single-threaded case is in a ratio of 1:1:1.

[0045] That is, we can easily improve efficiency by expanding capacity on demand according to the bottleneck of the process. In this example, the time consumed by the second message queue is 2 seconds. Since we can use twice the computing resources on the second message queue alone, theoretically, the time consumed by each message queue is 1 second. Therefore, the time consumed by 10 tasks is 4+(99999*max(1,1,1))=100003 seconds. If we open 10 threads, 100003 / 10=10000.3 seconds. We only need to increase resources for the bottleneck process nodes to maximize efficiency.

[0046] Specifically, in an embodiment provided by the present invention, the data request service is pre-deployed across regions, and a method for preventing network fluctuations is also included, including:

[0047] When an abnormality is detected in the network line of a certain area, the tasks corresponding to that area will be distributed to other areas for deployment and processing.

[0048] In this embodiment, the service is deployed through cross-cloud and cross-region multi-IDC distributed redundant deployment. For example, the Kubernetes clusters of different vendors in different regions, such as Alibaba Cloud, Huawei Cloud, AWS Amazon Cloud, etc., are used, and VPN dedicated line connections are used. Under the containerization technology, the services are deployed to the Kubernetes clusters in different regions of each cloud vendor, and cross-cloud and cross-region multi-IDC distributed redundant deployment is achieved; multi-region deployment also ensures the stability of the service when the regional network is unstable.

[0049] In this embodiment, the report request service is deployed simultaneously in the Alibaba Cloud Silicon Valley region in the United States, the Huawei Cloud South China Guangzhou region, the Huawei Cloud Hong Kong region, and the AWS Amazon Cloud Oregon region. When an abnormality is detected in the network line in a certain area, the corresponding tasks in that area are allocated to other areas for deployment and processing to prevent the jitter of the international network line in a certain area from affecting the stability of the service.

[0050] Preferably, the speeds of connecting to the server in different regions are compared, and the CDN node of the fastest regional network is used as the responding node.

[0051] In this embodiment, under the ideal situation that the application network is stable in all regions, the CDN node with the fastest connection to the server in different regions can be used as the node for processing request services, which can greatly improve the network io efficiency.

[0052] Preferably, Figure 3 As shown, the scheduling method also includes a request deduplication step, and a distributed lock is set for any interface request service. The deduplication step includes:

[0053] S101: When a request for a second interface request service is detected when one interface data is already occupied by a first interface request service, a first distributed lock corresponding to the first interface request service and a second distributed lock corresponding to the second interface request are acquired.

[0054] S102: If it is detected that the second distributed lock and the first distributed lock are mutually exclusive, the second interface request service is put into a waiting queue.

[0055] S103: If no mutual exclusion is detected between the two distributed locks, the second interface request service is regarded as a duplicate request and is removed.

[0056] In this embodiment, asynchronous processing is used to improve system performance, reduce peak loads, and reduce system coupling.

[0057] In one embodiment, the present invention provides a scheduling device 100, wherein the device 100 stores a plurality of instructions, wherein the instructions are suitable for being loaded by a processor and executing a scheduling method, including:

[0058] S10: The report data download and storage business is pre-divided into three processes: downloading file data, parsing file data and persisting file data into the database.

[0059] S20: Create a first message queue, a second message queue and a third message queue, wherein the first message queue contains downloaded file data, the second message queue contains parsed file data, the third message queue contains persistent storage of file data, and the first message queue, the second message queue and the third message queue run in parallel.

[0060] For ease of description, the device 100 is divided into a functional module architecture, such as Figure 4 As shown, including:

[0061] The disassembly module 10 is used to disassemble the download and storage business of the report data into three processes in advance: downloading the file data, parsing the file data and persisting the file data into the database.

[0062] A multi-message queue creation module 20 is used to create a first message queue, a second message queue and a third message queue, wherein the first message queue contains downloaded file data, the second message queue contains parsed file data, the third message queue contains persistent storage of file data, and the first message queue, the second message queue and the third message queue run in parallel.

[0063] Specifically, in one embodiment of the present invention, a distributed lock is set for any interface request service, and the device 100 includes:

[0064] The deduplication detection module 30 is used to obtain a first distributed lock corresponding to the first interface request service and a second distributed lock corresponding to the second interface request when a request for a second interface request service is detected when one interface data is already occupied by a first interface request service.

[0065] The first execution module 40 is configured to put the second interface request service into a waiting queue if it is detected that the second distributed lock and the first distributed lock are mutually exclusive.

[0066] The second execution module 50 is configured to regard the second interface service request as a duplicate request and remove it if no mutual exclusion is detected between the two distributed locks.

[0067] The present invention also provides a computer-readable medium on which a computer program is stored, characterized in that the above-mentioned scheduling method is implemented when the program is processed and executed.

[0068] The present invention also provides an electronic device, comprising:

[0069] one or more processors;

[0070] The storage device is used to store one or more programs, and when the one or more programs are executed by the one or more processors, the one or more processors implement the above-mentioned scheduling method.

[0071] If the module / unit integrated in the electronic device is implemented in the form of a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the present invention implements all or part of the process in the multi-opening method of the application described in the above embodiment, and can also be completed by a computer program to indicate the relevant hardware. The computer program can be stored in a computer-readable storage medium. When the computer program is executed by the processor, the steps of the scheduling method described in the above method embodiment can be implemented. Among them, the computer program includes computer program code, and the computer program code can be in source code form, object code form, executable file or some intermediate form. The computer-readable medium may include: any entity or device that can carry the computer program code, recording medium, U disk, mobile hard disk, disk, optical disk, computer memory, read-only memory (ROM, Read-Only Memory), random access memory (RAM, Random Access Memory), electrical carrier signal, telecommunication signal and software distribution medium, etc. It should be noted that the content contained in the computer-readable medium can be appropriately increased or decreased according to the requirements of legislation and patent practice in the jurisdiction. For example, in some jurisdictions, according to legislation and patent practice, computer-readable media does not include electrical carrier signals and telecommunication signals.

[0072] In the specification and claims of the present application, the words "include / comprise" and the words "have / include" and their variations are used to specify the existence of stated features, values, steps or components, but do not exclude the existence or addition of one or more other features, values, steps, components or combinations thereof.

[0073] Some features of the present invention are described in different embodiments for clarity of description, however, these features may also be described in combination in a single embodiment. Conversely, some features of the present invention are described in only a single embodiment for brevity, however, these features may also be described in different embodiments alone or in any suitable combination.

[0074] The above describes in detail the various specific implementations of the scheduling method and device of the present invention. Finally, it should be noted that the above specific implementations are only used to illustrate the technical solution of the present invention rather than to limit it. Although the present invention has been described in detail with reference to the above specific implementations, a person of ordinary skill in the art should understand that the specific implementations of the present invention can still be modified or some technical features can be replaced by equivalents, and without departing from the spirit of the technical solution of the present invention, they should all be included in the scope of the technical solution for which protection is requested by the present invention.

Claims

1. A scheduling method, characterized in that: The method comprises the following steps: The download and storage of report data is pre-divided into three processes: downloading file data, parsing file data, and persisting file data into the database. Create a first message queue, a second message queue and a third message queue, wherein the first message queue contains downloaded file data, the second message queue contains parsed file data, the third message queue contains persistent storage of file data, and the first message queue, the second message queue and the third message queue run in parallel; The ratio of any two values ​​of the computing resources occupied by the first message queue, the computing resources occupied by the second message queue, and the computing resources occupied by the third message queue is equal to the ratio of the time required for the corresponding message queues with the same computing resources, so that the time spent by the first message queue, the second message queue and the third message queue in the single-threaded case is the same.

2. The scheduling method according to claim 1, characterized in that: Pre-deploy data request services across regions, including methods to prevent network fluctuations, including: When an abnormality is detected in the network line of a certain area, the tasks corresponding to that area will be distributed to other areas for deployment and processing.

3. The scheduling method according to claim 2, characterized in that: Compare the speeds of connecting to the server in different regions, and use the CDN node of the fastest regional network as the response node.

4. The scheduling method according to claim 1, characterized in that: It also includes a request deduplication method, which sets a distributed lock for any interface request service, and the deduplication method includes: When a request for a second interface request service is detected when one interface data is already occupied by a first interface request service, a first distributed lock corresponding to the first interface request service and a second distributed lock corresponding to the second interface request are acquired; If it is detected that the second distributed lock and the first distributed lock are mutually exclusive, the second interface request service is put into a waiting queue; If no mutual exclusion is detected between the two distributed locks, the second interface request service is regarded as a duplicate request and is removed.

5. A scheduling device, characterized in that: include: The disassembly module is used to disassemble the report data download and storage business into three processes in advance: downloading file data, parsing file data, and persisting file data into the database; A multi-message queue creation module is used to create a first message queue, a second message queue and a third message queue, wherein the first message queue contains downloaded file data, the second message queue contains parsed file data, and the third message queue contains persistent storage of file data, and the first message queue, the second message queue and the third message queue run in parallel, and the ratio of any two values ​​of the computing resources occupied by the first message queue, the computing resources occupied by the second message queue and the computing resources occupied by the third message queue is equal to the ratio of the time required for the corresponding message queues with the same computing resources, so that the time spent by the first message queue, the second message queue and the third message queue in the case of single thread is the same.

6. The dispatching device according to claim 5, characterized in that: A distributed lock is set for any interface request service, the device comprising: A deduplication detection module, configured to, when a request for a second interface request service is detected while one type of interface data is already occupied by a first interface request service, obtain a first distributed lock corresponding to the first interface request service and a second distributed lock corresponding to the second interface request service; A first execution module, configured to put a second interface request service into a waiting queue if it is detected that the second distributed lock and the first distributed lock are mutually exclusive; The second execution module is configured to regard the second interface request service as a duplicate request and remove it if no mutual exclusion is detected between the two distributed locks.

7. A computer readable medium having a computer program stored thereon, characterized in that: When the program is processed and executed, the scheduling method according to any one of claims 1 to 4 is implemented.

8. An electronic device, characterized in that: include: one or more processors; A storage device, used to store one or more programs, when the one or more programs are executed by the one or more processors, enables the one or more processors to implement the scheduling method as described in any one of claims 1-5.

Images