Scheduling method, system and computer device and storage medium for data processing

Abstract

The application relates to a scheduling method, system and computer device and storage medium for data processing. The scheduling method is applied to a data processing system, wherein the data processing system comprises multiple function models, multiple data sources and multiple scheduling queues. Each scheduling queue is connected with one function model and at least one data source, and one data source is connected with multiple scheduling queues. The scheduling method comprises the following steps: the data sources generate data combinations respectively and send the data combinations to the scheduling queues connected therewith, the data combinations comprising to-be-processed data and data source identifiers; the scheduling queues receive at least one data combination respectively, store the data combinations according to scheduling rules, and call the data combinations, and send the called data combinations to the function models; the function models receive the data combinations, process the to-be-processed data, and feed back processing results to the data sources based on the data source identifiers. The method can coordinate multiple function models to process data of multiple data sources, and is beneficial to improving coordination efficiency.

Description

Technical Field

[0001] This application relates to the field of software system technology, and in particular to a data processing scheduling method, a data processing system, a computer device, and a computer-readable storage medium. Background Technology

[0002] Current data storage technology has made significant progress compared to the past. While the storage capacity of data storage devices increases year by year, the price per unit of storage capacity is gradually decreasing. With advancements in data acquisition devices for data such as audio and video, and upgrades in storage technology, massive amounts of data are generated. Furthermore, thanks to the reduced price per unit of storage capacity, video and audio data are more easily accessed and obtained by users. Based on this, intelligent recognition, detection, and processing technologies for image / audio data have emerged, replacing traditional manual inspection in many fields.

[0003] However, there are still gaps in the technology for deploying multiple intelligent detection algorithms across multiple data sources. This results in low utilization rates of data from multiple data sources or multiple intelligent detection algorithms, leading to low efficiency in using various intelligent detection algorithms to detect data from multiple data sources. Summary of the Invention

[0004] Based on this, it is necessary to provide a data processing scheduling method, data processing system, computer equipment, and computer-readable storage medium that can coordinate multi-functional models to process data from multiple data sources and improve coordination efficiency, in order to address the above-mentioned technical problems.

[0005] On the one hand, a data processing scheduling method is provided, which is applied to a data processing system, wherein the data processing system includes multiple functional models, multiple data sources, and multiple scheduling queues; each scheduling queue is connected to a functional model and at least one data source, and a data source is connected to at least one scheduling queue; the scheduling method includes: each data source generating a data combination and sending it to the scheduling queue connected to it, the data combination including data to be processed and a data source identifier; each scheduling queue receiving at least one data combination, storing and retrieving it according to scheduling rules, and sending the retrieved data combination to the functional model; the functional model receiving the data combination, processing the data to be processed, and feeding back the processing result to the data source based on the data source identifier.

[0006] In one embodiment of this application, the scheduling queue includes multiple cache units; storing data according to scheduling rules includes: the scheduling queue sequentially storing each data combination in each cache unit according to the order in which the data combinations are received; wherein the number of cache units in each scheduling queue matches the number of data sources it is connected to.

[0007] In one embodiment of this application, storing and retrieving data according to scheduling rules and sending the retrieved data combination to the functional model includes: in response to the functional model being able to receive data to be processed, the functional model calls a data combination in a cache unit for processing; feeding back the processing result to the data source based on the data source identifier further includes: removing the processed data combination from the scheduling queue and receiving a new data combination, wherein the new data combination and the processed data combination originate from the same data source.

[0008] In one embodiment of this application, at least one scheduling model member identifier is predefined in the data source, and at least one data source member identifier is predefined in the functional model; the data processing scheduling method further includes: based on the completion of the configuration of the functional model and the data source, the data source and the functional model are connected through a scheduling queue using the scheduling model member identifier and the data source member identifier.

[0009] In one embodiment of this application, verifying the configuration completion of the functional model and data source includes: obtaining configuration information of the data source and configuration information of the functional model, wherein the configuration information includes the defined scheduling model member identifier / data source member identifier; comparing the configuration information with a preset configuration file to determine whether it is as required by the preset configuration file, wherein the preset configuration file includes multiple data source information, multiple functional model information, and connection relationship information between the data source and the functional model; and determining that the configuration completion of the functional model and data source is achieved in response to the configuration information being as required by the preset configuration file.

[0010] In one embodiment of this application, the data to be processed is obtained by copying the original data from the data source; and / or, the processing result is the original result after the functional model has completed the processing.

[0011] In one embodiment of this application, each scheduling queue receiving at least one data combination includes: the scheduling queue using an independent thread to parse and read the data combination from each data source.

[0012] On the other hand, a data processing system is provided, comprising: multiple data sources, multiple functional models, and multiple scheduling queues; each scheduling queue is connected to a functional model and at least one data source, and each data source is connected to at least one scheduling queue; the data sources are used to generate data combinations and send them to the scheduling queues they are connected to, the data combinations including data to be processed and a data source identifier; the scheduling queues are used to receive at least one data combination, store and retrieve it according to scheduling rules, and send the retrieved data combination to the functional model; the functional model is used to receive the data combination, process the data to be processed, and feed back the processing result to the data source based on the data source identifier.

[0013] On another front, a computer device is provided, including a memory, a processor, and a computer program stored in the memory and executable on the processor. When the processor executes the computer program, it performs the following steps: data sources generate data combinations and send them to a scheduling queue connected to them. The data combinations include data to be processed and a data source identifier. The scheduling queue receives at least one data combination, stores and retrieves it according to scheduling rules, and sends the retrieved data combination to a functional model. The functional model receives the data combination, processes the data to be processed, and feeds back the processing result to the data source based on the data source identifier.

[0014] On another front, a computer-readable storage medium is provided, on which a computer program is stored. When the computer program is executed by a processor, it performs the following steps: the data source generates data combinations and sends them to a scheduling queue connected to it. The data combinations include data to be processed and a data source identifier; the scheduling queue receives at least one data combination, stores and calls it according to scheduling rules, and sends the called data combination to the functional model; the functional model receives the data combination, processes the data to be processed, and feeds back the processing result to the data source based on the data source identifier.

[0015] The aforementioned data processing scheduling method, data processing system, computer equipment, and computer-readable storage medium involve a scheduling queue connected to a functional model and at least one data source. The data source is connected to at least one scheduling queue, i.e., connected to at least one functional model. When a data source receives data that needs processing, it generates a data combination and sends the data combination to the at least one scheduling queue it is connected to, so that the functional models connected to each scheduling queue can process the data combination. Simultaneously, the data combination includes not only the data to be processed by the functional models but also a data source identifier that identifies its source data source. After the functional model completes the processing of the data to be processed, it can feed back the processing result to the data source from which the current data combination originated, based on the data source identifier. Thus, this application can coordinate multiple functional models to process data from multiple data sources. Furthermore, since each functional model is connected to at least one data source through a scheduling queue, it reduces the need for additional scheduling devices to identify the data combination and confirm the at least one functional model to be invoked after the data source sends it out, thereby improving coordination efficiency. Attached Figure Description

[0016] Figure 1 This is an application environment diagram of an embodiment of the data processing scheduling method of this application;

[0017] Figure 2 This is a flowchart illustrating an embodiment of the data processing scheduling method of this application;

[0018] Figure 3 This is a structural block diagram of an embodiment of the data processing system of this application;

[0019] Figure 4 This is a schematic diagram of the structure of an embodiment of the data processing system of this application;

[0020] Figure 5 This is a flowchart illustrating another embodiment of the data processing scheduling method of this application;

[0021] Figure 6 This is a schematic diagram of the structure of an embodiment of the scheduling queue of this application;

[0022] Figure 7 This is a flowchart illustrating an embodiment of the verification method for the functional model and data source configuration of this application;

[0023] Figure 8 This is a schematic diagram of the internal structure of an embodiment of the computer device of this application. Detailed Implementation

[0024] To make the objectives, technical solutions, and advantages of this application clearer, the following detailed description is provided in conjunction with the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are merely illustrative and not intended to limit the scope of this application.

[0025] The data processing scheduling method provided in this application can be applied to, for example... Figure 1 In the application environment shown, Figure 1 This is an application environment diagram of an embodiment of the data processing scheduling method of this application.

[0026] Terminal 102 communicates with server 104 via a network. Terminal 102 can collect data that needs to be processed, such as video images, audio, and user physiological data. Terminal 102 sends the collected data to server 104, which processes the data. For example, server 104 can perform pedestrian detection, harmful animal detection, and road rage detection on video images collected by multiple terminals; server 104 can perform noise monitoring, speech-to-text recognition, and noise reduction on audio collected by multiple terminals; and server 104 can perform health status monitoring, fall detection, and traffic accident detection on user physiological data collected by multiple terminals, etc., which will not be elaborated further here.

[0027] The terminal 102 can be, but is not limited to, various monitoring devices, recording devices, personal computers, laptops, smartphones, tablets and portable wearable devices, and the server 104 can be implemented by a standalone server or a server cluster composed of multiple servers.

[0028] Taking the acquisition of video images by terminal 102 as an example, the functions that server 104 can perform are illustrated below. In the manufacturing field, multiple cameras (i.e., terminals 102) can be deployed on multiple production lines. By triggering image capture, convolutional neural network models can be used to intelligently detect products, such as cracks, dimensions, and color deviations. In the transportation field, by adding cameras to traffic light intersections, video image data from traffic intersections can be used for detection and structured information extraction, such as traffic flow statistics, identification of license plates of illegal vehicles, and pedestrian statistics. In the security field, existing cameras can be used to acquire on-site video data, and intelligent algorithms can be used to detect abnormal security behaviors in the video data.

[0029] In one embodiment, a data processing scheduling method is provided, such as Figure 2 As shown, Figure 2 This is a flowchart illustrating an embodiment of the data processing scheduling method of this application. The data processing scheduling method can be applied to, for example... Figure 1 In the application scenarios shown, Figure 1 Based on the application scenarios shown, a data processing system can be formed, that is, the data processing scheduling method can be specifically applied to a data processing system.

[0030] Please refer to both together. Figure 3 and Figure 4 , Figure 3 This is a structural block diagram of an embodiment of the data processing system of this application. Figure 4 This is a schematic diagram of the structure of an embodiment of the data processing system of this application.

[0031] The data processing system includes multiple data sources 301, multiple functional models 302, and multiple scheduling queues 303. Each scheduling queue 303 is connected to one functional model 302 and at least one data source 301, and one data source 301 is connected to at least one scheduling queue 303.

[0032] The data source 301 is a device for acquiring data to be processed through methods such as acquisition / collection. The functional model 302 is a device for processing the data to be processed, such as a neural network model.

[0033] The specific structure of the data processing system will be described in detail later.

[0034] Please continue reading. Figure 2 The following explanation uses the application of a data processing scheduling method to a data processing system as an example, including the following steps:

[0035] S201: The data source generates a data combination and sends it to the scheduling queue connected to it. The data combination includes the data to be processed and the data source identifier.

[0036] In this embodiment, the data to be processed is data that needs to be processed using a functional model, and can be images, audio, physiological data, etc. Since there are multiple data sources, a data source identifier is used to identify the data source from which the data to be processed originates.

[0037] Specifically, the data source acquires the data to be processed, generates a data source identifier before sending it to the functional model for processing, and sends the data to be processed and the data source identifier together (i.e., the data combination) to the scheduling queue connected to it. If the data source is connected to only one scheduling queue, the data combination is sent to that scheduling queue; if the data source is connected to multiple scheduling queues, the data combination can be sent to the scheduling queues simultaneously, or the data combination can be sent to each scheduling queue sequentially according to a preset rule. The preset rule may be waiting for notification from the functional model or sending the data combination to a scheduling queue at intervals, which is not limited here.

[0038] The data to be processed can be the original data that needs to be processed, or it can be copied / derived data obtained by processing the original data. There is no limitation here.

[0039] S202: The scheduling queue receives at least one data combination, stores and retrieves it according to the scheduling rules, and sends the retrieved data combination to the functional model.

[0040] In this embodiment, each scheduling queue receives data combinations originating from a data source connected to it. Since each scheduling queue is connected to at least one data source, the number of data combinations received by the scheduling queue may be one or more. The scheduling queue can store the received data combinations according to preset scheduling rules and send them to the functional model as needed.

[0041] S203: The functional model receives data combinations, processes the data to be processed, and feeds back the processing results to the data source based on the data source identifier.

[0042] In this embodiment, the functional model receives a data combination, processes the data to be processed in the data combination according to its preset functions, and obtains the processing result. Based on the data source identifier in the current data combination, the data source from which the data combination originates is determined, and the processing result is fed back to the data source from which the data combination originates.

[0043] For example, if the function of the current module is set to detect whether there is a person in the image, then the image is processed to extract features and detect whether there is a person. If a person is detected in the image, the coordinates of the person's position or the pedestrian bounding box formed based on the coordinates are output, and the coordinates of the position or the pedestrian bounding box is used as the processing result.

[0044] Therefore, in the above data processing scheduling method, one functional model is connected to at least one data source, and one data source is connected to at least one functional model, thereby enabling the connection between multiple data sources and multiple functional models. This facilitates scheduling multiple functional models to process data from multiple data sources. Simultaneously, after a functional model completes processing the data to be processed, it can feed back the processing result to the data source from which the current data combination originated, based on the data source identifier. Thus, the data processing scheduling method in this embodiment reduces the need for additional scheduling devices to identify the data combination after the data source sends out the data combination, to call the required functional model to process the data, and to separately identify the data source from which the processing result needs to be fed back, thus also improving coordination efficiency.

[0045] In one embodiment, a data processing scheduling method is also provided, such as Figure 5 As shown, Figure 5 This is a flowchart illustrating another embodiment of the data processing scheduling method of this application. Please refer to it in conjunction with the above. Figure 6 , Figure 6 This is a schematic diagram of the structure of an embodiment of the scheduling queue of this application.

[0046] In this embodiment, at least one scheduling model member identifier is predefined within the data source, and at least one data source member identifier is predefined within the functional model. After verifying the configuration of the functional model and data source, the data source and functional model are connected via a scheduling queue using the scheduling model member identifier and the data source member identifier. The verification method for the functional model and data source configuration will be described later.

[0047] S501: Data combinations are generated separately from the data sources.

[0048] In this embodiment, the data combination includes the data to be processed and the data source identifier. The data to be processed can be images, audio, etc.

[0049] Optionally, the data to be processed is obtained by copying the original data from the data source. This ensures that the original data remains in the data source when the data to be processed is sent to the functional model for processing and when the processing results are returned. This facilitates the isolation and decoupling of the original data from the data to be processed and the processing results. Alternatively, derived data can be obtained by performing super-resolution reconstruction / denoising and other processing on the original data or the copied data.

[0050] Furthermore, the data set can also include the configuration of the data source and information about the data to be processed. Taking an image as the data to be processed as an example, the data set can also include the image size, the resolution of the image obtained by the data source, the frame rate, and the storage method.

[0051] S502: Data combination sent by the data source.

[0052] In this embodiment, each data source sends combined data to at least one scheduling queue connected to it.

[0053] For example, if a data source connects to four functional models through four scheduling queues, meaning the data to be processed from this data source can undergo four different processing methods, then the original data is copied to obtain four copies of the data to be processed. These four copies of the data to be processed are then sent to their respective scheduling queues.

[0054] S503: The scheduling queue receives at least one data combination and stores it according to the scheduling rules.

[0055] In this embodiment, the scheduling queue uses an independent thread to parse and read the data combination of each data source, which helps to decouple different data streams from different data sources and makes the control and processing of any data stream more efficient and convenient.

[0056] Furthermore, the scheduling queue includes multiple cache units, which are used to store data combinations.

[0057] Specifically, the scheduling queue stores each data combination in its corresponding cache unit according to the order in which they are received. This allows the earlier-acquired data combination to be sent to the functional model when subsequent data combinations are pushed to it, ensuring that the functional model processes data on a first-come, first-served basis. This eliminates the inherent flaw in existing technologies where data may be lost while waiting, thus rationalizing the working mode of the data processing system.

[0058] In this system, the number of cache units in each scheduling queue is matched with the number of data sources they are connected to. For example, the number of cache units can be two times, three times, or equal to the number of data sources connected to them, or the data in each cache unit can be equal to the number of data sources connected to them.

[0059] With this design, each data source can only remove the completed / being processed data combination from the scheduling queue and push the new data combination into the message queue after receiving a notification from the functional model that it is processing the pending data of its current data source or has completed processing the pending data of its current data source. Furthermore, the number of data in the cache unit is equivalent to the number of connected data sources, which can reduce the memory consumption of the data processing system while ensuring that the pending data of each data source can be processed in a timely manner, and naturally also helps to limit the length of the message queue.

[0060] S504: The scheduling queue calls data combinations according to the scheduling rules and sends the called data combinations to the functional model.

[0061] In this embodiment, in response to the functional model receiving and processing the data to be processed, the functional model calls a data combination in a cache unit for processing.

[0062] Optionally, after the functional model completes the previous processing of the data to be processed, or when the data processing system initially starts working, the scheduling queue calls the first data combination obtained from the stored data combinations and pushes it to the functional model for processing.

[0063] S505: The functional model receives data combinations, processes the data to be processed, and obtains the processing result.

[0064] In this embodiment, the functional model processes the data to be processed in the received data combination to obtain the processing result.

[0065] Optionally, the processing result is the original result after the functional model has completed processing. Alternatively, the processing result may be a prompt message to notify the data source that the processing is complete.

[0066] For example, a functional model's function is to detect the presence of a person in an image. After receiving a combination of data, the functional model performs feature extraction and detection on the image as the data to be processed. If a person is detected in the image, the four positioning points of the outer rectangle of the person's outline are used as the processing result. This allows for more flexible acquisition and display of the intelligent algorithm results, and also facilitates subsequent connections with the user's front-end webpage.

[0067] S506: The functional model feeds back the processing results to the data source based on the data source identifier.

[0068] In this embodiment, in response to the inclusion of a data source identifier in the data combination, the functional model can determine the data source from which the data combination originates based on the data source identifier. The functional model can then feed the processing result back to the data source. Taking the processing result as a prompt message as an example, the data source obtains the returned processing result, retrieves the completed processing result, and displays it according to preset display rules.

[0069] In this embodiment, if the scheduling queue is not empty, each data combination is retrieved from iteratively for processing. After processing each piece of data, the processing result is notified to the data source through a defined data source member identifier. Furthermore, when the functional model feeds back the processing result to the data source, the scheduling queue can remove the processed data combination. New data combinations are received, and these new data combinations originate from the same data source as the processed data combinations. In other words, the data source sends a new data combination to the scheduling queue only after receiving notification from the functional model that it has processed the previously sent data combination, i.e., step S502 is executed. The generation of data combinations in step S501 can be achieved either after the data source receives the data to be processed, or after the data source receives notification from the functional model that the data combination processing is complete; this is not limited here.

[0070] In an alternative embodiment, the functional model may also identify the data source identifier within the data combination after receiving it to determine the data source from which the data combination originates. A notification is then pushed to the corresponding data source, which, in response to receiving that a data combination is being processed by the functional model, sends a new data combination to a scheduling queue.

[0071] In the above embodiments, the data processing scheduling method ends in response to receiving an exit instruction.

[0072] In one embodiment, a data processing scheduling method is also provided, which includes a verification method for functional models and data source configurations, such as... Figure 7 As shown, Figure 7 This is a flowchart illustrating an embodiment of the verification method for the functional model and data source configuration of this application.

[0073] S701: Obtain configuration information for the data source and configuration information for the functional model.

[0074] In this embodiment, the configuration information of the data source includes the defined scheduling model member identifier, and the configuration information of the functional model includes the defined data source member identifier.

[0075] The scheduling model member identifier is used to identify the functional model that the data source needs to connect to. Similarly, the data source member identifier is used to identify the data source that the scheduling model connects to. In this way, by defining member identifiers for each other within the data source and the functional model, references and access to each other can be achieved, allowing for more flexible control and data access over the real-time process of algorithm detection.

[0076] S702: Compare the configuration information with the preset configuration file to determine whether it meets the requirements of the preset configuration file.

[0077] In this embodiment, if the configuration information and the preset configuration file are as required by the preset configuration file, it is assumed that the data source and the functional model can be connected, and step S703 is executed. If the configuration information and the preset configuration file are not as required by the preset configuration file, it is assumed that the data source and the functional model are not yet connected, and at least one scheduling model member identifier is predefined in the data source again, and at least one data source member identifier is predefined in the functional model, and step S701 is repeated; or the process ends and feedback information is given to the user.

[0078] The default configuration file includes information on multiple data sources, multiple functional models, and the connection relationship between the data sources and functional models.

[0079] S703: The configuration of the judgment and verification function model and data source is complete.

[0080] In this embodiment, in response to the configuration information as required by the preset configuration file, it is determined that the verification function model and the data source are configured, the data source and the function model are connected, and the function model can receive the data to be processed for processing.

[0081] Optionally, the data source and functional model can be defined during initialization. Taking the video source as the data source and the algorithm entity as the functional model as an example, a video source class is defined to abstract the video hardware device, and the acquired image data, such as resolution, frame rate, and storage method, is initialized. An algorithm entity member identifier is defined within the video source class to import the algorithm entity. An algorithm entity class is defined to abstract the intelligent detection algorithm, and the algorithm's scheduling queue is initialized. Information such as the algorithm name and the imported video source are also initialized. A video source member variable is defined within the algorithm entity class to import the video source.

[0082] It should be understood that, although Figure 2 , Figure 5 as well as Figure 7 The steps in the flowchart are shown sequentially as indicated by the arrows, but these steps are not necessarily executed in the order indicated by the arrows. Unless otherwise specified in this document, there is no strict order in which these steps are executed, and they can be performed in other orders. Furthermore, Figure 2 , Figure 5 as well as Figure 7 At least some of the steps in the process may include multiple sub-steps or multiple stages. These sub-steps or stages are not necessarily completed at the same time, but can be executed at different times. The execution order of these sub-steps or stages is not necessarily sequential, but can be executed in turn or alternately with other steps or at least some of the sub-steps or stages of other steps.

[0083] In one embodiment, a data processing system is provided, such as Figure 8 As shown, Figure 8 This is a schematic diagram of the structure of an embodiment of the data processing system of this application.

[0084] The data processing system includes multiple data sources, multiple functional models, and multiple scheduling queues. Each scheduling queue is connected to one functional model and at least one data source, and each data source is connected to at least one scheduling queue.

[0085] The data source is used to generate data combinations and send them to the scheduling queue connected to it. The data combination includes the data to be processed and the data source identifier.

[0086] The scheduling queue is used to receive at least one data combination, store and retrieve it according to the scheduling rules, and send the retrieved data combination to the functional model.

[0087] The functional model is used to receive data combinations, process the data to be processed, and feed the processing results back to the data source based on the data source identifier.

[0088] Furthermore, the scheduling queue includes multiple buffer units, and the scheduling queue stores each data combination sequentially in the buffer unit according to the order in which the data combinations are received. The number of buffer units in each scheduling queue matches the number of data sources it is connected to.

[0089] For specific limitations regarding the data processing system, please refer to the limitations on the data processing scheduling method described above, which will not be repeated here. Each module in the aforementioned data processing system can be implemented entirely or partially through software, hardware, or a combination thereof. These modules can be embedded in hardware or independent of the processor in the computer device, or stored in software in the memory of the computer device, so that the processor can call and execute the operations corresponding to each module.

[0090] In one embodiment, a computer device is provided, which may be a server, and its internal structure diagram may be as follows: Figure 8 As shown, Figure 8 This is a schematic diagram of the internal structure of a computer device according to an embodiment of this application. The computer device includes a processor, a memory, and a network interface connected via a system bus. The processor provides computing and control capabilities. The memory includes a non-volatile storage medium and internal memory. The non-volatile storage medium stores an operating system, computer programs, and a database. The internal memory provides an environment for the operation of the operating system and computer programs in the non-volatile storage medium. The network interface is used for communication with external terminals via a network connection. When the computer program is executed by the processor, it implements a data processing scheduling method.

[0091] Those skilled in the art will understand that Figure 8 The structure shown is merely a block diagram of a portion of the structure related to the present application and does not constitute a limitation on the computer device to which the present application is applied. Specific computer devices may include more or fewer components than those shown in the figure, or combine certain components, or have different component arrangements.

[0092] In one embodiment, a computer device is provided, including a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor executes the computer program to perform the following steps:

[0093] S201: The data source generates a data combination and sends it to the scheduling queue connected to it. The data combination includes the data to be processed and the data source identifier.

[0094] S202: The scheduling queue receives at least one data combination, stores and retrieves it according to the scheduling rules, and sends the retrieved data combination to the functional model.

[0095] S203: The functional model receives data combinations, processes the data to be processed, and feeds back the processing results to the data source based on the data source identifier.

[0096] In one embodiment, when the processor executes the computer program, it also implements another implementation of the data processing scheduling method that the data processing system of this application can achieve, specifically as follows:

[0097] In this embodiment, at least one scheduling model member identifier is predefined within the data source, and at least one data source member identifier is predefined within the functional model. After verifying the configuration of the functional model and data source, the data source and functional model are connected via a scheduling queue using the scheduling model member identifier and the data source member identifier. The verification method for the functional model and data source configuration will be described later.

[0098] S501: Data combinations are generated separately from the data sources.

[0099] In this embodiment, the data combination includes the data to be processed and the data source identifier. The data to be processed is obtained by copying the original data from the data source.

[0100] S502: Data combination sent by the data source.

[0101] In this embodiment, each data source sends combined data to at least one scheduling queue connected to it.

[0102] S503: The scheduling queue receives at least one data combination and stores it according to the scheduling rules.

[0103] In this embodiment, the scheduling queue uses an independent thread to parse and read the data combination for each data source.

[0104] The scheduling queue comprises multiple buffer units, which store data combinations sequentially in the order they are received. The number of buffer units in each scheduling queue corresponds to the number of data sources it connects to.

[0105] S504: The scheduling queue calls data combinations according to the scheduling rules and sends the called data combinations to the functional model.

[0106] In this embodiment, in response to the functional model receiving and processing the data to be processed, the functional model calls a data combination in a cache unit for processing.

[0107] S505: The functional model receives data combinations, processes the data to be processed, and obtains the processing result.

[0108] S506: The functional model feeds back the processing results to the data source based on the data source identifier.

[0109] In this embodiment, the processing result is the original result after the functional model has completed processing.

[0110] Furthermore, when the functional model feeds back the processing result to the data source, the scheduling queue can remove the data combination that has been processed. A new data combination is received, and this new data combination originates from the same data source as the processed data combination. In other words, the data source only sends a new data combination to the scheduling queue after receiving notification from the functional model that it has finished processing the previously sent data combination, i.e., step S502 is executed. The generation of the data combination in step S501 can be achieved either after the data source receives the data to be processed, or after the data source receives notification from the functional model that the data combination processing is complete; this is not limited here.

[0111] In an alternative embodiment, the functional model may also identify the data source identifier within the data combination after receiving it to determine the data source from which the data combination originates. A notification is then pushed to the corresponding data source, which, in response to receiving that a data combination is being processed by the functional model, sends a new data combination to a scheduling queue.

[0112] In one embodiment, a data processing scheduling method is also provided, which includes a verification method for a functional model and a data source configuration. When the processor executes the computer program, it further implements the following steps:

[0113] S701: Obtain configuration information for the data source and configuration information for the functional model.

[0114] In this embodiment, the configuration information of the data source includes the defined scheduling model member identifier, and the configuration information of the functional model includes the defined data source member identifier.

[0115] S702: Compare the configuration information with the preset configuration file to determine whether it meets the requirements of the preset configuration file.

[0116] In this embodiment, if the configuration information and the preset configuration file are as required by the preset configuration file, it is assumed that the data source and the functional model can be connected, and step S703 is executed. If the configuration information and the preset configuration file are not as required by the preset configuration file, it is assumed that the data source and the functional model are not yet connected, and at least one scheduling model member identifier is predefined in the data source again, and at least one data source member identifier is predefined in the functional model, and step S701 is repeated; or the process ends and feedback information is given to the user.

[0117] The default configuration file includes information on multiple data sources, multiple functional models, and the connection relationship between the data sources and functional models.

[0118] S703: The configuration of the judgment and verification function model and data source is complete.

[0119] In this embodiment, in response to the configuration information as required by the preset configuration file, it is determined that the verification function model and data source configuration are complete.

[0120] In one embodiment, a computer-readable storage medium is provided having a computer program stored thereon, the computer program performing the following steps when executed by a processor:

[0121] S201: The data source generates a data combination and sends it to the scheduling queue connected to it. The data combination includes the data to be processed and the data source identifier.

[0122] S202: The scheduling queue receives at least one data combination, stores and retrieves it according to the scheduling rules, and sends the retrieved data combination to the functional model.

[0123] S203: The functional model receives data combinations, processes the data to be processed, and feeds back the processing results to the data source based on the data source identifier.

[0124] In one embodiment, when the computer program is executed by the processor, it also implements another implementation of the data processing scheduling method that the data processing system of this application can achieve, specifically as follows:

[0125] In this embodiment, at least one scheduling model member identifier is predefined within the data source, and at least one data source member identifier is predefined within the functional model. After verifying the configuration of the functional model and data source, the data source and functional model are connected via a scheduling queue using the scheduling model member identifier and the data source member identifier. The verification method for the functional model and data source configuration will be described later.

[0126] S501: Data combinations are generated separately from the data sources.

[0127] In this embodiment, the data combination includes the data to be processed and the data source identifier. The data to be processed is obtained by copying the original data from the data source.

[0128] S502: Data combination sent by the data source.

[0129] In this embodiment, each data source sends combined data to at least one scheduling queue connected to it.

[0130] S503: The scheduling queue receives at least one data combination and stores it according to the scheduling rules.

[0131] In this embodiment, the scheduling queue uses an independent thread to parse and read the data combination for each data source.

[0132] The scheduling queue comprises multiple buffer units, which store data combinations sequentially in the order they are received. The number of buffer units in each scheduling queue corresponds to the number of data sources it connects to.

[0133] S504: The scheduling queue calls data combinations according to the scheduling rules and sends the called data combinations to the functional model.

[0134] In this embodiment, in response to the functional model receiving and processing the data to be processed, the functional model calls a data combination in a cache unit for processing.

[0135] S505: The functional model receives data combinations, processes the data to be processed, and obtains the processing result.

[0136] S506: The functional model feeds back the processing results to the data source based on the data source identifier.

[0137] In this embodiment, the processing result is the original result after the functional model has completed processing.

[0138] Furthermore, when the functional model feeds back the processing result to the data source, the scheduling queue can remove the data combination that has been processed. A new data combination is received, and this new data combination originates from the same data source as the processed data combination. In other words, the data source only sends a new data combination to the scheduling queue after receiving notification from the functional model that it has finished processing the previously sent data combination, i.e., step S502 is executed. The generation of the data combination in step S501 can be achieved either after the data source receives the data to be processed, or after the data source receives notification from the functional model that the data combination processing is complete; this is not limited here.

[0139] In an alternative embodiment, the functional model may also identify the data source identifier within the data combination after receiving it to determine the data source from which the data combination originates. A notification is then pushed to the corresponding data source, which, in response to receiving that a data combination is being processed by the functional model, sends a new data combination to a scheduling queue.

[0140] In one embodiment, a data processing scheduling method is also provided, which includes a verification method for the functional model and the data source configuration, specifically as follows:

[0141] S701: Obtain configuration information for the data source and configuration information for the functional model.

[0142] In this embodiment, the configuration information of the data source includes the defined scheduling model member identifier, and the configuration information of the functional model includes the defined data source member identifier.

[0143] S702: Compare the configuration information with the preset configuration file to determine whether it meets the requirements of the preset configuration file.

[0144] In this embodiment, if the configuration information and the preset configuration file are as required by the preset configuration file, it is assumed that the data source and the functional model can be connected, and step S703 is executed. If the configuration information and the preset configuration file are not as required by the preset configuration file, it is assumed that the data source and the functional model are not yet connected, and at least one scheduling model member identifier is predefined in the data source again, and at least one data source member identifier is predefined in the functional model, and step S701 is repeated; or the process ends and feedback information is given to the user.

[0145] The default configuration file includes information on multiple data sources, multiple functional models, and the connection relationship between the data sources and functional models.

[0146] S703: The configuration of the judgment and verification function model and data source is complete.

[0147] In this embodiment, in response to the configuration information as required by the preset configuration file, it is determined that the verification function model and data source configuration are complete.

[0148] Those skilled in the art will understand that all or part of the processes in the methods of the above embodiments can be implemented by a computer program instructing related hardware. The computer program can be stored in a non-volatile computer-readable storage medium, and when executed, it can include the processes of the embodiments of the above methods. Any references to memory, storage, databases, or other media used in the embodiments provided in this application can include non-volatile and / or volatile memory. Non-volatile memory can include read-only memory (ROM), programmable ROM (PROM), electrically programmable ROM (EPROM), electrically erasable programmable ROM (EEPROM), or flash memory. Volatile memory can include random access memory (RAM) or external cache memory. By way of illustration and not limitation, RAM is available in various forms, such as static RAM (SRAM), dynamic RAM (DRAM), synchronous DRAM (SDRAM), dual data rate SDRAM (DDRSDRAM), enhanced SDRAM (ESDRAM), synchronous link DRAM (SLDRAM), Rambus direct RAM (RDRAM), direct memory bus dynamic RAM (DRDRAM), and memory bus dynamic RAM (RDRAM), etc.

[0149] The technical features of the above embodiments can be combined in any way. For the sake of brevity, not all possible combinations of the technical features in the above embodiments are described. However, as long as there is no contradiction in the combination of these technical features, they should be considered to be within the scope of this specification.

[0150] The embodiments described above are merely illustrative of several implementation methods of this application, and while the descriptions are relatively specific and detailed, they should not be construed as limiting the scope of the invention patent. It should be noted that those skilled in the art can make various modifications and improvements without departing from the concept of this application, and these all fall within the protection scope of this application. Therefore, the protection scope of this patent application should be determined by the appended claims.

Claims

1. A scheduling method of data processing, applied to a data processing system, characterized in that, The data processing system includes multiple functional models, multiple data sources, and multiple scheduling queues; each scheduling queue is connected to a functional model and at least one data source, and each data source is connected to at least one scheduling queue. The scheduling method includes: The data sources generate data combinations and send them to the scheduling queues they are connected to. The data combinations include data to be processed and a data source identifier. The scheduling queue receives at least one data combination, stores and retrieves it according to the scheduling rules, and sends the retrieved data combination to the functional model. The functional model receives the data combination, processes the data to be processed, and feeds back the processing result to the data source based on the data source identifier; At least one scheduling model member identifier is predefined within the data source, and at least one data source member identifier is predefined within the functional model; the data processing scheduling method further includes: Based on the verification of the functional model and the data source configuration, the data source and the functional model are connected through the scheduling queue using the scheduling model member identifier and the data source member identifier; The verification of the functional model and the data source configuration includes: Obtain the configuration information of the data source and the configuration information of the functional model, wherein the configuration information includes the defined scheduling model member identifier / data source member identifier; The configuration information is compared with the preset configuration file to determine whether it is as required by the preset configuration file. The preset configuration file includes multiple data source information, multiple functional model information, and connection relationship information between the data source and the functional model. In response to the configuration information as required by the preset configuration file, it is determined that the configuration of the functional model and the data source is complete.

2. The scheduling method as claimed in claim 1, characterized in that, The scheduling queue includes multiple cache units; The storage according to the scheduling rules includes: The scheduling queue stores each data combination in its respective cache unit in the order in which the data combinations are received. The number of cache units in each scheduling queue is matched with the number of data sources they are connected to.

3. The scheduling method as claimed in claim 2, characterized in that, The step of storing and retrieving data according to scheduling rules, and sending the retrieved data combination to the functional model, includes: In response to the functional model receiving the data to be processed, the functional model calls the data combination in the cache unit for processing; The step of feeding back the processing result to the data source based on the data source identifier also includes: The scheduling queue removes the data combination that has been processed and receives a new data combination, the new data combination and the data combination that has been processed originating from the same data source.

4. The scheduling method as claimed in claim 1, wherein, The data to be processed is obtained by copying the original data from the data source; and / or, the processing result is the original result after the functional model has completed the processing.

5. The scheduling method as claimed in claim 1, wherein, Each of the aforementioned scheduling queues receives at least one data combination, including: The scheduling queue uses an independent thread to parse and read the data combination for each of the data sources.

6. A data processing system implementing a scheduling method of data processing as claimed in any one of claims 1 to 5, characterized in that, The data processing system includes: multiple data sources, multiple functional models, and multiple scheduling queues; each scheduling queue is connected to a functional model and at least one data source, and each data source is connected to at least one scheduling queue. The data source is used to generate data combinations and send them to the scheduling queue connected to it. The data combination includes the data to be processed and the data source identifier. The scheduling queue is used to receive at least one data combination, store and call it according to the scheduling rules, and send the called data combination to the functional model. The functional model is used to receive the data combination, process the data to be processed, and feed back the processing result to the data source based on the data source identifier.

7. A computer device comprising a memory, a processor, and a computer program stored on the memory and executable on the processor, characterized in that, When the processor executes the computer program, it implements the steps of the data processing scheduling method according to any one of claims 1 to 5.

8. A computer-readable storage medium having stored thereon a computer program, characterized in that, When the computer program is executed by the processor, it implements the steps of the data processing scheduling method according to any one of claims 1 to 5.

Images