Data processing method and device, computer equipment and storage medium

Abstract

The embodiment of the invention provides a data processing method, a data processing device, computer equipment and a computer storage medium, and relates to the technical field of computers. The method comprises the following steps: in response to a received target configuration file of a target application program, performing configuration analysis on the target configuration file based on an analyzer to obtain scheduling parameter information; wherein the target configuration file at least comprises one or more to-be-processed first data streams of the target application program, the scheduling parameter information at least comprises a loop-free directed DAG graph corresponding to the first data streams, and an execution sequence of each first processing unit is configured in the DAG graph; and the control scheduler schedules the first processing units to process the one or more first data streams based on the execution sequence of the first processing units. The method reduces the workload of engineers so as to reduce the research and development cost, and then improves the operation efficiency of the whole project.

Description

Technical Field

[0001] The present application relates to the field of computer technology, and in particular, to a data processing method, a data processing apparatus, a computer device, and a computer storage medium. Background Art

[0002] In existing software development engineering practices, improving program performance through concurrent task scheduling has become a common trend. This technology enables multiple computing tasks to be executed simultaneously within overlapping time periods, thereby improving computing resource utilization and system throughput. However, in existing development practices, engineers specializing in domain-specific algorithms or those outside of software engineering fields not only need to develop their specific algorithms but also invest significant time in researching concurrent scheduling techniques to optimize program performance. This not only increases the engineers' workload and learning costs, but also leads to a waste of internal technical resources and low overall project development efficiency. Summary of the Invention

[0003] In the embodiments of the present application, a data processing method, a data processing method apparatus, a computer device, and a computer storage medium are provided, thereby overcoming, to at least a certain extent, the technical problems caused by the limitations and defects of related technologies, which not only require undertaking algorithm development tasks in specific fields, but also require investing a lot of time in researching concurrent scheduling technologies to optimize program performance, thereby increasing the workload and learning costs of engineers, and subsequently affecting the overall project development efficiency.

[0004] A first aspect of an embodiment of the present application provides a data processing method, which is applied to a server, wherein the server is configured with an application layer development framework, and the application layer development framework includes at least a parser and a scheduler. The method includes: in response to receiving a target configuration file of a target application, performing configuration parsing on the target configuration file based on the parser to obtain scheduling parameter information; wherein the target configuration file includes at least one or more first data streams to be processed of the target application, the scheduling parameter information includes at least a loop-free directed DAG graph corresponding to the first data stream, and the execution order of each first processing unit is configured in the DAG graph; and controlling the scheduler to schedule the first processing unit to process the one or more first data streams based on the execution order of each first processing unit.

[0005] A second aspect of an embodiment of the present application provides a data processing device, which is applied to a server, wherein the server is configured with an application layer development framework, and the application layer development framework includes at least a parser and a scheduler, and the device includes: a configuration parsing module, which is configured to execute a target configuration file in response to receiving a target application, and performs configuration parsing on the target configuration file based on the parser to obtain scheduling parameter information; wherein the target configuration file includes at least one or more first data streams to be processed of the target application, and the scheduling parameter information includes at least a loop-free directed DAG graph corresponding to the first data stream, and the execution order of each first processing unit is configured in the DAG graph; a data processing module, which is configured to execute a control scheduler based on the execution order of each first processing unit to schedule the first processing unit to process one or more first data streams.

[0006] According to a third aspect of an embodiment of the present application, a computer device is provided, including a memory and a processor, wherein the memory stores a computer program, and the processor implements the steps of any of the above data processing methods when executing the computer program.

[0007] According to a fourth aspect of the embodiments of the present application, a computer-readable storage medium is provided, on which a computer program is stored. When the computer program is executed by a processor, the steps of any of the above data processing methods are implemented.

[0008] According to a fifth aspect of the embodiments of the present application, a computer program product is provided, comprising a computer program, which is executed by a processor to implement the steps of any one of the above data processing methods.

[0009] The technical solution of this application has the following beneficial effects:

[0010] According to the data processing method, the server responds to receiving a target configuration file of a target application, and parses the target configuration file based on a parser to obtain scheduling parameter information; wherein the target configuration file contains at least one or more first data streams to be processed of the target application, and the scheduling parameter information contains at least a loop-free directed DAG graph corresponding to the first data stream, and the execution order of each first processing unit is configured in the DAG graph; the control scheduler schedules the first processing unit to process the one or more first data streams based on the execution order of each first processing unit. On the one hand, the method decomposes the complex logic code in the concurrent scheduling process into small processing units (PUs) to achieve modularization of the above logic code, thereby facilitating code reuse in different engineering scenarios, enhancing the maintainability and scalability of the code, thereby avoiding the need for developers to invest a lot of time in researching concurrent scheduling technology to optimize program performance in related technical solutions, thereby reducing the workload of developers. In addition, by decomposing the complex logic code into multiple processing units, the boundaries / interfaces between the processing units are clarified, thereby facilitating the friendliness of multi-person collaborative development application scenarios, thereby ensuring efficient division of labor and cooperation between team members, and significantly improving the efficiency and quality of software engineering development. On the other hand, the method uses a DAG graph to configure the execution order of each first processing unit, so that multiple first data streams are processed simultaneously through concurrent scheduling of each first processing unit, thereby providing flexible process control while improving data processing efficiency and system throughput. BRIEF DESCRIPTION OF THE DRAWINGS

[0011] The drawings described herein are used to provide a further understanding of the present application and constitute a part of the present application. The illustrative embodiments of the present application and their descriptions are used to explain the present application and do not constitute an improper limitation on the present application. In the drawings:

[0012] Figure 1 A schematic diagram of the architecture of a data processing system provided in one embodiment of the present application;

[0013] Figure 2 A flowchart of a data processing method provided in one embodiment of the present application;

[0014] Figure 3 This is a schematic rendering of a DAG graph provided in one embodiment of the present application;

[0015] Figure 4 A flowchart of a method for scheduling a first processing unit to process multiple first data streams provided in one embodiment of the present application;

[0016] Figure 5A schematic rendering of a DAG graph including a first processing unit and a second processing unit provided in one embodiment of the present application;

[0017] Figure 6 A schematic diagram of an overall data processing method provided in one embodiment of the present application;

[0018] Figure 7 A schematic diagram of the structure of a data processing device provided in one embodiment of the present application;

[0019] Figure 8 A schematic diagram of the structure of a computer device provided in one embodiment of the present application. DETAILED DESCRIPTION

[0020] Example embodiments will now be described more fully with reference to the accompanying drawings. However, example embodiments can be implemented in a variety of forms and should not be construed as limited to the examples set forth herein; rather, these embodiments are provided so that the present disclosure will be more comprehensive and complete and will fully convey the concepts of the example embodiments to those skilled in the art. The described features, structures, or characteristics may be combined in any suitable manner in one or more embodiments. In the following description, many specific details are provided to provide a full understanding of the embodiments of the present disclosure. However, those skilled in the art will appreciate that the technical solutions of the present disclosure may be practiced while omitting one or more of the specific details, or that other methods, components, devices, steps, etc. may be employed. In other cases, well-known technical solutions are not shown or described in detail to avoid obscuring various aspects of the present disclosure.

[0021] In addition, the accompanying drawings are merely schematic illustrations of the present disclosure and are not necessarily drawn to scale. Identical reference numerals in the figures denote identical or similar parts, and thus repetitive descriptions thereof will be omitted. Some of the block diagrams shown in the accompanying drawings are functional entities that do not necessarily correspond to physically or logically separate entities. These functional entities may be implemented in software, in one or more hardware modules or integrated circuits, or in different networks and / or processor devices and / or microcontroller devices.

[0022] The flowcharts shown in the accompanying drawings are merely illustrative and do not necessarily include all steps. For example, some steps may be decomposed, while some steps may be combined or partially combined, so the actual execution order may change according to actual circumstances.

[0023] In order to help those skilled in the art better understand the technical solution of the present disclosure, the relevant contents involved in the technical solution of the present disclosure are introduced below.

[0024] 1) Concurrency: In computer science, it refers to the ability of multiple computing tasks to occur or execute within the same time interval. It does not require that multiple tasks must be executed simultaneously, but rather emphasizes that multiple tasks can be executed within overlapping time periods at a macro level to achieve the technical effect of improving resource utilization and throughput.

[0025] 2) Scheduling: A core concept in computer science that involves the process of deciding which tasks should be executed and when and how.

[0026] 3) YAML (YAML Ain't Markup Language): is a format for data serialization that is easy for humans to read and write, and easy for machines to parse and generate.

[0027] 4) Process Unit (PU / PU Operator): A processing unit (PU) is the basic execution unit in the application layer development framework (also called the PU framework), responsible for processing specific business logic or computing tasks. Each PU operator encapsulates one or more operations, such as data processing, calculation, conversion, or any other form of business logic operation. PU operators are designed to improve the concurrent processing capabilities of software applications.

[0028] In the practical application of existing software development projects, concurrent scheduling, optimized data processing and other technologies are often used as one of the common technical means to improve program performance. Among them, concurrent scheduling uses a time-sharing method to process multiple computing tasks simultaneously, thereby improving the utilization of computing resources and system throughput. Mainstream concurrent scheduling technology mainly relies on the multi-threaded interface provided by each operating system. For program developers, the above method has certain technical barriers. It is difficult to flexibly customize the order of concurrent calls, thereby limiting the precise control of the program execution process. In data transmission tasks, if the data transmission mechanisms provided by different programming languages ​​​​(such as Python, C++, JAVA, assembly language, etc.) are improperly applied, it is very easy to cause additional performance loss.

[0029] In addition, in existing development practices, for program developers or non-software engineers who focus on algorithms in specific fields, they not only need to undertake algorithm development and business implementation tasks in specific fields, but also need to invest a lot of time in in-depth research on concurrent scheduling and university data transmission technologies to improve program running performance, thereby increasing the workload of program developers, resulting in repeated investment and waste of internal technical resources of the enterprise, and then affecting the R&D efficiency and progress of the overall project.

[0030] Therefore, there is an urgent need to provide a general application layer development framework that can support concurrent scheduling and achieve efficient data transmission.

[0031] In response to the above-mentioned technical problems existing in the relevant technical solutions, the embodiments of the present disclosure provide a data processing method. On the one hand, the method decomposes the complex logic code in the concurrent scheduling process into small processing units (ProcessUnit, PU) to realize the modularization of the above-mentioned logic code, thereby facilitating the reuse of the code in different engineering scenarios, enhancing the maintainability and scalability of the code, thereby avoiding the need for developers to invest a lot of time in researching concurrent scheduling technology to optimize program performance in the relevant technical solutions, thereby reducing the workload of developers. In addition, by decomposing the complex logic code into multiple processing units, the boundaries / interfaces between the various processing units are clarified, thereby facilitating the improvement of the friendliness of multi-person collaborative development application scenarios, thereby ensuring efficient division of labor and cooperation among team members, and significantly improving the efficiency and quality of software engineering development. On the other hand, the method uses a DAG graph to configure the execution order of each first processing unit, thereby simultaneously processing multiple first data streams through the concurrent scheduling of each first processing unit, thereby providing flexible process control while improving data processing efficiency and system throughput.

[0032] The following will be combined Figure 1 , a data processing method and device proposed in this disclosure is applied to Figure 1 The embodiments are described in the system architecture of the exemplary application environment shown.

[0033] like Figure 1 As shown, it is a schematic diagram of an application scenario provided by an embodiment of the present application, in which a terminal device 101 and a server 102 may be included.

[0034] The terminal device 101 can be, for example, a mobile phone, a tablet computer, a laptop computer, a desktop computer, a smart TV, a smart car device, a smart wearable device, a smart TV, an aircraft, or any other device involving data processing and data transmission. The terminal device 101 can be installed with a target application, and the target application can have functions such as generating one or more data streams to be processed and generating a target configuration file. The target application can be, for example, an instant messaging application, a music application, a game application, a video application, a short video application, a news application, and a shopping application, or any other type of application. The application involved in the embodiment of the present application can be a software client, or a client such as a web page or a small program. The server 102 is a server corresponding to the software or a web page, a small program, etc., and does not limit the specific type of the client.

[0035] The server 102 can be an independent physical server, or a server cluster or distributed system composed of multiple physical servers, or a cloud server that provides basic cloud computing services such as cloud services, cloud databases, cloud computing, cloud functions, cloud storage, network services, cloud communications, middleware services, domain name services, security services, i.e., content delivery networks (CDNs), and big data and artificial intelligence platforms, but is not limited thereto. The server 102 can be a backend server for the target application, used to provide corresponding backend services for it, such as hosting an application layer development framework to support concurrent scheduling and high-speed data transmission. The application layer development framework includes at least a parser and a scheduler. The server 102 can respond to receiving a target configuration file of the target application and parse the target configuration file based on the parser to obtain scheduling parameter information; wherein the target configuration file includes at least one or more first data streams to be processed of the target application, and the scheduling parameter information includes at least a directed DAG graph with no loop corresponding to the first data stream, and the execution order of each first processing unit is configured in the DAG graph; the control scheduler schedules the first processing unit to process the one or more first data streams based on the execution order of each first processing unit.

[0036] In the embodiment of the present application, the terminal device 101 and the server 102 can be directly or indirectly connected to each other through one or more networks 103. The network 103 can be a wired network or a wireless network. For example, the wireless network can be a mobile cellular network or a Wireless Fidelity (WIFI) network. Of course, it can also be other possible networks, which are not limited in the embodiment of the present application. It should be noted that Figure 1 The figures are only examples. In fact, the number of terminal devices and servers is not limited and is not specifically limited in the embodiments of this application.

[0037] It is easy for those skilled in the art to understand that the above application scenarios are only for example and are not limited to this exemplary embodiment.

[0038] After understanding the system architecture diagram of the data processing method provided by the present disclosure, the data processing method is applied to the above-mentioned server 102 as an example for explanation below. Figure 2 A flowchart schematically illustrating a data processing method in this exemplary embodiment is shown in FIG. Figure 2 The data processing method provided in the embodiment of the present application includes the following steps 201-202:

[0039] Step 201: In response to receiving a target configuration file of a target application, the target configuration file is configured and parsed based on a parser to obtain scheduling parameter information; wherein, the target configuration file at least includes one or more first data streams to be processed of the target application, and the scheduling parameter information at least includes a loop-free directed DAG graph corresponding to the first data stream, and the execution order of each first processing unit is configured in the DAG graph.

[0040] Step 202: The control scheduler schedules the first processing units to process one or more first data streams based on the execution order of the first processing units.

[0041] Through the above Figure 2 The data processing method shown is that the server responds to receiving a target configuration file of a target application, and parses the target configuration file based on a parser to obtain scheduling parameter information; wherein the target configuration file contains at least one or more first data streams to be processed of the target application, and the scheduling parameter information contains at least a loop-free directed DAG graph corresponding to the first data stream, and the execution order of each first processing unit is configured in the DAG graph; the control scheduler schedules the first processing unit to process the one or more first data streams based on the execution order of each first processing unit. On the one hand, the method decomposes the complex logic code in the concurrent scheduling process into small processing units (PUs) to achieve modularization of the above-mentioned logic code, thereby facilitating code reuse in different engineering scenarios, enhancing the maintainability and scalability of the code, thereby avoiding the need for developers to invest a lot of time in researching concurrent scheduling technology to optimize program performance in related technical solutions, thereby reducing the workload of developers. In addition, by decomposing the complex logic code into multiple processing units, the boundaries / interfaces between the processing units are clarified, thereby facilitating the friendliness of multi-person collaborative development application scenarios, thereby ensuring efficient division of labor and cooperation between team members, and significantly improving the efficiency and quality of software engineering development. On the other hand, the method uses a DAG graph to configure the execution order of each first processing unit, so that multiple first data streams are processed simultaneously through concurrent scheduling of each first processing unit, thereby providing flexible process control while improving data processing efficiency and system throughput.

[0042] The following will be combined with specific embodiments to Figure 2 The specific implementation of each step in the embodiment shown is described in detail:

[0043] In step 201, in response to receiving a target configuration file of a target application, the target configuration file is configured and parsed based on the parser to obtain scheduling parameter information; wherein, the target configuration file at least includes one or more first data streams to be processed of the target application, and the scheduling parameter information at least includes a loop-free directed DAG graph corresponding to the first data stream, and the execution order of each first processing unit is configured in the DAG graph.

[0044] The target application runs in the terminal device, and generates a target configuration file and sends it to the server. The target configuration file of the target application can at least define one or more first data streams to be processed.

[0045] It is understandable that the target configuration file can be a configuration file in any format. In an optional embodiment, the target configuration file can be a configuration file in YAML format. This is because YAML format is a format for data serialization, which is not only easy for program developers to read and write, but also easy for machines to parse and generate.

[0046] For example, an application layer development framework is configured in the server. Upon receiving a target configuration file, a parser in the application layer development framework can parse the target configuration file to obtain scheduling parameter information. For example, if the target configuration file is in YAML format, the parser can parse the YAML configuration file and store the parsed scheduling parameter information for subsequent use.

[0047] In this embodiment, a DAG graph can be extracted by parsing the configuration file. The DAG graph can be obtained by parsing the DAG graph configuration file provided by the user. The DAG graph defines the execution order of each first processing unit used in the process of processing multiple concurrent first data streams.

[0048] Figure 3 This is a schematic diagram of a DAG diagram provided in one embodiment of the present application, refer to Figure 3As shown, the DAG diagram defines the execution order of each first processing unit through the directed edges between the first processing units. For example, the downstream processing units of the first processing unit A and the first processing unit B are both the first processing unit D, and the downstream processing unit of the first processing unit C is the first processing unit E, which is used to represent that after the first data stream 1 is processed by calling the first processing unit A, it is processed by the first processing unit D. Similarly, after the first data stream 2 is processed by calling the first processing unit B, it is processed by the first processing unit D, and after the first data stream 3 is processed by calling the first processing unit C, it is processed by the first processing unit E. It can be seen that by using DAG (directed acyclic graph) to configure the execution order of the first processing units, program developers can flexibly control the concurrent call process, and ensure efficient concurrent processing between multiple data streams, thereby ensuring the sequentiality and integrity of data stream processing. It should be understood that Figure 3 The number of first processing units and the directed edges between the first processing units shown are merely exemplary and non-limiting, and more or less belong to the embodiments of the present application.

[0049] It should be explained that a DAG is a directed acyclic graph, which means a directed graph without loops. In addition to meeting the execution order specified by the directed edges, a DAG must also be acyclic: starting from any vertex, no matter the order in which the directed edges are executed, the graph will not return to the starting vertex. For example Figure 3 contains three starting vertices, namely the first processing unit A, the first processing unit B, and the first processing unit C. Then, during the execution process, the first processing unit A, the first processing unit B, and the first processing unit C will no longer be used to perform subsequent processing of the first data stream.

[0050] Furthermore, since the DAG graph is parsed from the DAG graph configuration file provided by the user / program developer, before using the execution order of each first processing unit configured in the DAG graph to perform concurrent scheduling processing of multiple data streams, the DAG graph needs to be verified for legitimacy to verify whether it meets the basic requirements of a directed acyclic graph, namely directed edges and no cycles.

[0051] In an optional embodiment of the present application, the DAG graph is verified for legitimacy in advance; if there is a single directed edge between each first processing unit in the DAG graph and there is no loop between each first processing unit, then based on the execution order of each first processing unit configured in the DAG graph, the first processing unit is scheduled by the scheduler to process one or more first data streams.

[0052] For example, when the DAG graph is checked to have unidirectional directed edges and there are no loops between the first processing units, it can be determined that the DAG graph has passed the legality verification, and the execution order of the first processing units specified by the DAG graph can be used to control the caller to call the first processing units to be used in sequence to perform corresponding data processing.

[0053] In this embodiment, pre-validation of the DAG graph is an important step in ensuring the correctness of the DAG graph and the stability of the system. It eliminates possible circular dependencies and ensures the feasibility of topological sorting of the graph, thereby providing a solid foundation for the subsequent scheduling and execution of the first processing unit.

[0054] In one embodiment, in the process of implementing concurrent scheduling of multiple data streams, logical processing is also required based on a pre-configured scheduling policy to ensure the order and accuracy of the concurrent scheduling process.

[0055] In an optional embodiment of the present application, the scheduling parameter information includes at least a scheduling strategy, and the scheduling strategy includes a first scheduling strategy or a second scheduling strategy. When the server schedules the first data stream based on the first processing unit, in response to receiving one or more second data streams to be processed of the target application, the server may save the second data stream according to the first scheduling strategy until the scheduling processing process for the first data stream is completed to start the scheduling processing process for the second data stream; or, in response to receiving the second data stream, the server may discard the second data stream according to the second scheduling strategy.

[0056] For example, by parsing the target configuration file, the scheduling strategy (also called the discard strategy) adopted by the scheduler can also be extracted, so as to indicate through the scheduling strategy the processing strategy for restarting a new round of processing units to process the second data stream during the processing of the first data stream in the application layer development framework.

[0057] In this embodiment, one of the optional discarding strategies provided is the first scheduling strategy (which may be called the ONLY_NEW strategy): that is, when scheduling the first data stream based on the first processing unit, if in response to receiving one or more second data streams to be processed from the target application, the system will only retain the latest task scheduling, and the processing task being executed for the first data stream will continue without being affected by the new task.

[0058] Another optional discarding strategy is the second scheduling strategy (which can be called the ABANDON strategy): under this strategy, when the first processing unit schedules the first data stream, if in response to receiving one or more second data streams to be processed from the target application, the system chooses to abandon the newly arrived task scheduling.

[0059] It should be explained that the first scheduling strategy or the second scheduling strategy included in the above scheduling strategies are merely exemplary, and other scheduling strategies may also be used. This disclosure again does not impose any limitation or exhaustive list.

[0060] In step 202 , the control scheduler schedules the first processing units to process one or more first data streams based on the execution order of the first processing units.

[0061] Exemplarily, after determining the execution order of each first processing unit based on the DAG graph, the scheduler can be controlled to schedule the first processing unit to process one or more first data streams.

[0062] In the process of using the execution order of each first processing unit in the DAG graph to schedule the corresponding first processing unit to process one or more first data streams, in order to ensure efficient processing of multiple concurrent data streams, processing can be performed based on the following embodiments.

[0063] In an optional embodiment of the present application, the first data stream includes at least a first sub-data stream and a second sub-data stream. When executing the above step 202 to control the scheduler to schedule the first processing unit to process multiple first data streams based on the execution order of each first processing unit, see Figure 4 The method for scheduling the first processing unit to process the first data stream provided in the embodiment of the present application includes the following steps 401 to 404:

[0064] Step 401: The control scheduler schedules a first sub-processing unit to process a first sub-data stream, and calls a second sub-processing unit to process a second sub-data stream.

[0065] The first sub-processing unit and the second sub-processing unit are the same sub-processing unit or different sub-processing units.

[0066] Step 402: Determine whether the next processing unit of the first sub-processing unit is the same as the next processing unit of the second sub-processing unit.

[0067] If the next-order processing unit of the first sub-processing unit is the same as the next-order processing unit of the second sub-processing unit and both are first target processing units, execute step 403 and control the scheduler to continue scheduling the first target processing unit upon receiving an instruction from the first sub-processing unit to complete processing the first sub-data stream and an instruction from the second sub-processing unit to complete processing the second sub-data stream.

[0068] On the contrary, the next-order processing unit of the first sub-processing unit is different from the next-order processing unit of the second sub-processing unit, that is, the next-order processing unit of the first sub-processing unit is the second target processing unit, and the next-order processing unit of the second sub-processing unit is the third target processing unit, then step 404 is executed, the control scheduler continues to call the second target processing unit when receiving the processing completion instruction of the first sub-processing unit on the first sub-data stream, and the control scheduler continues to call the third target processing unit when receiving the processing completion instruction of the second sub-processing unit on the second sub-data stream.

[0069] The second target processing unit is different from the third target processing unit. The first sub-processing unit and the second sub-processing unit can be the same processing unit or different processing units. Optionally, in order to improve data processing efficiency, different data streams can be assigned to different first sub-processing units for processing.

[0070] Depend on Figure 4 As can be seen from the illustrated embodiment, the execution of each sub-processing unit in the first processing unit is based on the completion status of one or more previous sub-processing units. This dependency relationship is clearly displayed in the DAG diagram.

[0071] It should be noted that the first data stream shown in the embodiment of the present application includes at least the first sub-data stream and the second sub-data stream. In other words, the first sub-data stream and the second sub-data stream are used to represent the data stream processed by the first sub-processing unit and the second sub-processing unit in the DAG graph, respectively. The number of sub-data streams included in the first data stream can be any number of two or more, and each sub-data stream can also be a single data stream or a collection of multiple data streams.

[0072] Accordingly, the number of the first sub-processing unit and the second sub-processing unit in this embodiment is only exemplary, and the embodiment may also include any number of sub-processing units. Figure 3 As shown, three sub-processing units A, B, and C are used to process the first data streams 1, 2, and 3 respectively. In this case, this embodiment includes three sub-processing units. Specifically, sub-processing unit A can correspond to the first sub-processing unit shown in the above embodiment, sub-processing unit B can correspond to the second sub-processing unit shown in the above embodiment, and sub-processing unit C can be the third sub-processing unit. If more than three sub-processing units are determined to be concurrent based on the DAG graph, they can be called the fourth sub-processing unit, ..., the Nth sub-processing unit (N>4) in sequence, which can be determined specifically according to the DAG graph.

[0073] For ease of understanding, the following Figure 3 The DAG diagram shown is used as an example for explanation.

[0074] Reference Figure 3 As shown, it is assumed that the first processing unit is determined to include sub-processing units A, B, C, D, E, and F based on the DAG graph (the following embodiments may also be referred to as first processing units A, B, C, D, E, and F). When the scheduling processing flow is started based on the concurrently executed sub-processing units A, B, and C (which may also be referred to as the first sub-processing unit, the second sub-processing unit, and the third sub-processing unit), the first data streams 1, 2, and 3 are processed respectively. This parallel processing mechanism significantly improves the data processing efficiency and system throughput. Accordingly, sub-processing units A, B, and C can obtain any data stream for data processing. For example, sub-processing unit A obtains the first data streams 1 and 2 for processing. At this time, the first sub-data stream contains the first data streams 1 and 2; or sub-processing unit A obtains the first data stream 2 or 3 for processing. At this time, the first sub-data stream contains the first data stream 2 or 3. In other words, each processing unit can obtain any number and type of data streams for data processing, and the embodiments of the present application do not impose any special restrictions on this.

[0075] After sub-processing units A and B complete, the system automatically calls the next sub-processing unit, D. This sequential execution ensures the logic and consistency of data processing. Similarly, similar to the call to sub-processing unit D, after sub-processing unit C completes, sub-processing unit E will be called. Finally, after both sub-processing units D and E complete, sub-processing unit F will be called, completing the entire data processing flow.

[0076] In an optional embodiment of the present application, the scheduling parameter information may further include the data stream name and data stream type associated with each first data stream, so that when executing the above step 401, the corresponding first processing unit can be matched according to the data stream name and data stream type, that is, the first processing unit matching the data stream type can be allocated to the first data stream according to the data stream name and data stream type. Figure 3 The first data stream 1 matches the data stream type of the first processing unit A. For example, the first processing unit A processes the data stream type of the first data stream 1 faster or consumes the least resources.

[0077] Furthermore, in an optional embodiment of the present application, in addition to the first processing unit, the DAG graph can also configure a second processing unit, and load the second processing unit into the system memory based on the Linux dynamic loading mechanism. This process not only optimizes resource utilization, but also improves the response speed of the system.

[0078] Among them, the second processing unit can be a special first processing unit, which is used to initialize a new round of scheduling cycle. It is a preset processing unit in the application layer development framework, and can automatically start a new round of scheduling cycle without the need for program developers to call the multi-threaded interface.

[0079] For example, the first and second processing units can be loaded during the initialization phase of the application layer development framework. After the second processing unit (RoundLeader) is successfully loaded, the application layer development framework will automatically perform initialization operations. During the initialization phase, RoundLeader will configure the necessary input data streams and set specific operations for reading the data streams. In addition, when loading the first processing unit, Linux's dynamic loading function can also be used to load the first processing unit into memory and execute the initialization operations of the first processing unit.

[0080] In an optional embodiment, during the operation phase of the above-mentioned application layer development framework, the above-mentioned step 202 is executed to control the scheduler to schedule the first processing unit to process one or more first data streams based on the execution order of each first processing unit. The scheduler can be controlled to schedule the second processing unit to start the scheduling processing flow for the first data stream, so as to automatically start a new round of scheduling cycle for the first data stream.

[0081] Figure 5 A schematic effect diagram of a DAG diagram including a first processing unit and a second processing unit provided in one embodiment of the present application; Figure 5 As shown, the second processing unit is scheduled first to automatically start the scheduling process for the first data stream without the participation of the program developer, and then the scheduler is controlled based on the execution order of each first processing unit to schedule the first processing unit to process one or more first data streams.

[0082] In this embodiment, the second processing unit (RoundLeader) ensures the timeliness and accuracy of the scheduled tasks, while the concurrent execution mechanism maximizes the efficiency of resource utilization.

[0083] In another optional embodiment of the present application, weight parameters can also be configured for each data stream, that is, the scheduling parameter information includes a first weight value for the first sub-data stream and a second weight value for the second sub-data stream, so that the second processing unit (RoundLeader) can be controlled to determine whether to start a new round of scheduling cycle for the first data stream based on the superposition of the weights of each sub-data stream in the first data stream.

[0084] Exemplarily, the sum of the first weight value of the first sub-data stream and the second weight value of the second sub-data stream is determined to obtain the target weight of the first data stream; if the target weight of the first data stream is greater than or equal to the preset weight value, the second processing unit is scheduled to start the scheduling processing process for the first data stream.

[0085] The preset weight value may be 100%, and the weight value of each sub-data stream may be a number less than 100%. Of course, the weight value may also be a specific numerical value, and the embodiment of the present application does not impose any special limitation on this.

[0086] Exemplarily, when the first weight value of the first sub-data stream and the second weight value of the second sub-data stream are detected, the first weight value of the first sub-data stream and the second weight value of the second sub-data stream can be superimposed to obtain the total weight of the current first data stream. If the total weight of the first data stream is greater than or equal to the preset weight value, the second processing unit is scheduled to start the scheduling processing flow for the first data stream. Conversely, if the total weight of the first data stream is less than the preset weight value, the data stream can continue to be waited for and received until the weight value of the received data stream is greater than or equal to the preset weight value, and then the second processing unit can be scheduled to start the scheduling processing flow for the above data stream.

[0087] In this embodiment, by superimposing the weight values ​​of multiple sub-data streams as a factor in determining whether to start a processing flow for the data stream, it is possible to avoid wasting resources and affecting performance caused by starting a new processing flow as soon as a data stream is received, thereby reducing resource consumption and improving system performance.

[0088] In an optional embodiment of the present application, each first processing unit is connected to a data bus component; wherein the data bus is configured in the application layer development framework based on a static memory allocation strategy so that the size and storage location of the data bus are predetermined.

[0089] For example, in this application layer development framework, to provide efficient data transmission capabilities for each first processing unit, a DataBus component can be configured within the application layer development framework. The DataBus component provides an efficient intra-process data transmission mechanism, making data exchange between different first processing units faster and more reliable, thereby improving the performance of the overall application layer development framework. Furthermore, the DataBus component provides an interface to program developers, facilitating data transmission when developing different first processing units.

[0090] Exemplarily, the design of the DataBus component is based on a static memory allocation strategy, which can predetermine the size and storage location of the data bus DataBus during the initialization phase, thereby avoiding the overhead of determining memory allocation during program execution to further improve data transmission efficiency.

[0091] In an optional embodiment, the DataBus component supports lock-free operation, which is achieved by internally using the ARM / x86 instruction set, greatly improving data access efficiency. Lock-free operation also reduces the performance bottlenecks caused by traditional locking mechanisms and enhances data processing capabilities in a multi-threaded environment.

[0092] In an optional embodiment, the DataBus component supports zero-copy functionality by directly returning a pointer variable to the memory area. When retrieving data from a data stream through the DataBus component, the developer obtains a pointer to the data memory block, rather than a copy of the data. This design reduces data copying overhead, improves data transmission efficiency, and reduces the burden on the CPU.

[0093] The following will refer to Figure 6 The entire data processing process of the data processing method according to the exemplary embodiment of the present disclosure is described in detail.

[0094] Reference Figure 6 As shown, the terminal device configures a user-level custom configuration, which may include a target configuration file in YAML format, a first processing unit (containing specific logic code), and other configuration files required for executing the first processing unit. These configuration files provide the necessary parameters and environment settings for the operation of the framework. The target configuration file contains one or more first data streams to be processed.

[0095] The following mainly uses multiple first data streams to be processed as an example to illustrate that in the application layer development framework, after receiving a target configuration file for a target application, a configuration parser is used to perform configuration parsing operations on the target configuration file to obtain scheduling parameter information.

[0096] Among them, the scheduling parameter information may include a DAG graph, data flow information, a discarding strategy, and the specific storage location of the first processing unit. The PU loader can read the corresponding first processing unit according to the specific storage location of the first processing unit provided by the parser and load it into the computer memory to prepare for subsequent execution; the execution order between the first processing units can be determined through the DAG graph; through the data flow information, each first data flow can be associated with a weight value, a data flow name, a data flow type and other key information, and stored for subsequent use. This process ensures that the application layer development framework can correctly identify and process various data flows according to the target configuration file, providing a basis for the efficient operation of the entire application layer development framework.

[0097] Finally, during the operation phase of the application layer development framework, the scheduler will schedule each first processing unit based on the above-mentioned scheduling parameter information, and through the above-mentioned discard strategy, when a new second data stream is received during the operation of the application layer development framework, the corresponding ONLY_NEW discard strategy will be adopted to retain the latest scheduling task for the second data stream, or the ABANDON discard strategy will be adopted to discard the latest scheduling task for the second data stream.

[0098] Providing efficient data transmission capabilities for each first processing unit, and providing efficient data transmission within the process through the DataBus component makes data exchange between different PU operators faster and more reliable, thereby improving the performance of the entire framework.

[0099] It should be understood that, although the various steps in the flowchart are shown in sequence as indicated by the arrows, these steps are not necessarily performed in the order indicated by the arrows. Unless otherwise specified herein, there is no strict order restriction on the execution of these steps, and these steps may be performed in other orders. Moreover, at least a portion of the steps in the figure may include multiple sub-steps or multiple stages, and these sub-steps or stages are not necessarily performed at the same time, but may be performed at different times. The execution order of these sub-steps or stages is not necessarily to be performed in sequence, but may be performed in turn or alternately with other steps or at least a portion of the sub-steps or stages of other steps.

[0100] To implement the above data processing method, see Figure 7 An embodiment of the present application provides a data processing device, which is applied to a server. The server configures an application layer development framework, and the application layer development framework includes at least a parser and a scheduler, which may include: a configuration parsing module 701 and a data processing module 702.

[0101] The configuration parsing module 701 is configured to execute, in response to receiving a target configuration file of a target application, configuration parsing of the target configuration file based on a parser to obtain scheduling parameter information; wherein the target configuration file includes at least one or more first data streams to be processed of the target application, and the scheduling parameter information includes at least a loop-free directed DAG graph corresponding to the first data stream, and the execution order of each first processing unit is configured in the DAG graph;

[0102] The data processing module 702 is configured to execute a control scheduler to schedule the first processing units to process one or more first data streams based on the execution order of the first processing units.

[0103] In an optional embodiment of the present disclosure, the first data stream includes at least a first sub-data stream and a second sub-data stream, and the data processing module 702 can be specifically configured to execute the control scheduler to schedule the first sub-processing unit to process the first sub-data stream, and call the second sub-processing unit to process the second sub-data stream; wherein the first sub-processing unit and the second sub-processing unit are the same sub-processing unit or different sub-processing units; if the next-order processing unit of the first sub-processing unit and the next-order processing unit of the second sub-processing unit are both the first target processing unit, then the control scheduler receives the processing completion instruction of the first sub-processing unit on the first sub-data stream. The control scheduler controls the scheduler to continue to call the second target processing unit upon receiving the instruction from the second sub-processing unit that the processing of the second sub-data stream is completed, and the control scheduler controls the scheduler to continue to call the third target processing unit upon receiving the instruction from the second sub-processing unit that the processing of the second sub-data stream is completed, wherein the second target processing unit is different from the third target processing unit.

[0104] In an optional embodiment of the present disclosure, at least a second processing unit is configured in the DAG graph, and the device may further include a loading module, which is used to load the second processing unit into the system memory based on the Linux dynamic loading mechanism; the processing unit scheduling module can also be used to control the scheduler to schedule the second processing unit to start the scheduling processing flow for the first data stream.

[0105] In an optional embodiment of the present disclosure, the scheduling parameter information includes a first weight value of the first sub-data stream and a second weight value of the second sub-data stream. The processing unit scheduling module is specifically used to determine the sum of the first weight value of the first sub-data stream and the second weight value of the second sub-data stream to obtain the target weight of the first data stream; if the target weight of the first data stream is greater than the preset weight value, the second processing unit is scheduled to start the scheduling processing flow for the first data stream.

[0106] In an optional embodiment of the present disclosure, the device may also include a verification module, which is used to pre-verify the legitimacy of the DAG graph; the data processing module 702 is configured to execute if a single directed edge is satisfied between each first processing unit in the DAG graph and there is no loop between each first processing unit, then based on the execution order of each first processing unit configured in the DAG graph, the first processing unit is scheduled by the scheduler to process one or more first data streams.

[0107] In an optional embodiment of the present disclosure, the scheduling parameter information includes at least a scheduling strategy, and the scheduling strategy includes a first scheduling strategy or a second scheduling strategy. The device further includes a data stream discarding module or a data stream saving module. The data stream saving module is used to, when scheduling the first data stream based on the first processing unit, respond to receiving one or more second data streams to be processed of the target application, save the second data stream according to the first scheduling strategy until the scheduling processing process of the first data stream is completed to start the scheduling processing process of the second data stream; or, the data stream discarding module is used to respond to receiving the second data stream and discard the second data stream according to the second scheduling strategy.

[0108] In an optional embodiment of the present disclosure, the device may further include a connection module, which is used to connect each first processing unit to a data bus component; wherein the data bus is configured in the application layer development framework based on a static memory allocation strategy so that the size and storage location of the data bus are predetermined.

[0109] For the specific definition of the above-mentioned data processing device, please refer to the definition of the data processing method above and will not be repeated here. Each module in the above-mentioned data processing device can be implemented in whole or in part by software, hardware, or a combination thereof. Each of the above-mentioned modules can be embedded in or independent of the processor in the computer device in the form of hardware, or can be stored in the memory of the computer device in the form of software so that the processor can call and execute the operations corresponding to each of the above modules.

[0110] In one embodiment, a computer device is provided. The internal structure diagram of the computer device can be as follows: Figure 8 As shown. The computer device includes a processor, a memory, a network interface and a database connected via a system bus. The processor of the computer device is used to provide computing and control capabilities. The memory of the computer device includes a non-volatile storage medium and an internal memory. The non-volatile storage medium stores an operating system, a computer program and a database. The internal memory provides an environment for the operation of the operating system and the computer program in the non-volatile storage medium. The database of the computer device is used to store data. The network interface of the computer device is used to communicate with an external terminal via a network connection. When the computer program is executed by the processor, a data processing method as described above is implemented. It includes: a memory and a processor, the memory stores a computer program, and when the processor executes the computer program, any step in the data processing method as described above is implemented.

[0111] In one embodiment, a computer-readable storage medium is provided, on which a computer program is stored. When the computer program is executed by a processor, any step in the above data processing method can be implemented.

[0112] Those skilled in the art will appreciate that the embodiments of the present application can be provided as methods, systems, or computer program products. Therefore, the present application can adopt the form of a complete hardware embodiment, a complete software embodiment, or an embodiment in combination with software and hardware. Moreover, the present application can adopt the form of a computer program product implemented on one or more computer-usable storage media (including but not limited to magnetic disk storage, CD-ROM, optical storage, etc.) that contain computer-usable program code.

[0113] The present application is described with reference to the flowcharts and / or block diagrams of the methods, devices (systems), and computer program products according to the embodiments of the present application. It should be understood that each process and / or box in the flowchart and / or block diagram, as well as the combination of the processes and / or boxes in the flowchart and / or block diagram, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, a special-purpose computer, an embedded processor, or other programmable data processing device to produce a machine, so that the instructions executed by the processor of the computer or other programmable data processing device generate instructions for implementing the steps in the process. Figure 1 a process or multiple processes and / or boxes Figure 1 A device that provides the functions specified in a block or multiple blocks.

[0114] These computer program instructions may also be stored in a computer readable memory that can direct a computer or other programmable data processing device to work in a specific manner, so that the instructions stored in the computer readable memory produce an article of manufacture comprising an instruction device, which implements the process Figure 1 a process or multiple processes and / or boxes Figure 1 The function specified in one or more boxes.

[0115] These computer program instructions can also be loaded onto a computer or other programmable data processing device so that a series of operational steps are executed on the computer or other programmable device to produce a computer-implemented process, thereby providing the instructions executed on the computer or other programmable device for implementing the process. Figure 1 a process or multiple processes and / or boxes Figure 1 A step that specifies a function in one or more boxes.

[0116] Although the preferred embodiments of the present application have been described, those skilled in the art may make additional changes and modifications to these embodiments once they have learned the basic creative concept. Therefore, the appended claims are intended to be interpreted as including the preferred embodiments and all changes and modifications that fall within the scope of the present application.

[0117] Obviously, those skilled in the art may make various changes and modifications to this application without departing from the spirit and scope of this application. Thus, if these modifications and variations of this application fall within the scope of the claims of this application and their equivalents, this application is intended to include these modifications and variations.

Claims

1. A data processing method, characterized in that: Applied to a server, the server is configured with an application layer development framework, and the application layer development framework includes at least a parser and a scheduler, the method includes: In response to receiving a target configuration file of a target application, performing configuration parsing on the target configuration file based on the parser to obtain scheduling parameter information; wherein the target configuration file at least includes one or more first data streams to be processed of the target application, and the scheduling parameter information at least includes a loop-free directed DAG graph corresponding to the first data stream, and the execution order of each first processing unit is configured in the DAG graph; The scheduler is controlled to schedule the first processing units to process one or more first data streams based on the execution order of the first processing units.

2. The method according to claim 1, characterized in that The first data stream includes at least a first sub-data stream and a second sub-data stream, and the controlling the scheduler to schedule the first processing unit to process the plurality of first data streams based on the execution order of the first processing units includes: Controlling the scheduler to schedule a first sub-processing unit to process the first sub-data stream, and calling a second sub-processing unit to process the second sub-data stream; wherein the first sub-processing unit and the second sub-processing unit are the same sub-processing unit or different sub-processing units; If the processing unit next in sequence to the first sub-processing unit and the processing unit next in sequence to the second sub-processing unit are both first target processing units, controlling the scheduler to continue scheduling the first target processing unit upon receiving an instruction from the first sub-processing unit that processing of the first sub-data stream is completed and receiving an instruction from the second sub-processing unit that processing of the second sub-data stream is completed; Alternatively, if the next-order processing unit of the first sub-processing unit is the second target processing unit, and the next-order processing unit of the second sub-processing unit is the third target processing unit, then the scheduler is controlled to continue calling the second target processing unit upon receiving the processing completion instruction of the first sub-data stream from the first sub-processing unit, and the scheduler is controlled to continue calling the third target processing unit upon receiving the processing completion instruction of the second sub-processing unit for the second sub-data stream; wherein the second target processing unit is different from the third target processing unit.

3. The method according to claim 1, characterized in that At least a second processing unit is further configured in the DAG graph, and the method further includes: Loading the second processing unit into the system memory based on the Linux dynamic loading mechanism; Before controlling the scheduler to schedule the first processing units to process one or more first data streams based on the execution order of the first processing units, the method further includes: The scheduler is controlled to schedule the second processing unit to start a scheduling processing flow for the first data flow.

4. The method according to claim 3, characterized in that The first data stream includes at least a first sub-data stream and a second sub-data stream, the scheduling parameter information includes a first weight value of the first sub-data stream and a second weight value of the second sub-data stream, and the controlling the scheduler to schedule the second processing unit to start a scheduling processing flow for the first data stream includes: Determine a sum of a first weight value of the first sub-data stream and a second weight value of the second sub-data stream to obtain a target weight of the first data stream; If the target weight of the first data flow is greater than a preset weight value, the second processing unit is scheduled to start a scheduling processing flow for the first data flow.

5. The method according to claim 1 or 2, characterized in that The method further comprises: Preliminary verification of the legitimacy of the DAG graph; If a single directed edge is satisfied between each first processing unit in the DAG graph and there is no loop between each first processing unit, then based on the execution order of each first processing unit configured in the DAG graph, the first processing unit is scheduled by the scheduler to process one or more first data streams.

6. The method according to claim 1, characterized in that The scheduling parameter information includes at least a scheduling strategy, and the scheduling strategy includes a first scheduling strategy or a second scheduling strategy. The method further includes: When scheduling the first data stream based on the first processing unit, in response to receiving one or more second data streams to be processed of the target application, the second data stream is saved according to the first scheduling policy until the scheduling processing process for the first data stream is completed to start the scheduling processing process for the second data stream; or, in response to receiving the second data stream, the second data stream is discarded according to the second scheduling policy.

7. The method according to claim 1, characterized in that The method further comprises: connecting each of the first processing units to a data bus component; The data bus is configured in the application layer development framework based on a static memory allocation strategy, so that the size and storage location of the data bus are predetermined.

8. A data processing device, characterized in that: Applied to a server, the server is configured with an application layer development framework, and the application layer development framework includes at least a parser and a scheduler, and the device includes: a configuration parsing module configured to execute, in response to receiving a target configuration file of a target application, performing configuration parsing on the target configuration file based on the parser to obtain scheduling parameter information; wherein the target configuration file includes at least one or more first data streams to be processed of the target application, and the scheduling parameter information includes at least a loop-free directed DAG graph corresponding to the first data stream, and the execution order of each first processing unit is configured in the DAG graph; The data processing module is configured to execute and control the scheduler to schedule the first processing units to process one or more first data streams based on the execution order of the first processing units.

9. A computer device comprising: The method comprises a memory and a processor, wherein the memory stores a computer program, and is characterized in that the processor implements the steps of the data processing method according to any one of claims 1 to 7 when executing the computer program.

10. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, the steps of the data processing method according to any one of claims 1 to 7 are implemented.

Images