Data processing method and device, electronic equipment and computer storage medium

Abstract

The invention relates to a data processing method and device, electronic equipment and a computer storage medium, and the method comprises the steps: obtaining a processing request for to-be-processed data, splitting the to-be-processed data into at least one task package according to a parameter type identifier of the to-be-processed data and a business processing flow corresponding to the processing request, according to the type identifier of each task package, determining a target algorithm plug-in corresponding to each task package in each processing stage, and for each processing stage, based on the state information corresponding to each task package corresponding to the processing stage, generating a calling sequence of the target algorithm plug-in corresponding to each task package corresponding to the processing stage; and calling a target algorithm plug-in corresponding to each task package corresponding to the processing stage to perform corresponding processing until processing data corresponding to each task package in each processing stage is obtained. By means of the method, direct access and processing of the data to be processed in the service framework are achieved through IO interface abstraction and algorithm plug-in design.

Description

Technical Field

[0001] The present invention relates to the field of data processing technology, and in particular to a data processing method, device, electronic equipment and computer storage medium. Background Art

[0002] With the development of computer technology, more and more business platforms in various industries are migrating to computer systems, relying on the powerful data storage and computing analysis capabilities of computer systems to achieve automated and intelligent data processing. As a computer software system, in addition to meeting functional requirements and performance requirements, stability, scalability, and maintainability are important indicators that need to be fully considered in the early stages of system design, especially in the face of scenarios where system requirements need to keep pace with changing business, and the software framework is bound to evolve over a long period of time. Therefore, a data processing software business framework that adapts to multiple types of inputs is an important foundation for achieving flexible migration and expansion of business platforms.

[0003] In the prior art, the data processing software framework is usually divided into processes such as data management, data processing and analysis, output organization and presentation, and expert-assisted decision-making. When faced with the need to process large amounts of data with multiple modes and complex characteristics, this solution uses a hard-coded approach to parse, store, and process the input of each mode one-to-one, resulting in poor platform consistency and difficulty in expansion. When adding new data types and processing algorithms, the entire platform needs to be reconstructed, delaying the production cycle and reducing the robustness of the platform. Therefore, the application of this solution is greatly limited in actual applications. Summary of the invention

[0004] The technical problem to be solved by the present invention is to provide a data processing method, device, electronic device and computer storage medium, aiming to solve at least one of the above technical problems.

[0005] In the first aspect, the technical solution of the present invention to solve the above technical problem is as follows: a data processing method, the method comprising:

[0006] Obtaining a processing request for the data to be processed, wherein the processing request includes a parameter type identifier of the data to be processed;

[0007] According to the parameter type identifier of the data to be processed and the business processing flow corresponding to the processing request, the data to be processed is split into at least one task package, each of the task packages includes a type identifier and status information, each of the status information is used to characterize the processing order of the corresponding task package in each processing stage, and the business processing flow corresponds to at least one processing stage;

[0008] According to the type identifier of each task package, determine the target algorithm plug-in corresponding to each task package in each processing stage, each target algorithm plug-in corresponds to a processing algorithm, and each target algorithm plug-in is a virtual plug-in formed by abstracting the corresponding processing algorithm through an IO interface;

[0009] For each processing stage, based on the status information corresponding to each task package corresponding to the processing stage, a calling sequence of the target algorithm plug-in corresponding to each task package corresponding to the processing stage is generated;

[0010] For each processing stage, the target algorithm plug-ins corresponding to each task package in the processing stage are called in the order of calling the target algorithm plug-ins corresponding to each task package in the processing stage to perform corresponding processing until the processing data corresponding to each task package in each processing stage is obtained.

[0011] The beneficial effects of the present invention are as follows: through IO interface abstraction and algorithm plug-in design, direct access and processing of the data to be processed in the business framework are realized without the need for modification at the platform layer; through the use of the algorithm engine and the determination of the task package, the process of the calling sequence of the target algorithm plug-ins corresponding to each task package corresponding to each processing stage is realized, and on this basis, through the calling sequence of the target algorithm plug-ins corresponding to each task package corresponding to the processing stage, the processing of the businesses corresponding to different task packages is realized without the need for modification at the platform layer.

[0012] Based on the above technical solution, the present invention can also be improved as follows.

[0013] Furthermore, the method further comprises:

[0014] According to a preset data regularization method, the target data is integrated to obtain integrated data corresponding to the target data, wherein the target data includes the data to be processed and / or the processed data corresponding to each of the task packages, and the integrated data includes the data type identifier, source, parsing method, storage method, reading rule, reading location and algorithm plug-in of the target data;

[0015] The integrated data is stored in the business platform.

[0016] Furthermore, the method further comprises:

[0017] For each of the task packets, according to the first target data type corresponding to the preset input adaptation interface, the data type corresponding to the task packet is converted into the first target data type.

[0018] Furthermore, the method further comprises:

[0019] According to the second target data type corresponding to the preset output adaptation interface, the quantity type of the processed data corresponding to each of the task packets is converted into the second target data type.

[0020] Further, for each target algorithm plug-in, the data output type of the processed data obtained by the target algorithm plug-in performing corresponding processing is a single-record output type, a multi-record output type or a combined output type.

[0021] Furthermore, each of the target algorithm plug-ins includes corresponding plug-ins for basic information acquisition, initialization, parameter configuration and reading, input setting, processing logic execution, post-processing output acquisition, status acquisition, message notification, data processing completion notification and post-resource release.

[0022] In a second aspect, in order to solve the above technical problem, the present invention further provides a data processing device, the device comprising:

[0023] An acquisition module, used to acquire a processing request for the data to be processed, wherein the processing request includes a parameter type identifier of the data to be processed;

[0024] A splitting module, used for splitting the data to be processed into at least one task package according to the parameter type identifier of the data to be processed and the business processing flow corresponding to the processing request, each of the task packages includes a type identifier and status information, each of the status information is used to characterize the processing order of the corresponding task package in each processing stage, and the business processing flow corresponds to at least one processing stage;

[0025] A target algorithm plug-in determination module is used to determine the target algorithm plug-in corresponding to each task package in each processing stage according to the type identifier of each task package, each target algorithm plug-in corresponds to a processing algorithm, and each target algorithm plug-in is a virtual plug-in formed by abstracting the corresponding processing algorithm through an IO interface;

[0026] A calling sequence determination module, for each processing stage, based on the status information corresponding to each task package corresponding to the processing stage, generates a calling sequence of the target algorithm plug-in corresponding to each task package corresponding to the processing stage;

[0027] The processing module is used to call the target algorithm plug-ins corresponding to each task package in each processing stage for corresponding processing in the calling order of the target algorithm plug-ins corresponding to each task package in the processing stage, until the processing data corresponding to each task package in each processing stage is obtained.

[0028] In a third aspect, in order to solve the above-mentioned technical problem, the present invention also provides an electronic device, which includes a memory, a processor, and a computer program stored in the memory and executable on the processor, and the data processing method of the present application is implemented when the processor executes the computer program.

[0029] In a fourth aspect, in order to solve the above-mentioned technical problem, the present invention further provides a computer-readable storage medium, on which a computer program is stored, and when the computer program is executed by a processor, the data processing method of the present application is implemented.

[0030] Additional aspects and advantages of the present application will be partially given in the following description, which will become apparent from the following description, or will be understood through the practice of the present application. BRIEF DESCRIPTION OF THE DRAWINGS

[0031] In order to more clearly illustrate the technical solutions in the embodiments of the present invention, the drawings required for describing the embodiments of the present invention are briefly introduced below.

[0032] Figure 1 A flowchart of a data processing method provided by an embodiment of the present invention;

[0033] Figure 2 A data processing software business framework flow chart provided for one embodiment of the present invention;

[0034] Figure 3 A schematic diagram of the structure of a data processing device provided by one embodiment of the present invention;

[0035] Figure 4 A schematic diagram of the structure of an electronic device provided by an embodiment of the present invention. DETAILED DESCRIPTION

[0036] The principles and features of the present invention are described below. The examples given are only used to explain the present invention and are not used to limit the scope of the present invention.

[0037] The technical solution of the present invention and how the technical solution of the present invention solves the above-mentioned technical problems are described in detail below with specific embodiments. The following specific embodiments can be combined with each other, and the same or similar concepts or processes may not be repeated in some embodiments. The embodiments of the present invention will be described below in conjunction with the accompanying drawings.

[0038] The solution provided by the embodiment of the present invention can be applied to any application scenario where business processing is required for the data to be processed. The solution provided by the embodiment of the present invention can be executed by any electronic device, for example, it can be a user's terminal device, including at least one of the following: a smart phone, a tablet computer, a laptop computer, a desktop computer, a smart speaker, a smart watch, a smart TV, and a smart car device.

[0039] The embodiment of the present invention provides a possible implementation method, such as Figure 1 As shown, a flowchart of a data processing method is provided. The scheme can be executed by any electronic device, for example, a terminal device, or a terminal device and a server. For the convenience of description, the method provided by the embodiment of the present invention will be described below by taking the terminal device as the execution subject as an example. Figure 1 The method may include the following steps:

[0040] S10, obtaining a processing request for the data to be processed, wherein the processing request includes a parameter type identifier of the data to be processed;

[0041] S20, splitting the data to be processed into at least one task package according to the parameter type identifier of the data to be processed and the business processing flow corresponding to the processing request, each of the task packages including a type identifier and status information, each of the status information is used to characterize the processing order of the corresponding task package in each processing stage, and the business processing flow corresponds to at least one processing stage;

[0042] S30, determining, according to the type identifier of each task package, a target algorithm plug-in corresponding to each task package in each processing stage, each target algorithm plug-in corresponds to a processing algorithm, and each target algorithm plug-in is a virtual plug-in formed by abstracting the corresponding processing algorithm through an IO interface;

[0043] S40, for each processing stage, based on the status information corresponding to each task package corresponding to the processing stage, generating a calling sequence of the target algorithm plug-in corresponding to each task package corresponding to the processing stage;

[0044] S50, for each processing stage, the target algorithm plug-ins corresponding to the task packages corresponding to the processing stage are called in the calling order of the target algorithm plug-ins corresponding to the task packages corresponding to the processing stage to perform corresponding processing until the processing data corresponding to each task package in each processing stage is obtained.

[0045] Through the method of the present invention, through IO interface abstraction and algorithm plug-in design, direct access and processing of the data to be processed in the business framework are realized without the need for modification at the platform layer; through the use of the algorithm engine and the determination of the task package, the process of calling the target algorithm plug-in corresponding to each task package corresponding to each processing stage is realized, and on this basis, through the calling sequence of the target algorithm plug-in corresponding to each task package corresponding to the processing stage, the processing of the business corresponding to different task packages is realized without the need for modification at the platform layer.

[0046] The solution of the present invention is further described below in conjunction with the following specific embodiments. In this embodiment, a data processing method provided may include the following steps:

[0047] S10, obtaining a processing request for the data to be processed, wherein the processing request includes a parameter type identifier of the data to be processed;

[0048] The data to be processed may be data corresponding to a business function, and the processing request refers to a request to implement the business function. The corresponding business function can only be implemented by combining the algorithm corresponding to the business function and performing corresponding processing according to the data to be processed.

[0049] As an example, the above service function may be a service in the telecommunications field, such as a service for processing a traffic refueling package. The specific field to which the service function corresponds is not limited in the present application.

[0050] The parameter type identifier refers to the type identifier of the data to be processed. One parameter type identifier corresponds to one parameter. For example, there are five parameters, namely, name, position, date, working hours, and off-get off work time. Each parameter can correspond to one parameter type identifier.

[0051] The data to be processed may be obtained based on a user interaction interface, that is, the user may input the data to be processed through the user interaction interface.

[0052] S20, splitting the data to be processed into at least one task package according to the parameter type identifier of the data to be processed and the business processing flow corresponding to the processing request, each of the task packages including a type identifier and status information, each of the status information is used to characterize the processing order of the corresponding task package in each processing stage;

[0053] Among them, the business processing flow refers to the processing flow corresponding to the business function corresponding to the data to be processed. A business processing flow may correspond to at least one processing stage. The processing tasks in the same processing stage are similar processing tasks. A processing task corresponds to a task package. A processing task is regarded as a unit in the business processing flow. The data to be processed can be split into at least one task package according to the number of processing tasks corresponding to the business function. The task package is the basic unit for data processing and related information flow within the business framework. Based on the fact that each task package can complete the corresponding processing task, and at least one parameter may be required to complete a processing task, a task package may include at least one parameter.

[0054] A task package can be divided into different processing stages, that is, the processing task corresponding to a task package can be divided into different processing steps, and a processing stage is equivalent to completing a processing step corresponding to a processing task. The processing process of different task packages in the same processing stage can be the same.

[0055] The data in each task package can be recorded based on the task package information. For each task package, the task package information corresponding to the task package includes the basic task information corresponding to the task package, algorithm plug-in information, parameter data type information (integer, floating point, string, etc.), parameter information (including parameter configuration mode and specific parameter settings), status information (indicating the stage of the corresponding processing task that has been processed and the next processing stage), and other information needed in the task package processing process.

[0056] For each task package, a task package corresponds to a type identifier, which is used to characterize the type of information carried by the task package. For example, when the number of data and the number of algorithm plug-ins constituting the task package are different, they can correspond to different type identifiers.

[0057] It should be noted that different processing stages may include the same processing steps, that is, the same task packages. The processing steps corresponding to the same task package in different processing stages may be the same or different.

[0058] Optionally, the user may choose to generate a single or multiple task packages for the data to be processed according to data processing requirements corresponding to the data to be processed through a UI interface (user interaction interface).

[0059] Optionally, before executing S20, the data to be processed may be processed according to a preset data regularization method to obtain processed data, so that in the subsequent processing process, the data type of the data to be processed may be unified first, thereby enabling the processing of data of different data types. Among them, one of the purposes of processing by the data regularization method is to convert the data to be processed into a data type specified in the data regularization method, and the specified data type facilitates the processing of the business architecture. Other purposes of the data regularization method will be described in detail below and will not be repeated here.

[0060] S30, determining, according to the type identifier of each task package, a target algorithm plug-in corresponding to each task package in each processing stage, each target algorithm plug-in corresponds to a processing algorithm, and each target algorithm plug-in is a virtual plug-in formed by abstracting the corresponding processing algorithm through an IO interface;

[0061] Among them, the virtual plug-in obtained by abstracting the algorithm plug-in through the IO interface can be understood as virtualizing the algorithm into a plug-in, which is convenient for calling, modifying, and editing different algorithms. If there is a new algorithm, the new algorithm can be directly virtualized into a new plug-in by using the IO interface abstraction method and connected to the business framework without the need for transformation at the platform layer.

[0062] Each task package can correspond to at least one target algorithm plug-in. If a task package in a certain processing stage does not have a corresponding target algorithm plug-in, it means that the processing stage does not involve complex algorithms and can be simply completed in the original business platform. That is, the algorithm logic in the business platform can be used to achieve it without the need for an algorithm plug-in.

[0063] Optionally, after S30 and before S40, the method further includes:

[0064] S31 , for each of the task packets, converting the data type corresponding to the task packet into a first target data type corresponding to the input adaptation interface through a preset input adaptation interface.

[0065] The input adapter interface refers to converting the data in the task package into the input data type required by the algorithm plug-in, that is, the first target data type, according to the needs of the algorithm plug-in. It is understandable that if the data type of the task package is the same as the first target data type, no conversion is required.

[0066] The purpose of the input adapter interface is to convert the data in the task package into data that is easy to use for the corresponding target algorithm plug-in, and is not limited to the conversion of data types.

[0067] Optionally, in the above S30, determining the target algorithm plug-in corresponding to each task package in each processing stage according to the type identifier of each task package includes:

[0068] S301, for each processing stage, receiving a selection operation of a user for a target algorithm plug-in identifier displayed on the user interaction interface, wherein the user interaction interface displays an algorithm plug-in identifier corresponding to each task package corresponding to different processing stages, and each of the algorithm plug-in identifiers includes the target algorithm plug-in identifier;

[0069] S302: For each processing stage, according to the selection operation, the algorithm plug-in corresponding to the target algorithm plug-in identifier is used as the target algorithm plug-in corresponding to the task package in the processing stage.

[0070] Optionally, each of the target algorithm plug-ins includes corresponding plug-ins for basic information acquisition, initialization, parameter configuration and reading, input setting, processing logic execution, post-processing output acquisition, status acquisition, message notification, data processing completion notification and post-resource release.

[0071] Among them, basic information acquisition, initialization, parameter configuration and reading, input setting, processing logic execution, post-processing output acquisition, status acquisition, message notification, data processing completion notification and subsequent resource release can each be understood as a data processing algorithm.

[0072] S40, for each processing stage, based on the status information corresponding to each task package corresponding to the processing stage, generating a calling sequence of the target algorithm plug-in corresponding to each task package corresponding to the processing stage;

[0073] Alternatively, for each task package, the calling sequence of the target algorithm plug-in corresponding to the task package in each processing stage is determined for use in S50. Regardless of which method is used to obtain the calling sequence, in order to complete the business function, the corresponding target algorithm plug-in can be called based on the determined calling sequence.

[0074] There is a corresponding processing flow for each processing stage, and the calling order of the target algorithm plug-in corresponding to each task package corresponding to the processing stage can be generated based on the state information corresponding to each task package corresponding to the processing stage. The calling order is the execution order of each target algorithm plug-in.

[0075] S50, for each processing stage, the target algorithm plug-ins corresponding to the task packages corresponding to the processing stage are called in the calling order of the target algorithm plug-ins corresponding to the task packages corresponding to the processing stage to perform corresponding processing until the processing data corresponding to each task package in each processing stage is obtained.

[0076] Optionally, after S50, the method further includes:

[0077] S60, converting the quantity type of the processed data corresponding to each of the task packets into the second target data type according to the second target data type corresponding to the preset output adaptation interface.

[0078] The second target data type refers to the

[0079] Optionally, the method further comprises:

[0080] According to a preset data regularization method, the target data is integrated to obtain integrated data corresponding to the target data, wherein the target data includes the data to be processed and / or the processed data corresponding to each of the task packages, and the integrated data includes the data type identifier, source, parsing method, storage method, reading rule, reading location and algorithm plug-in of the target data;

[0081] The integrated data is stored in the business platform, so that the business platform can manage different data.

[0082] Among them, the specific implementation process of data integration for the processed data corresponding to each task package is: when the input is regularized (data integration for the data to be processed), the specific requirements such as the parsing method and storage location when the data to be processed is output after processing have been recorded, and the output result (the processed data corresponding to each task package) can be restored according to this information.

[0083] Optionally, for each target algorithm plug-in, the data output type of the processed data obtained by the target algorithm plug-in performing corresponding processing is a single-record output type, a multi-record output type or a combined output type.

[0084] Among them, data output type refers to the form of data expression, single record output type refers to the output in the form of a single numerical value, multi-record output type refers to the output in the form of an array, and combined output type refers to the output of key-value pairs similar to the map container type. The generation of data output in different modes is mainly to consider the convenience of connection between similar algorithm plug-ins and the convenience of the algorithm plug-in output itself. For example, in a processing stage, the previous target algorithm plug-in calculates the time difference, and the next target algorithm plug-in converts the time representation type. The input of the previous target algorithm plug-in is two times and the output is a time, and the input of the next target algorithm plug-in is a time and the output is seconds. In this case, there is no need for input adaptation and output adaptation between the two target algorithm plug-ins, and the input and output of the target algorithm plug-in itself are relatively logical. However, when the target algorithm plug-in that calculates the time difference is in the last processing stage corresponding to the entire business processing flow, its output result needs output adaptation to convert the output result into a regular type of data.

[0085] Further, for each processing stage, the target algorithm plug-in corresponding to each task package corresponding to the processing stage is called to perform corresponding processing according to the calling order of the target algorithm plug-in corresponding to each task package corresponding to the processing stage, including:

[0086] For each processing stage, the algorithm engine calls the target algorithm plug-ins corresponding to each task package corresponding to the processing stage to perform corresponding processing according to the calling order of the target algorithm plug-ins corresponding to each task package corresponding to the processing stage.

[0087] Among them, the algorithm engine is mainly used to connect various target algorithm plug-ins based on the calling order.

[0088] Optionally, the algorithm engine may use interface programming to implement the call to each target algorithm plug-in.

[0089] In order to better illustrate and understand the principle of the method provided by the present invention, the scheme of the present invention is described below in conjunction with an optional specific embodiment. It should be noted that the specific implementation of each step in the specific embodiment should not be understood as a limitation to the scheme of the present invention. On the basis of the principle of the scheme provided by the present invention, other implementations that can be thought of by those skilled in the art should also be considered within the scope of protection of the present invention.

[0090] The entire framework involved in this solution adopts an embedded approach to replace all the original data processing algorithms. Through data regularization, the data of the input algorithm of the business platform remains unchanged. After the data of the input processing framework is regularized by the framework itself, the user divides the data to be processed into multiple different task packages through the UI interface. After input adaptation, the task package meets the input conditions of the task flow (business processing flow). The task flow originates from the algorithm plug-in (target algorithm plug-in) to be used by the task package combined by the user through the UI interface. The algorithm plug-in is a summary of all data processing algorithms of the original business platform (the business platform that does not adopt this solution). The missing business logic part can be written and completed by the algorithm engine. Then the algorithm engine is called to arrange different task processing flows for each task package. In each processing stage, each task package has different processing steps. The specific processing steps depend on the status information of the current stage carried in the task package. Therefore, multiple different types of task packages can be processed simultaneously in one task flow, that is, each task package passes through different types of data processing algorithms. After the processing is completed sooner or later, it is decomposed into regularized data through output adaptation, and then the output data is regularized and restored to the original multiple data types of the business platform.

[0091] In this embodiment, see Figure 2 A data processing method is provided, which may include the following processing steps:

[0092] Step 1: Input data formatting

[0093] According to the type, source, parsing method, storage method, etc. of the selected data (data to be processed), a unique ID number (parameter type identifier) ​​is assigned to each parameter type of data, so as to identify and index information within the entire business framework processing range, and can be recorded through a database table.

[0094] Specifically, during the processing, the type name, parsing method, reading rule, and reading location are retrieved according to the parameter type identifier of the data to be processed, and the output data processed by the algorithm plug-in is parsed according to the parameter type identifier to obtain the output storage format and location, etc. In the subsequent use process, new parameter types can be directly registered, and information such as parsing methods and storage locations can be registered at the same time to achieve extended support for data of new parameter types.

[0095] Step 2: Writing the algorithm plugin

[0096] The commonalities of all data processing algorithms of the original business platform are extracted to form a universal algorithm plug-in. The same algorithm plug-in can construct a variety of different types of input and output forms (including single-record input / output, multi-record input / output, and combined input / output) to enhance the connection capabilities between different algorithms. The algorithm plug-in can be written through a dynamic link library.

[0097] Specifically, the algorithm plug-in currently designed mainly includes basic information acquisition, initialization, parameter configuration and reading, input setting, processing logic execution, post-processing output acquisition, status acquisition, message notification, data processing completion notification, post-event resource release, etc. As long as the plug-in conforms to the algorithm input and output adaptation interface, it can be loaded and used by the business framework.

[0098] Step 3: Task package generation and target algorithm plug-in selection

[0099] In the UI interface (user interaction interface), according to the processing process of a specific data processing algorithm of the original business platform and the data type used, select the data with different parameter type identifiers as the data to be processed, as well as the algorithm plug-in required by the algorithm, and record all information in the task package. Different classes can be written as the carrier of each task package. In the UI interface, select the appropriate target algorithm plug-in according to the needs of the task processing flow, laying the foundation for the subsequent algorithm engine to process the business. A task package can correspond to at least one target algorithm plug-in.

[0100] Specifically, the attached UI interface is used to select the data to be processed according to the data processing requirements, and generate a single or multiple task packages. The task package is the basic unit for data processing and related information flow within the business framework. The task package information of a task package mainly records the basic task information, the algorithm plug-in information to be executed, the parameter information (including the parameter configuration mode and the specific parameter setting), and the state maintenance information (state information), especially the state maintenance information. Each task package has specific state information at different task processing stages. The subsequent algorithm engine mainly determines the subsequent processing actions based on the current state of the task package.

[0101] Step 4: Input Adaptation

[0102] Since different processing algorithm plug-ins need to process input data in different modes, including single-record input, multi-record input, and combined input, etc., the algorithm plug-in is combined with strong type constraints to clarify the data type of the input data required by the algorithm plug-in. The input data can be abstracted through the interface. That is, the data type of the data in the task package is converted to the first target data type required by the algorithm plug-in. Corresponding data rationality checks can also be performed according to the requirements of the algorithm plug-in.

[0103] Step 5: Algorithm engine call

[0104] Based on the task package and interface design, the algorithm engine mainly forms a variety of data processing function modules by arranging the serial and parallel connection between target algorithm plug-ins (the calling order of each target algorithm plug-in), and realizes the organization and process connection of single-stage data processing business logic. This can be achieved by writing specific processing process functions.

[0105] Step 6: Task flow execution

[0106] According to the state maintenance information of the task package obtained in the execution phase of the task flow, different algorithm engines are called in sequence to form a complete algorithm processing flow. Different task packages can be processed in each phase in a multi-threaded manner.

[0107] Step 7: Output Adaptation

[0108] Since different algorithm plug-ins will produce different modes of data output, including single record output, multi-record output and combined output, it is necessary to clarify the data type that the algorithm plug-in will output and decompose it into regular data types within the framework. The output data can be converted through the interface.

[0109] Step 8: Output data normalization

[0110] According to the data information that is regularized when the data is input, the output adaptation data is re-regularized and restored to the data type that can be normally recognized and read by the business platform. This can be achieved by reading the database table and parsing according to the corresponding information. That is, the data type of the processed data of each task package is converted into the second target data type, and the second target data type can be the data type of the data to be processed.

[0111] Specifically, based on the abstraction and interface design of data types, input adaptation, functional plug-ins, output adaptation and other related aspects, the algorithm engine mainly implements the organization and flow of data processing business logic through interface programming, that is, the call of each target algorithm plug-in is realized. According to the status maintenance information of the task package obtained in the execution phase of the task flow, a variety of data processing function modules are formed by arranging the series and parallel connection between plug-ins, realizing a multi-level and multi-mode business flow processing mode. After the task flow processing is completed, the calculation results are converted into a unified labeled data type through the algorithm input and output adaptation, and then the data resource abstraction module restores the algorithm output data to the specific data type required by the business platform, and outputs it in the same way as when it was registered.

[0112] Furthermore, the present application scheme is described in conjunction with a specific embodiment:

[0113] In this embodiment, the multi-type differentiated data input (also referred to as raw data) is different types of parameters such as name, position, date, working hours, and off-duty time. These parameters will be assigned different ID numbers (parameter type identifiers) for identification after data regularization. The main content input by the user in the UI interface is to select which types of data to be processed as the data to be processed, and can also choose what kind of processing tasks to complete. The data to be processed in this embodiment includes the five parameters of name, position, date, working hours, and off-duty time. Then, the algorithm plug-in to be used to process these parameters is selected, that is, the target algorithm plug-in, for example, a plug-in for calculating the working hours of a single employee every day, a plug-in for calculating the total working hours of a single employee in a year, and a plug-in for sorting the total working hours of all employees in a year. These three algorithm plug-ins + the above five parameters are divided into one task package, and the task package record information corresponding to the task package includes but is not limited to information needed in the task package processing process, such as status maintenance information (the stage where the task has been processed and the next processing stage), algorithm plug-in information (which plug-ins are there), data types (integer, floating point, string, etc.).

[0114] In this solution, input adaptation refers to converting the data in the task package into the input data type required by the algorithm plug-in according to the needs of the algorithm plug-in. The input required by the plug-in for calculating the working hours of a single employee every day is a single string pair (map) of name + position, the date is an integer array of the number of days in a year, the working hours and off-get off work times are the accumulated seconds (floating-point type array) relative to a certain reference time (such as 0:0:0:0:0 milliseconds on January 1, 2024). The output of the algorithm plug-in is name, position, date, and working hours on that day. If the subsequent task flow stage is to calculate the total working hours of the employee in a year, the working time parameters of the day can be directly used for summation calculation. If the subsequent task flow stage is to determine whether the employee's working hours on that day meet the company's requirements, it may be necessary to re-adapt the input in the algorithm engine.

[0115] The algorithm engine is mainly used to connect various target algorithm plug-ins. A task package can be divided into three processing stages: 1 processing stage, counting the working hours of a single employee every day, 2 processing stage, counting the total working hours of a single employee in a year, 3 processing stage, sorting the working hours of all employees in a year, and the algorithm engine is responsible for the intermediate business process and input and output adaptation. When the selected data to be processed keeps these 5 parameters unchanged, and the algorithm plug-in is changed to a plug-in for calculating the working hours of a single employee every day, and a plug-in for judging whether the working hours of the employee on that day meet the company's requirements, a different task package is formed. This task package is the same as the previous task package in the 1 processing stage in the task flow, and the processing process in the 2 processing stage in the task flow is different (both in a task flow, when a single task flow is specifically implemented, the form of switch case or if can be used to determine which stage the task package will enter later). Then the processing processes with higher similarity can be divided into one processing stage, and those with greater differences can be divided into different processing stages.

[0116] If a business function needs to combine the processing results corresponding to multiple processing tasks (processing data corresponding to multiple task packages), then

[0117] The solution of the present invention has the following beneficial effects:

[0118] 1) Through data resource abstraction and supporting data type registration design, new data types can be directly accessed in the business framework without the need for transformation at the platform level;

[0119] 2) Through IO interface abstraction and algorithm plug-in design, new business functions can be directly connected and activated in the business framework without the need for modification at the platform layer;

[0120] 3) Through the design of algorithm engine and task package definition, the customized organization of business logic and process integration are realized. On this basis, the multi-level and multi-mode business flow processing mode is realized through the series and parallel connection between plug-ins, without the need for modification at the platform level.

[0121] Based on Figure 1 Based on the same principle as the method shown in , the embodiment of the present invention further provides a data processing device 20, such as Figure 3 As shown in , the data processing device 20 may include an acquisition module 210, a splitting module 220, a target algorithm plug-in determination module 230, a call sequence determination module 240 and a processing module 250, wherein:

[0122] An acquisition module 210 is used to acquire a processing request for the data to be processed, wherein the processing request includes a parameter type identifier of the data to be processed;

[0123] A splitting module 220 is used to split the data to be processed into at least one task package according to the parameter type identifier of the data to be processed and the business processing flow corresponding to the processing request, each of the task packages includes a type identifier and status information, each of the status information is used to characterize the processing order of the corresponding task package in each processing stage, and the business processing flow corresponds to at least one processing stage;

[0124] The target algorithm plug-in determination module 230 is used to determine the target algorithm plug-in corresponding to each task package in each processing stage according to the type identifier of each task package, each target algorithm plug-in corresponds to a processing algorithm, and each target algorithm plug-in is a virtual plug-in formed by abstracting the corresponding processing algorithm through an IO interface;

[0125] A calling sequence determining module 240 is used to generate, for each processing stage, a calling sequence of target algorithm plug-ins corresponding to each task package corresponding to the processing stage based on the status information corresponding to each task package corresponding to the processing stage;

[0126] The processing module 250 is used to call the target algorithm plug-ins corresponding to each task package in each processing stage for corresponding processing in the calling order of the target algorithm plug-ins corresponding to each task package in the processing stage, until the processing data corresponding to each task package in each processing stage is obtained.

[0127] Optionally, a data regularization module is used to integrate the target data according to a preset data regularization method to obtain integrated data corresponding to the target data, wherein the target data includes the data to be processed and / or the processed data corresponding to each of the task packages, and the integrated data includes the data type identifier, source, parsing method, storage method, reading rules, reading location and algorithm plug-in of the target data; and the integrated data is stored in the business platform.

[0128] Optionally, the device further comprises:

[0129] The data regularization module is used to integrate the target data according to a preset data regularization method to obtain integrated data corresponding to the target data, wherein the target data includes the data to be processed and / or the processed data corresponding to each of the task packages, and the integrated data includes the data type identifier, source, parsing method, storage method, reading rules, reading location and algorithm plug-in of the target data; and the integrated data is stored in the business platform.

[0130] Optionally, the device further comprises:

[0131] The first data type conversion module is used to convert the data type corresponding to each of the task packets into the first target data type according to the first target data type corresponding to the preset input adaptation interface.

[0132] Optionally, the device further comprises:

[0133] The second data type conversion module is used to convert the quantity type of the processed data corresponding to each of the task packets into the second target data type according to the second target data type corresponding to the preset output adaptation interface.

[0134] Optionally, for each target algorithm plug-in, the data output type of the processed data obtained by the target algorithm plug-in performing corresponding processing is a single-record output type, a multi-record output type or a combined output type.

[0135] Optionally, each of the target algorithm plug-ins includes corresponding plug-ins for basic information acquisition, initialization, parameter configuration and reading, input setting, processing logic execution, post-processing output acquisition, status acquisition, message notification, data processing completion notification and post-resource release.

[0136] The data processing device of the embodiment of the present invention can execute the data processing method provided by the embodiment of the present invention, and the implementation principle is similar. The actions performed by each module and unit in the data processing device in each embodiment of the present invention correspond to the steps in the data processing method in each embodiment of the present invention. For the detailed functional description of each module of the data processing device, please refer to the description in the corresponding data processing method shown in the previous text, which will not be repeated here.

[0137] The data processing device may be a computer program (including program code) running in a computer device, for example, the data processing device is an application software; the device may be used to execute corresponding steps in the method provided in an embodiment of the present invention.

[0138] In some embodiments, the data processing device provided by the embodiment of the present invention can be implemented in a combination of software and hardware. As an example, the data processing device provided by the embodiment of the present invention can be a processor in the form of a hardware decoding processor, which is programmed to execute the data processing method provided by the embodiment of the present invention. For example, the processor in the form of a hardware decoding processor can adopt one or more application-specific integrated circuits (ASICs), DSPs, programmable logic devices (PLDs), complex programmable logic devices (CPLDs), field programmable gate arrays (FPGAs) or other electronic components.

[0139] In other embodiments, the data processing device provided by the embodiments of the present invention may be implemented in software. Figure 3 A data processing device stored in a memory is shown, which may be software in the form of a program and a plug-in, and includes a series of modules, including an acquisition module 210, a splitting module 220, a target algorithm plug-in determination module 230, a call sequence determination module 240 and a processing module 250, for implementing the data processing method provided in an embodiment of the present invention.

[0140] The modules involved in the embodiments of the present invention may be implemented by software or hardware, wherein the name of a module does not limit the module itself in some cases.

[0141] Based on the same principle as the method shown in the embodiments of the present invention, an electronic device is also provided in the embodiments of the present invention, which may include but is not limited to: a processor and a memory; the memory is used to store a computer program; the processor is used to execute the method shown in any embodiment of the present invention by calling the computer program.

[0142] In an alternative embodiment, an electronic device is provided, such as Figure 4 As shown, Figure 4 The electronic device 4000 shown includes: a processor 4001 and a memory 4003. The processor 4001 and the memory 4003 are connected, such as through a bus 4002. Optionally, the electronic device 4000 may also include a transceiver 4004, which may be used for data interaction between the electronic device and other electronic devices, such as data transmission and / or data reception. It should be noted that in actual applications, the transceiver 4004 is not limited to one, and the structure of the electronic device 4000 does not constitute a limitation on the embodiments of the present invention.

[0143] Processor 4001 may be a CPU (Central Processing Unit), a general-purpose processor, a DSP (Digital Signal Processor), an ASIC (Application Specific Integrated Circuit), an FPGA (Field Programmable Gate Array) or other programmable logic devices, transistor logic devices, hardware components or any combination thereof. It may implement or execute various exemplary logic blocks, modules and circuits described in conjunction with the disclosure of the present invention. Processor 4001 may also be a combination that implements computing functions, such as a combination of one or more microprocessors, a combination of a DSP and a microprocessor, etc.

[0144] The bus 4002 may include a path to transmit information between the above components. The bus 4002 may be a PCI (Peripheral Component Interconnect) bus or an EISA (Extended Industry Standard Architecture) bus. The bus 4002 may be divided into an address bus, a data bus, a control bus, etc. For ease of representation, Figure 4 Only one thick line is used in the diagram, but this does not mean that there is only one bus or only one type of bus.

[0145] The memory 4003 can be a ROM (Read Only Memory) or other types of static storage devices that can store static information and instructions, a RAM (Random Access Memory) or other types of dynamic storage devices that can store information and instructions, or an EEPROM (Electrically Erasable Programmable Read Only Memory), a CD-ROM (Compact Disc Read Only Memory) or other optical disk storage, optical disk storage (including compressed optical disk, laser disk, optical disk, digital versatile disk, Blu-ray disk, etc.), a magnetic disk storage medium or other magnetic storage device, or any other medium that can be used to carry or store the desired program code in the form of instructions or data structures and can be accessed by a computer, but is not limited to this.

[0146] The memory 4003 is used to store the application code (computer program) for executing the solution of the present invention, and the execution is controlled by the processor 4001. The processor 4001 is used to execute the application code stored in the memory 4003 to implement the content shown in the above method embodiment.

[0147] The electronic device may also be a terminal device. Figure 4 The electronic device shown is only an example and should not bring any limitation to the functions and scope of use of the embodiments of the present invention.

[0148] An embodiment of the present invention provides a computer-readable storage medium, on which a computer program is stored. When the computer-readable storage medium is run on a computer, the computer can execute the corresponding contents of the aforementioned method embodiment.

[0149] According to another aspect of the present invention, a computer program product or a computer program is also provided, the computer program product or the computer program comprising computer instructions, the computer instructions being stored in a computer-readable storage medium. A processor of a computer device reads the computer instructions from the computer-readable storage medium, and the processor executes the computer instructions, so that the computer device executes the methods provided in the implementation modes of the above-mentioned various embodiments.

[0150] Computer program code for performing the operations of the present invention may be written in one or more programming languages, or a combination thereof, including object-oriented programming languages, such as Java, Smalltalk, C++, and conventional procedural programming languages, such as "C" or similar programming languages. The program code may be executed entirely on the user's computer, partially on the user's computer, as a separate software package, partially on the user's computer and partially on a remote computer, or entirely on a remote computer or server. In cases involving a remote computer, the remote computer may be connected to the user's computer through any type of network, including a local area network (LAN) or a wide area network (WAN), or may be connected to an external computer (e.g., through the Internet using an Internet service provider).

[0151] It should be understood that the flow charts and block diagrams in the accompanying drawings illustrate the possible implementation architecture, functions and operations of the method and computer program product according to various embodiments of the present invention. In this regard, each box in the flow chart or block diagram can represent a module, a program segment, or a part of a code, and the module, the program segment, or a part of the code contains one or more executable instructions for realizing the specified logical function. It should also be noted that in some alternative implementations, the functions marked in the box can also occur in a sequence different from that marked in the accompanying drawings. For example, two boxes represented in succession can actually be executed substantially in parallel, and they can sometimes be executed in the opposite order, depending on the functions involved. It should also be noted that each box in the block diagram and / or flow chart, and the combination of the boxes in the block diagram and / or flow chart can be implemented with a dedicated hardware-based system that performs a specified function or operation, or can be implemented with a combination of dedicated hardware and computer instructions.

[0152] The computer-readable storage medium provided by the embodiment of the present invention may be, for example, but not limited to, an electrical, magnetic, optical, electromagnetic, infrared, or semiconductor system, device or device, or any combination thereof. More specific examples of computer-readable storage media may include, but are not limited to, an electrical connection with one or more wires, a portable computer disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination thereof. In the present invention, a computer-readable storage medium may be any tangible medium containing or storing a program that may be used by or in conjunction with an instruction execution system, device, or device.

[0153] The computer-readable storage medium carries one or more programs. When the one or more programs are executed by the electronic device, the electronic device executes the method shown in the above embodiment.

[0154] The above description is only a preferred embodiment of the present invention and an explanation of the technical principles used. Those skilled in the art should understand that the disclosure scope involved in the present invention is not limited to the technical solution formed by a specific combination of the above technical features, but should also cover other technical solutions formed by any combination of the above technical features or their equivalent features without departing from the above disclosed concept. For example, the above features are replaced with the technical features with similar functions disclosed in the present invention (but not limited to) by each other.

Claims

1. A data processing method, characterized in that: The following steps are involved: Obtaining a processing request for the data to be processed, wherein the processing request includes a parameter type identifier of the data to be processed; According to the parameter type identifier of the data to be processed and the business processing flow corresponding to the processing request, the data to be processed is split into at least one task package, each of the task packages includes a type identifier and status information, each of the status information is used to characterize the processing order of the corresponding task package in each processing stage, and the business processing flow corresponds to at least one processing stage; According to the type identifier of each task package, determine the target algorithm plug-in corresponding to each task package in each processing stage, each target algorithm plug-in corresponds to a processing algorithm, and each target algorithm plug-in is a virtual plug-in formed by abstracting the corresponding processing algorithm through an IO interface; For each processing stage, based on the status information corresponding to each task package corresponding to the processing stage, a calling sequence of the target algorithm plug-in corresponding to each task package corresponding to the processing stage is generated; For each processing stage, the target algorithm plug-ins corresponding to each task package in the processing stage are called in the order of calling the target algorithm plug-ins corresponding to each task package in the processing stage to perform corresponding processing until the processing data corresponding to each task package in each processing stage is obtained.

2. The method according to claim 1, characterized in that The method further comprises: According to a preset data regularization method, the target data is integrated to obtain integrated data corresponding to the target data, wherein the target data includes the data to be processed and / or the processed data corresponding to each of the task packages, and the integrated data includes the data type identifier, source, parsing method, storage method, reading rule, reading location and algorithm plug-in of the target data; The integrated data is stored in the business platform.

3. The method according to claim 1, characterized in that The method further comprises: For each of the task packets, according to the first target data type corresponding to the preset input adaptation interface, the data type corresponding to the task packet is converted into the first target data type.

4. The method according to any one of claims 1 to 3, characterized in that The method further comprises: According to the second target data type corresponding to the preset output adaptation interface, the quantity type of the processed data corresponding to each of the task packets is converted into the second target data type.

5. The method according to any one of claims 1 to 3, characterized in that For each target algorithm plug-in, the data output type of the processed data obtained by the target algorithm plug-in performing corresponding processing is a single record output type, a multi-record output type or a combined output type.

6. The method according to any one of claims 1 to 3, characterized in that Each of the target algorithm plug-ins includes corresponding plug-ins for basic information acquisition, initialization, parameter configuration and reading, input setting, processing logic execution, post-processing output acquisition, status acquisition, message notification, data processing completion notification and post-resource release.

7. A data processing device, characterized in that: The following steps are involved: An acquisition module, used to acquire a processing request for the data to be processed, wherein the processing request includes a parameter type identifier of the data to be processed; A splitting module, used for splitting the data to be processed into at least one task package according to the parameter type identifier of the data to be processed and the business processing flow corresponding to the processing request, each of the task packages includes a type identifier and status information, each of the status information is used to characterize the processing order of the corresponding task package in each processing stage, and the business processing flow corresponds to at least one processing stage; A target algorithm plug-in determination module is used to determine the target algorithm plug-in corresponding to each task package in each processing stage according to the type identifier of each task package, each target algorithm plug-in corresponds to a processing algorithm, and each target algorithm plug-in is a virtual plug-in formed by abstracting the corresponding processing algorithm through an IO interface; A calling sequence determination module, for each processing stage, based on the status information corresponding to each task package corresponding to the processing stage, generates a calling sequence of the target algorithm plug-in corresponding to each task package corresponding to the processing stage; The processing module is used to call the target algorithm plug-ins corresponding to each task package in each processing stage for corresponding processing in the calling order of the target algorithm plug-ins corresponding to each task package in the processing stage, until the processing data corresponding to each task package in each processing stage is obtained.

8. The device according to claim 7, characterized in that The device also includes: The data regularization module is used to integrate the target data according to a preset data regularization method to obtain integrated data corresponding to the target data, wherein the target data includes the data to be processed and / or the processed data corresponding to each of the task packages, and the integrated data includes the data type identifier, source, parsing method, storage method, reading rules, reading location and algorithm plug-in of the target data; and the integrated data is stored in the business platform.

9. An electronic device, characterized in that: The method comprises a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor implements the method according to any one of claims 1 to 6 when executing the computer program.

10. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores a computer program, and when the computer program is executed by a processor, the method according to any one of claims 1 to 6 is implemented.

Images