Data processing method and device, electronic equipment and storage medium
By receiving and analyzing the parameters of data processing requests and triggering the corresponding data service, the problem of separation between data processing and data service is solved, and the stability and consistency of data service is achieved.
Patent Information
- Application Number
- CN202411928677.1
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2024-12-25
- Publication Date
- 2025-04-22
AI Technical Summary
In the prior art, data processing and data services are separated and cannot interact directly, resulting in missing or errors in data services. How to avoid these problems is urgently needed.
By receiving data processing requests, analyzing request parameters, and triggering corresponding data services based on these parameters, the connection between data processing and data services is realized, ensuring the timely triggering of data services and the integrity of data.
It realizes direct interaction between data processing and data services, avoids data loss and error, and ensures the stability and consistency of data services.
Smart Images
Figure CN119862189A_ABST
Abstract
Description
Technical Field
[0001] The present application relates to the field of data processing technology, and in particular to a data processing method, device, electronic device and storage medium. Background Art
[0002] With the continuous development of science and technology, data processing businesses are increasing. The data processing process is generally data integration, data processing and data services. Among them, data services are mainly some functions other than data warehouse functions, such as model feature data acquisition, report email push, data delivery, data push, data selection, etc. Since DataWorks does not meet the data service function, data services are all developed by the company itself. Previously, data processing and data services were separated and could not interact directly. The data service was triggered by time, which would cause the following problems: because the data processing time is not fixed, it may be later than the trigger time of the data service. In this case, the data in the data service is missing or even wrong. How to avoid the problem of missing data and errors in the use of data services is an urgent problem to be solved. Summary of the invention
[0003] The purpose of some embodiments of the present application is to provide a data processing method, device, electronic device and storage medium. Through the technical solution of the embodiments of the present application, a data processing request is received; the data processing request is parsed to obtain request parameters corresponding to the data processing request; and according to the request parameters, a data service corresponding to the request parameters is triggered. In the embodiments of the present application, a trigger function for connecting data processing to data service is provided, which receives a data processing request for data processing, routes it to the corresponding data service through the parameters in the request, triggers the service, and avoids data missing and errors in the use of data services.
[0004] In a first aspect, some embodiments of the present application provide a data processing method, including:
[0005] receiving data processing requests;
[0006] Parsing the data processing request to obtain request parameters corresponding to the data processing request;
[0007] According to the request parameter, a data service corresponding to the request parameter is triggered.
[0008] Some embodiments of the present application provide a trigger function that connects data processing to data services. The function receives a data processing request for data processing, routes it to the corresponding data service through parameters in the request, and triggers the service, thereby avoiding data missing and errors in the use of data services.
[0009] Optionally, the request parameters include at least one or more of a service type, a service address and a service parameter.
[0010] Some embodiments of the present application can search for corresponding data services based on various parameters in the request parameters by obtaining these parameters.
[0011] Optionally, triggering a data service corresponding to the request parameter according to the request parameter includes:
[0012] Through the preset shell script file, according to the request parameters, the data service corresponding to the request parameters is determined, and the data service is triggered.
[0013] Some embodiments of the present application have a unified scheduling method, and the configurable scheduling parameters in the DataWorks shell itself are richer, making the overall service more stable and ensuring data consistency.
[0014] Optionally, the receiving a data processing request includes:
[0015] Receive post request for data processing.
[0016] In some embodiments of the present application, a post request for data processing is received, and the data can be placed in the message body of the request, which has high security and can send large amounts of data.
[0017] In a second aspect, some embodiments of the present application provide a data processing device, including:
[0018] A receiving module, used for receiving a data processing request;
[0019] A parsing module, used to parse the data processing request to obtain request parameters corresponding to the data processing request;
[0020] The trigger module is used to trigger the data service corresponding to the request parameter according to the request parameter.
[0021] Some embodiments of the present application provide a trigger function that connects data processing to data services. The function receives a data processing request for data processing, routes it to the corresponding data service through parameters in the request, and triggers the service, thereby avoiding data missing and errors in the use of data services.
[0022] Optionally, the request parameters include at least one or more of a service type, a service address and a service parameter.
[0023] Some embodiments of the present application can search for corresponding data services based on various parameters in the request parameters by obtaining these parameters.
[0024] Optionally, the trigger module is used to:
[0025] Through the pre-set shell script file, according to the request parameters, the data service corresponding to the request parameters is determined and the data service is triggered. Some embodiments of the present application have a unified scheduling method, and the configurable scheduling parameters in the shell of DataWorks itself are richer, making the overall service more stable and ensuring data consistency.
[0026] Optionally, the receiving module is used to:
[0027] Receive post request for data processing.
[0028] In some embodiments of the present application, a post request for data processing is received and the data can be placed in the message body of the request, which has high security and can send large amounts of data.
[0029] In a third aspect, some embodiments of the present application provide an electronic device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor, when executing the program, can implement the data processing method as described in any embodiment of the first aspect.
[0030] In a fourth aspect, some embodiments of the present application provide a computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, can implement the data processing method as described in any embodiment of the first aspect.
[0031] In a fifth aspect, some embodiments of the present application provide a computer program product, wherein the computer program product comprises a computer program, wherein the computer program, when executed by a processor, can implement the data processing method as described in any embodiment of the first aspect. BRIEF DESCRIPTION OF THE DRAWINGS
[0032] In order to more clearly illustrate the technical solutions of some embodiments of the present application, the drawings required for use in some embodiments of the present application will be briefly introduced below. It should be understood that the following drawings only show some embodiments of the present application and therefore should not be regarded as limiting the scope. For ordinary technicians in this field, other related drawings can be obtained based on these drawings without paying creative work.
[0033] Figure 1 A flowchart of a data processing method provided in an embodiment of the present application;
[0034] Figure 2 A flowchart of another data processing method provided in an embodiment of the present application;
[0035] Figure 3 A schematic diagram of the structure of a data processing device provided in an embodiment of the present application;
[0036] Figure 4 A schematic diagram of an electronic device provided in an embodiment of the present application. DETAILED DESCRIPTION
[0037] The technical solutions in some embodiments of the present application will be described below in conjunction with the drawings in some embodiments of the present application.
[0038] It should be noted that similar reference numerals and letters represent similar items in the following drawings, so once an item is defined in one drawing, it does not need to be further defined and explained in subsequent drawings. At the same time, in the description of this application, the terms "first", "second", etc. are only used to distinguish the description and cannot be understood as indicating or implying relative importance.
[0039] At present, with the continuous development of science and technology, there are more and more data processing businesses, and the data processing process is generally data integration, data processing and data services. Among them, data services are mainly some functions other than data warehouse functions, such as model feature data acquisition, report email push, data delivery, data push, data selection, etc. Since DataWorks does not meet the data service function, data services are all developed by the company itself. Previously, data processing and data services were separated and could not interact directly. The timed triggering of data services was adopted, which would cause problems: because the time of data processing is not fixed, it may be later than the triggering timing of the data service. In this case, the data in the data service is missing or even erroneous. How to avoid the problem of missing data and errors in the use of data services is an urgent problem to be solved. In view of this, some embodiments of the present application provide a data processing method, which receives a data processing request; parses the data processing request to obtain the request parameters corresponding to the data processing request; and triggers the data service corresponding to the request parameters according to the request parameters. In an embodiment of the present application, a trigger function that connects data processing to data services is provided. The function receives a data processing request for data processing, routes it to the corresponding data service through parameters in the request, triggers the service, and avoids data missing and errors in the use of data services.
[0040] like Figure 1 As shown, an embodiment of the present application provides a data processing method, the method comprising:
[0041] S101, receiving a data processing request;
[0042] The terminal device receives a data processing request sent by other terminals, such as an HTTP request, or a post request in an HTTP request;
[0043] S102, parsing the data processing request to obtain request parameters corresponding to the data processing request;
[0044] The terminal device parses the data processing request to obtain request parameters in the data processing request, such as one or more of a service type, a service address, and a service parameter. The request parameter is business-related, and different request parameters can be set according to different businesses.
[0045] S103: According to the request parameters, trigger a data service corresponding to the request parameters.
[0046] The terminal device uses the shell script provided in DataWorks in the cloud product to trigger different data services according to different request parameters.
[0047] Some embodiments of the present application provide a trigger function that connects data processing to data services. The function receives a data processing request for data processing, routes it to the corresponding data service through parameters in the request, and triggers the service, thereby avoiding data missing and errors in the use of data services.
[0048] Another embodiment of the present application further supplements the data processing method provided in the above embodiment.
[0049] Optionally, the request parameters include at least one or more of a service type, a service address and a service parameter.
[0050] Some embodiments of the present application can search for corresponding data services based on various parameters in the request parameters by obtaining these parameters.
[0051] Optionally, according to the request parameters, triggering a data service corresponding to the request parameters includes:
[0052] Through the pre-set shell script file, according to the request parameters, the data service corresponding to the request parameters is determined and the data service is triggered.
[0053] In order to solve this problem, an embodiment of the present application provides a trigger function that connects data processing to data services. This function receives an http request for data processing, routes it to the corresponding data service through parameters in the request, and triggers the service.
[0054] Shell script is similar to batch processing under Windows / Dos, that is, various commands are pre-placed in a file for easy one-time execution of a program file, mainly for administrators to set up or manage. However, it is more powerful than batch processing under Windows and more efficient than programs edited with other programming programs, and it uses commands under Linux / Unix.
[0055] like Figure 2 As shown, the terminal device obtains data from the Alibaba Cloud Data Warehouse, and performs data integration, data cleaning and data processing operations on the data in the Alibaba Cloud Data Warehouse. After the processing is completed, a processing completion notification is returned to the terminal device; and for the terminal device scheduling dependency trigger service, downstream services can be called, where downstream services include at least algorithm model services, data report services or data delivery system services, etc., which are not specifically limited in this application.
[0056] Among them, the algorithm model service includes at least the following:
[0057] Supervised learning: This is one of the most well-known types of algorithm models. It trains the model through a data set of known input-output pairs so that it can predict the output of new data. In the fields of image recognition and speech recognition, supervised learning algorithms have shown powerful performance.
[0058] Unsupervised learning: Unlike supervised learning, unsupervised learning algorithms do not require pre-labeled datasets, and they are able to discover hidden structures and patterns from large amounts of unlabeled data. This has a wide range of applications in recommendation systems, anomaly detection, and other fields.
[0059] Reinforcement learning: Reinforcement learning algorithms learn how to complete tasks by allowing the model to continuously try and optimize strategies in its interaction with the environment. AlphaGo defeating the human Go champion is a classic example of a reinforcement learning algorithm.
[0060] Generative Adversarial Networks (GANs): GANs consist of two parts: a generator and a discriminator. Through adversarial training between the two, the generator can generate realistic data. In the fields of image generation and style transfer, GANs have shown amazing creativity.
[0061] The embodiment of this application is to write and deploy a service that receives post requests. The main parameters received in the service include: service type, service address, service parameters, etc. Then, the built-in shell script of Alibaba Cloud's DataWorks in the cloud product is used to push the post request to this service. In this way, this capability is realized.
[0062] Some embodiments of the present application have a unified scheduling method, and the configurable scheduling parameters in the DataWorks shell itself are richer, making the overall service more stable and ensuring data consistency.
[0063] Optionally, receiving a data processing request includes:
[0064] Receive post request for data processing.
[0065] Among them, the post request is an HTTP method used to submit data or send a request to the server. The post request sends the data in the message body of the request instead of as part of the URL. It is often used in situations such as "submitting form data to the server", "sending large amounts of data" and "performing sensitive operations".
[0066] In some embodiments of the present application, a post request for data processing is received, and the data can be placed in the message body of the request, which has high security and can send large amounts of data.
[0067] The embodiment of the present application can open up the data service link to ensure the freshness of the data in the service. The data service link is opened up, which is convenient for finding upstream and downstream and locating problems. The scheduling method is unified, and the configurable scheduling parameters in the shell of DataWorks itself are richer, making the overall service more stable and ensuring data consistency.
[0068] It should be noted that each implementable method in this embodiment may be implemented separately, or may be implemented in combination in any combination without conflict, and this application is not limited thereto.
[0069] Another embodiment of the present application provides a data processing device for executing the data processing method provided in the above embodiment.
[0070] like Figure 3 , which is a schematic diagram of the structure of a data processing device provided in an embodiment of the present application. The data processing device includes a receiving module 301, a parsing module 302 and a triggering module 303, wherein:
[0071] The receiving module 301 is used to receive a data processing request;
[0072] The parsing module 302 is used to parse the data processing request to obtain request parameters corresponding to the data processing request;
[0073] The trigger module 303 is used to trigger the data service corresponding to the request parameter according to the request parameter.
[0074] Some embodiments of the present application provide a trigger function that connects data processing to data services. The function receives a data processing request for data processing, routes it to the corresponding data service through parameters in the request, and triggers the service, thereby avoiding data missing and errors in the use of data services.
[0075] Regarding the device in this embodiment, the specific manner in which each module performs operations has been described in detail in the embodiment of the method, and will not be elaborated here.
[0076] Another embodiment of the present application further supplements the data processing device provided in the above embodiment.
[0077] Optionally, the request parameters include at least one or more of a service type, a service address and a service parameter.
[0078] Some embodiments of the present application can search for corresponding data services based on various parameters in the request parameters by obtaining these parameters.
[0079] Optionally, the trigger module is used to:
[0080] Through the pre-set shell script file, according to the request parameters, the data service corresponding to the request parameters is determined and the data service is triggered. Some embodiments of the present application have a unified scheduling method, and the configurable scheduling parameters in the shell of DataWorks itself are richer, making the overall service more stable and ensuring data consistency.
[0081] Optionally, the receiving module is used to:
[0082] Receive post request for data processing.
[0083] In some embodiments of the present application, a post request for data processing is received and the data can be placed in the message body of the request, which has high security and can send large amounts of data.
[0084] Regarding the device in this embodiment, the specific manner in which each module performs operations has been described in detail in the embodiment of the method, and will not be elaborated here.
[0085] It should be noted that each implementable method in this embodiment may be implemented separately, or may be implemented in combination in any combination without conflict, and this application is not limited thereto.
[0086] An embodiment of the present application also provides a computer-readable storage medium on which a computer program is stored. When the program is executed by a processor, the operation of the method corresponding to any embodiment of the data processing method provided in the above embodiments can be implemented.
[0087] An embodiment of the present application further provides a computer program product, wherein the computer program product includes a computer program, wherein when the computer program is executed by a processor, it can implement the operations of the method corresponding to any embodiment of the data processing method provided in the above embodiments.
[0088] like Figure 4 As shown, some embodiments of the present application provide an electronic device 400, which includes: a memory 410, a processor 420, and a computer program stored in the memory 410 and executable on the processor 420, wherein the processor 420 can implement a method of any embodiment included in the above-mentioned data processing method when reading the program from the memory 410 through a bus 430 and executing the program.
[0089] Processor 420 can process digital signals and can include various computing structures, such as complex instruction set computer structure, reduced instruction set computer structure, or a structure that implements a combination of multiple instruction sets. In some examples, processor 420 can be a microprocessor.
[0090] The memory 410 may be used to store instructions executed by the processor 420 or data related to the execution of instructions. These instructions and / or data may include codes for implementing some or all functions of one or more modules described in the embodiments of the present application. The processor 420 of the disclosed embodiment may be used to execute instructions in the memory 410 to implement the method shown above. The memory 410 includes a dynamic random access memory, a static random access memory, a flash memory, an optical memory, or other memory known to those skilled in the art.
[0091] The above are only embodiments of the present application and are not intended to limit the scope of protection of the present application. For those skilled in the art, the present application may have various changes and variations. Any modifications, equivalent substitutions, improvements, etc. made within the spirit and principles of the present application should be included in the scope of protection of the present application. It should be noted that similar reference numerals and letters represent similar items in the following drawings, so once an item is defined in one drawing, it does not need to be further defined and explained in the subsequent drawings.
[0092] The above is only a specific implementation of the present application, but the protection scope of the present application is not limited thereto. Any technician familiar with the technical field can easily think of changes or substitutions within the technical scope disclosed in the present application, which should be included in the protection scope of the present application. Therefore, the protection scope of the present application should be based on the protection scope of the claims.
[0093] It should be noted that, in this article, relational terms such as first and second, etc. are only used to distinguish one entity or operation from another entity or operation, and do not necessarily require or imply any such actual relationship or order between these entities or operations. Moreover, the terms "include", "comprise" or any other variants thereof are intended to cover non-exclusive inclusion, so that a process, method, article or device including a series of elements includes not only those elements, but also other elements not explicitly listed, or also includes elements inherent to such process, method, article or device. In the absence of further restrictions, the elements defined by the sentence "comprise a ..." do not exclude the existence of other identical elements in the process, method, article or device including the elements.
Claims
1. A data processing method, characterized in that: The method comprises: receiving data processing requests; Parsing the data processing request to obtain request parameters corresponding to the data processing request; According to the request parameter, a data service corresponding to the request parameter is triggered.
2. The data processing method according to claim 1, characterized in that: The request parameters include at least one or more of a service type, a service address, and a service parameter.
3. The data processing method according to claim 1, characterized in that: The triggering, according to the request parameter, a data service corresponding to the request parameter, includes: Through the preset shell script file, according to the request parameters, the data service corresponding to the request parameters is determined, and the data service is triggered.
4. The data processing method according to claim 1, characterized in that: The receiving of the data processing request comprises: Receive post request for data processing.
5. A data processing device, characterized in that: The device comprises: A receiving module, used for receiving a data processing request; A parsing module, used to parse the data processing request to obtain request parameters corresponding to the data processing request; The trigger module is used to trigger the data service corresponding to the request parameter according to the request parameter.
6. The data processing device according to claim 5, characterized in that: The request parameters include at least one or more of a service type, a service address, and a service parameter.
7. The data processing device according to claim 5, characterized in that: The trigger module is used to: Through the preset shell script file, according to the request parameters, the data service corresponding to the request parameters is determined, and the data service is triggered.
8. The data processing device according to claim 5, characterized in that: The receiving module is used for: Receive post request for data processing.
9. An electronic device, characterized in that: The invention comprises a memory, a processor and a computer program stored in the memory and executable on the processor, wherein the processor can implement the data processing method described in any one of claims 1 to 4 when executing the program.
10. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores a computer program, wherein the program, when executed by a processor, can implement the data processing method described in any one of claims 1 to 4.