Data processing method and device, computer device, and storage medium
By sending data stream logic objects to non-Java operators on the Java platform and recording their information, the problem of the Java platform's difficulty in calling non-Java operators is solved, thus improving the platform's scalability and data processing efficiency.
Patent Information
- Authority / Receiving Office
- CN · China
- Patent Type
- Patents(China)
- Current Assignee / Owner
- CHENGDU UNION BIG DATA TECH CO LTD
- Filing Date
- 2022-10-10
- Publication Date
- 2026-06-02
AI Technical Summary
In existing technologies, platforms built with the Java language have difficulty effectively calling data processing operators developed in non-Java languages.
The Java platform sends the data stream logic object to a non-Java operator, and the non-Java operator writes operator information into the data stream logic object. The Java platform then uses this information to call the non-Java operator to process the data.
It enables the calling of non-Java operators on the Java platform, improving the platform's scalability and maintainability, and increasing data processing efficiency.
Smart Images
Figure CN115525277B_ABST
Abstract
Description
Technical Field
[0001] This invention relates to the field of computer technology, and more specifically, to a data processing method, apparatus, computer equipment, and storage medium. Background Technology
[0002] In data processing, data processing operators are mainly implemented using programming languages such as Java, Scala, and Python, and data processing operators developed in each programming language can often only run on platforms built with the same programming language.
[0003] Given the relatively mature ecosystem of the Java language, the current challenge is to overcome the problem of how to call data processing operators developed in other non-Java languages on a platform built with Java. Summary of the Invention
[0004] To overcome the shortcomings of the prior art, embodiments of the present invention provide a data processing method, apparatus, computer device, and computer-readable storage medium.
[0005] The technical solution of this invention can be implemented as follows:
[0006] In a first aspect, embodiments of the present invention provide a data processing method applied to a computer device, the computer device being deployed with a Java platform and non-Java operators, the method comprising:
[0007] The Java platform sends the pre-generated data flow logic object to the non-Java operator through the operator initialization interface of the non-Java operator. The data flow logic object includes multiple items to be written, and each item to be written is used to record operator information of one of the non-Java operators.
[0008] The non-Java operator writes the operator information of the non-Java operator corresponding to each item to be written into the data stream logic object, and returns the data stream logic object to the Java platform;
[0009] The Java platform calls the non-Java operators to perform data processing based on the operator information of the non-Java operators recorded in each item to be written.
[0010] Optionally, the data stream logic object further includes a Java interface corresponding to each item to be written, and the step of writing the operator information of the non-Java operator corresponding to each item to be written into the data stream logic object by the non-Java operator includes:
[0011] The non-Java operator calls the Java interface corresponding to each item to be written through its own operator initialization interface to write the operator information of the non-Java operator corresponding to each item to be written into the data stream logic object.
[0012] The beneficial effect of the above implementation method is that non-Java operators call multiple Java interfaces of the Java platform through their own operator initialization interface to inform the Java platform of the operator information of non-Java operators.
[0013] Optionally, the operator information of the non-Java operator includes functional description information, parameter information, and interface information. The step of the Java platform calling the non-Java operator based on the operator information of the non-Java operator recorded for each item to be written includes:
[0014] The Java platform determines whether the non-Java operator meets the first preset condition based on the functional description information, whereby the first preset condition represents the data processing requirements of the Java platform.
[0015] If the conditions are met, the Java platform calls the non-Java operator based on the parameter information and the interface information.
[0016] The beneficial effect of the above implementation method is that the Java platform determines the non-Java operators that meet the current data processing requirements based on the functional description information.
[0017] Optionally, the interface information includes the data processing interface of the non-Java operator, and the step of the Java platform calling the non-Java operator based on the parameter information and the interface information includes:
[0018] The Java platform creates a parameter object for the non-Java operator based on the parameter information;
[0019] The Java platform sends the parameter object and the data stream to be processed to the data processing interface of the non-Java operator in order to invoke the non-Java operator.
[0020] The beneficial effect of the above implementation method is that the Java platform can call non-Java operators based on the data processing interface and parameter information of the non-Java operators.
[0021] Optionally, the parameter object includes multiple parameters to be determined, and the method further includes:
[0022] The non-Java operator takes the undetermined parameter that satisfies the second preset condition as the target parameter, and the second preset condition represents the algorithm logic of the non-Java operator.
[0023] The non-Java operator processes the data stream to be processed based on the target parameters.
[0024] The beneficial effect of the above implementation method is that non-Java operators use the undetermined parameters that satisfy their Java operators as the target parameters for their runtime, thus ensuring the accuracy of the data processing results.
[0025] Optionally, before the Java platform sends the pre-generated data stream logic object to the non-Java operator through the operator initialization interface of the non-Java operator, the method further includes:
[0026] The Java platform obtains the interface name of each interface of the non-Java operator;
[0027] The Java platform uses the interface with the preset name as the operator initialization interface for the non-Java operator.
[0028] The beneficial effect of the above implementation method is that the Java platform determines the operator initialization interface based on the name of each interface of the non-Java operator, so as to realize the sending of data stream logic objects to the non-Java operator.
[0029] Secondly, embodiments of the present invention provide a data processing apparatus applied to a computer device, the apparatus including a Java platform and non-Java operators;
[0030] The Java platform is used to send a pre-generated data flow logic object to the non-Java operator through the operator initialization interface of the non-Java operator. The data flow logic object includes multiple items to be written, and each item to be written is used to record operator information of the non-Java operator.
[0031] The non-Java operator is used to write the operator information of the non-Java operator corresponding to each item to be written into the data stream logic object, and return the data stream logic object to the Java platform;
[0032] The Java platform is also used to call the non-Java operator for data processing based on the operator information of the non-Java operator recorded in each of the items to be written.
[0033] Optionally, the data flow logic object also includes a Java interface corresponding to each item to be written;
[0034] The non-Java operator is specifically used to call the Java interface corresponding to each item to be written through its own operator initialization interface, so as to write the operator information of the non-Java operator corresponding to each item to be written into the data stream logic object.
[0035] Thirdly, embodiments of the present invention provide a computer device including a memory and a processor, wherein the memory stores a computer program, and the processor executes the computer program to implement the data processing method as described in the first aspect above.
[0036] Fourthly, the present invention provides a computer-readable storage medium containing a computer program, which, when executed by a processor, implements the data processing method described in the first aspect above.
[0037] Compared to existing technologies, the data processing method, apparatus, computer device, and storage medium provided in this embodiment of the invention firstly involve a Java platform sending a pre-generated data stream logic object to a non-Java operator through a non-Java operator's operator initialization interface. The data stream logic object includes multiple items to be written, each recording operator information for a non-Java operator. Then, the non-Java operator writes the operator information corresponding to each item to be written into the data stream logic object and returns the data stream logic object to the Java platform. Finally, the Java platform invokes the non-Java operator based on the operator information recorded in each item to be written to perform data processing. Because in this embodiment of the invention, the Java platform sends the data stream logic object to the non-Java operator, and the non-Java operator writes the operator information corresponding to each item to be written into the data stream logic object before returning the data stream logic object to the Java platform, the Java platform can invoke the non-Java operator to perform data processing based on the operator information recorded in the data stream logic object. Attached Figure Description
[0038] To more clearly illustrate the technical solutions of the embodiments of the present invention, the accompanying drawings used in the embodiments will be briefly introduced below. It should be understood that the following drawings only show some embodiments of the present invention and should not be regarded as a limitation on the scope. For those skilled in the art, other related drawings can be obtained based on these drawings without creative effort.
[0039] Figure 1 A flowchart illustrating a data processing method provided in an embodiment of the present invention. Figure 1 ;
[0040] Figure 2A flowchart illustrating a data processing method provided in an embodiment of the present invention. Figure 2 ;
[0041] Figure 3 A functional unit block diagram of a data counting device provided in an embodiment of the present invention;
[0042] Figure 4 This is a schematic block diagram of a computer device provided in an embodiment of the present invention.
[0043] Icons: 100 - Data processing device; 101 - Java platform; 102 - Non-Java operator; 200 - Computer equipment; 210 - Memory; 220 - Processor. Detailed Implementation
[0044] To make the objectives, technical solutions, and advantages of the embodiments of the present invention clearer, the technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. The components of the embodiments of the present invention described and shown in the accompanying drawings can generally be arranged and designed in various different configurations.
[0045] Therefore, the following detailed description of the embodiments of the invention provided in the accompanying drawings is not intended to limit the scope of the claimed invention, but merely to illustrate selected embodiments of the invention. All other embodiments obtained by those skilled in the art based on the embodiments of the invention without inventive effort are within the scope of protection of the invention.
[0046] It should be noted that similar labels and letters in the following figures indicate similar items. Therefore, once an item is defined in one figure, it does not need to be further defined and explained in subsequent figures.
[0047] Furthermore, the terms "first" and "second" are used only to distinguish descriptions and should not be interpreted as indicating or implying relative importance.
[0048] It should be noted that, where there is no conflict, the features in the embodiments of the present invention can be combined with each other.
[0049] To enable the use of data processing operators developed in other non-Java languages on a Java-based platform, this invention provides a data processing method applicable to computer devices equipped with a Java platform and non-Java operators. The method is described in detail below.
[0050] Please refer to Figure 1 The data processing method includes steps S101 to S103.
[0051] S101, the Java platform sends the pre-generated data flow logic object to the non-Java operator through the operator initialization interface of the non-Java operator.
[0052] The data flow logic object includes multiple items to be written, each of which is used to record operator information for a non-Java operator.
[0053] Non-Java operators refer to data processing operators written in non-Java languages, such as C, C++, Scala, or Python. Each non-Java operator deployed on a computer device has an operator initialization interface.
[0054] For non-Java operators, operator information includes functional description information, parameter information, and interface information.
[0055] The functional description information reflects the application scenario of non-Java operators. For example, if the functional description information of a non-Java operator is "sorting", it means that the non-Java operator can be applied to sorting scenarios.
[0056] The parameter information reflects the format requirements of the input / output data when the non-Java operator is running. For example, if the parameter information of the non-Java operator is a two-dimensional integer array, it means that the format of the input / output data when the non-Java operator is running is a two-dimensional integer array.
[0057] Since non-Java operators have multiple interfaces, it is understandable that the interface information reflects the interface used by the non-Java operator to receive data at runtime.
[0058] S102, non-Java operators write the operator information of the non-Java operator corresponding to each item to be written into the data stream logic object, and return the data stream logic object to the Java platform.
[0059] The data flow logic object also includes the Java interface corresponding to each item to be written. Understandably, the Java interface is an interface provided by the Java platform.
[0060] It should be noted that in this embodiment of the invention, the Java interface corresponding to each item to be written is different.
[0061] Optionally, step S102 can be implemented as follows:
[0062] Non-Java operators call the Java interface corresponding to each item to be written through their own operator initialization interface to write the operator information of the non-Java operator corresponding to each item to be written into the data stream logic object.
[0063] For example, item 1 is used to record functional description information. The Java interface A corresponding to item 1 is called by a non-Java operator through its own operator initialization interface to write the functional description information of the non-Java operator into item 1.
[0064] Item 2 to be written is used to record parameter information. The Java interface B corresponding to item 2 to be written is called by the non-Java operator through its own operator initialization interface to write the parameter information of the non-Java operator into item 2 to be written.
[0065] Item 3 to be written is used for interface information. The Java interface C corresponding to item 3 to be written is called by the non-Java operator through its own operator initialization interface, and the interface information of the non-Java operator is written into item 3 to be written.
[0066] S103, the Java platform calls non-Java operators to perform data processing based on the operator information of each non-Java operator in the record to be written.
[0067] Optionally, step S103 may include sub-steps S103-1 and S103-2.
[0068] S103-1, The Java platform determines whether a non-Java operator meets the first preset condition based on the functional description information.
[0069] The first preset condition represents the data processing requirements of the Java platform.
[0070] For example, assuming the data processing requirement of the Java platform is "sorting", if the functional description information of a non-Java operator is "sorting", it means that the non-Java operator can be used in the current data processing scenario of the Java platform; if the functional description information of a non-Java operator is "deduplication", it means that the non-Java operator cannot be used in the current data processing scenario of the Java platform.
[0071] S103-2, if satisfied, the Java platform will call a non-Java operator based on the parameter information and interface information.
[0072] The interface information includes data processing interfaces for non-Java operators.
[0073] In this embodiment of the invention, for each non-Java operator, the Java platform determines the non-Java operator that meets the first preset condition, i.e., can be used in the current data processing scenario of the Java platform, based on the functional description information recorded in the data stream logic object it returns.
[0074] The implementation process of step S103-2 can be as follows:
[0075] First, the Java platform creates a parameter object for non-Java operators based on the parameter information;
[0076] Then, the Java platform sends the parameter object and the data stream to be processed to the data processing interface of the non-Java operator to invoke the non-Java operator.
[0077] The parameter object includes multiple parameters to be determined.
[0078] Since the parameter information reflects the format requirements of the input / output data during the runtime of non-Java operators, it is understandable that there are multiple input / output data for non-Java operators. Therefore, a parameter to be determined is created for each input / output data format requirement.
[0079] For example, assuming that in the parameter information of a non-Java operator, input data 1 is a two-dimensional integer array, input data 2 is an integer constant, and output data is a one-dimensional floating-point array, then create a parameter 1 that is a two-dimensional integer array, a parameter 2 that is an integer constant, and a parameter 3 that is a one-dimensional floating-point array.
[0080] In this embodiment of the invention, after the Java platform sends the parameter object and the data stream to be processed to the non-Java interface data processing interface, the following process is also included:
[0081] First, non-Java operators take the undetermined parameters that satisfy the second preset condition as the target parameters.
[0082] The second preset condition represents the algorithm logic of non-Java operators.
[0083] Understandably, among the multiple undetermined parameters, there may be undetermined parameters that are not used in the runtime of non-Java operators, and these undetermined parameters cannot be used as target parameters.
[0084] For example, suppose the non-Java operator receives pending parameter 1, pending parameter 2, pending parameter 3, pending parameter 4 and pending parameter 5. Since pending parameter 3 does not satisfy the algorithm logic of the non-Java operator, that is, the non-Java operator does not use this pending parameter when running. Therefore, the determined target parameters include pending parameter 1, pending parameter 2, pending parameter 4 and pending parameter 5.
[0085] Then, the non-Java operators process the data stream based on the target parameters.
[0086] Understandably, non-Java operators execute their processing logic based on target parameters, process the data stream to be processed, and return the processed data stream to the Java platform.
[0087] Please refer to Figure 2 Prior to step S101, the data processing method includes steps S104 to S105.
[0088] S104, Java platform retrieves the interface name of each interface of non-Java operators.
[0089] Since non-Java operators have multiple interfaces, and the Java platform needs to know the operator initialization interface of the non-Java operator in order to send the data stream logic object to the non-Java operator, the operator initialization interface needs to be determined based on the interface name of each interface of the non-Java operator.
[0090] S105, the Java platform uses interfaces with preset names as operator initialization interfaces for non-Java operators.
[0091] Understandably, for each non-Java operator, the interface with the default name is used as the operator initialization interface for that non-Java operator.
[0092] Compared with the prior art, the embodiments of the present invention have the following technical effects:
[0093] (1) The Java platform sends the data stream logic object to the non-Java operator through the operator initialization interface of the non-Java operator. The non-Java operator calls the Java interface corresponding to each item to be written through its own operator initialization interface, and writes the operator information of the non-Java operator corresponding to each item to be written into the data stream logic object. In this way, the relevant information of the non-Java operator and the actual data processing interface are informed to the Java platform, and the data processing operators developed by other non-Java languages can be called on the platform built by the Java language.
[0094] (2) The calling process of the Java platform is decoupled from the data processing process of non-Java operators, thereby improving the scalability and maintainability of the Java platform and non-Java operators.
[0095] (3) It can achieve mixed execution of Java platform and multiple non-Java operators within a single process, thereby improving data processing efficiency.
[0096] In order to perform the corresponding steps in the above method embodiments and various possible implementations, an implementation of a data processing device 100 is given below.
[0097] Please refer to Figure 3 The data processing device 100 includes a Java platform 101 and non-Java operators 102.
[0098] Java Platform 101 is used to send pre-generated data stream logic objects to non-Java operators through the operator initialization interface of non-Java operators. The data stream logic objects include multiple items to be written, and each item is used to record operator information of a non-Java operator.
[0099] Non-Java operator 102 is used to write the operator information of the non-Java operator corresponding to each item to be written into the data stream logic object, and return the data stream logic object to the Java platform.
[0100] The Java platform 101 is also used to call non-Java operators for data processing based on the operator information of the non-Java operators in each record to be written.
[0101] Optionally, the data flow logic object also includes a Java interface corresponding to each item to be written. The non-Java operator 102 is specifically used to call the Java interface corresponding to each item to be written through its own operator initialization interface, so as to write the operator information of the non-Java operator corresponding to each item to be written into the data flow logic object.
[0102] Optionally, the operator information of the non-Java operator includes functional description information, parameter information, and interface information. The Java platform 101 is specifically used to determine whether the non-Java operator meets the first preset condition based on the functional description information. The first preset condition represents the data processing requirements of the Java platform. If it meets the condition, the non-Java operator is called based on the parameter information and interface information.
[0103] Optionally, the interface information includes a data processing interface for non-Java operators. When the Java platform 101 calls a non-Java operator based on the parameter information and the interface information, it specifically creates a parameter object for the non-Java operator based on the parameter information and sends the parameter object and the data stream to be processed to the data processing interface of the non-Java operator to call the non-Java operator.
[0104] Optionally, the parameter object includes multiple undetermined parameters, and the non-Java operator 102 is also used to take the undetermined parameters that satisfy the second preset condition as the target parameter, the second preset condition representing the algorithm logic of the non-Java operator; based on the target parameter, the data stream to be processed is processed.
[0105] Optionally, before sending the pre-generated data stream logic object to the non-Java operator through the operator initialization interface of the non-Java operator, the Java platform 101 is also used to obtain the interface name of each interface of the non-Java operator; and to use the interface with the preset name as the operator initialization interface of the non-Java operator.
[0106] Those skilled in the art will clearly understand that, for the sake of convenience and brevity, the specific working process of the data processing device 100 described above can be referred to the corresponding process in the foregoing method embodiments, and will not be repeated here.
[0107] Furthermore, this embodiment of the invention also provides a computer device 200, please refer to... Figure 4 The computer device 200 may include a memory 210 and a processor 220.
[0108] The processor 220 may be a general-purpose central processing unit (CPU), a microprocessor, an application-specific integrated circuit (ASIC), or one or more integrated circuits used to control the execution of a program for controlling the data processing method provided in the above method embodiments.
[0109] The memory 210 may be a ROM or other type of static storage device capable of storing static information and instructions, RAM or other type of dynamic storage device capable of storing information and instructions, or an electrically erasable programmable-only memory (EEPROM), a compact disc read-only memory (CD-ROM) or other optical disc storage, optical disc storage (including compressed optical discs, laser discs, optical discs, digital universal optical discs, Blu-ray discs, etc.), magnetic disk storage media or other magnetic storage devices, or any other medium capable of carrying or storing desired program code in the form of instructions or data structures and accessible by a computer, but not limited thereto. The memory 210 may exist independently and be connected to the processor 220 via a communication bus. The memory 210 may also be integrated with the processor 220. The memory 210 is used to store machine-executable instructions for executing the scheme of this application. The processor 220 is used to execute the machine-executable instructions stored in the memory 210 to implement the above-described method embodiments.
[0110] This invention also provides a computer-readable storage medium containing a computer program, which, when executed, can be used to perform related operations in the data processing method provided in the above-described method embodiments.
[0111] In summary, the data processing method, apparatus, computer device, and storage medium provided in this embodiment of the invention firstly involve a Java platform sending a pre-generated data stream logic object to a non-Java operator via a non-Java operator's operator initialization interface. The data stream logic object includes multiple items to be written, each recording operator information for a non-Java operator. Then, the non-Java operator writes the operator information corresponding to each item to be written into the data stream logic object and returns the data stream logic object to the Java platform. Finally, the Java platform invokes the non-Java operator based on the operator information recorded in each item to be written to perform data processing. Because in this embodiment, the Java platform sends the data stream logic object to the non-Java operator, and the non-Java operator writes the operator information corresponding to each item to be written into the data stream logic object before returning it to the Java platform, the Java platform can invoke the non-Java operator to perform data processing based on the operator information recorded in the data stream logic object.
[0112] The above are merely specific embodiments of the present invention, but the scope of protection of the present invention is not limited thereto. Any variations or substitutions that can be easily conceived by those skilled in the art within the technical scope disclosed in the present invention should be included within the scope of protection of the present invention. Therefore, the scope of protection of the present invention should be determined by the scope of the claims.
Claims
1. A data processing method, characterized in that, Applied to a computer device, wherein the computer device is deployed with a Java platform and non-Java operators, the method includes: The Java platform obtains the interface name of each interface of the non-Java operator; The Java platform uses the interface with the preset name as the operator initialization interface for the non-Java operator. The Java platform sends the pre-generated data flow logic object to the non-Java operator through the operator initialization interface of the non-Java operator. The data flow logic object includes multiple items to be written, and each item to be written is used to record operator information of one of the non-Java operators. The non-Java operator writes the operator information of the non-Java operator corresponding to each item to be written into the data stream logic object, and returns the data stream logic object to the Java platform; The Java platform calls the non-Java operators to perform data processing based on the operator information of the non-Java operators recorded in each item to be written.
2. The method as described in claim 1, characterized in that, The data stream logic object also includes a Java interface corresponding to each item to be written. The step of writing the operator information of the non-Java operator corresponding to each item to be written into the data stream logic object by the non-Java operator includes: The non-Java operator calls the Java interface corresponding to each item to be written through its own operator initialization interface to write the operator information of the non-Java operator corresponding to each item to be written into the data stream logic object.
3. The method as described in claim 1, characterized in that, The operator information of the non-Java operator includes functional description information, parameter information, and interface information. The steps by which the Java platform invokes the non-Java operator based on the operator information of the non-Java operator recorded for each item to be written include: The Java platform determines whether the non-Java operator meets the first preset condition based on the functional description information, whereby the first preset condition represents the data processing requirements of the Java platform. If the conditions are met, the Java platform calls the non-Java operator based on the parameter information and the interface information.
4. The method as described in claim 3, characterized in that, The interface information includes the data processing interface of the non-Java operator. The steps by which the Java platform calls the non-Java operator based on the parameter information and the interface information include: The Java platform creates a parameter object for the non-Java operator based on the parameter information; The Java platform sends the parameter object and the data stream to be processed to the data processing interface of the non-Java operator in order to invoke the non-Java operator.
5. The method as described in claim 4, characterized in that, The parameter object includes multiple parameters to be determined, and the method further includes: The non-Java operator takes the undetermined parameter that satisfies the second preset condition as the target parameter, and the second preset condition represents the algorithm logic of the non-Java operator. The non-Java operator processes the data stream to be processed based on the target parameters.
6. A data processing apparatus, characterized in that, Applied to computer equipment, the device includes a Java platform and non-Java operators; The Java platform is used to obtain the interface name of each interface of the non-Java operator; and to use the interface with the preset name as the operator initialization interface of the non-Java operator. The Java platform is also used to send the pre-generated data flow logic object to the non-Java operator through the operator initialization interface of the non-Java operator, wherein the data flow logic object includes multiple items to be written, and each item to be written is used to record operator information of the non-Java operator. The non-Java operator is used to write the operator information of the non-Java operator corresponding to each item to be written into the data stream logic object, and return the data stream logic object to the Java platform; The Java platform is also used to call the non-Java operator for data processing based on the operator information of the non-Java operator recorded in each of the items to be written.
7. The apparatus as claimed in claim 6, characterized in that, The data stream logic object also includes a Java interface corresponding to each item to be written; The non-Java operator is specifically used to call the Java interface corresponding to each item to be written through its own operator initialization interface, so as to write the operator information of the non-Java operator corresponding to each item to be written into the data stream logic object.
8. A computer device, characterized in that, It includes a memory and a processor, the memory storing a computer program, and the processor executing the computer program to implement the data processing method as described in any one of claims 1 to 5.
9. A computer-readable storage medium storing a computer program, characterized in that, When the computer program is executed by the processor, it implements the data processing method as described in any one of claims 1 to 5.