A data processing method and device

Abstract

The present invention discloses a data processing method and device, and relates to the field of computer technology. A specific implementation of the method includes: obtaining an original business request initiated by a first native process of a service requester, forwarding the original business request to a first sidecar process of the service requester, so that the first sidecar process obtains the first metadata of the service requester according to the request parameters carried by the original business request; the first sidecar process processes the original business request according to the first metadata and the set first data processing logic, obtains an intermediate business request, and forwards the intermediate business request to the service provider; the first sidecar process receives the response information returned by the service provider to the intermediate processing request, and forwards the response information to the first native process. This implementation isolates the functions of the bus system from the native process that implements the business logic, so that the bus system can be upgraded independently, avoiding the impact of the upgrade on the upstream system, and is transparent to the upstream system.

Description

Technical Field

[0001] The present invention relates to the field of computer technology, and in particular to a data processing method and device. Background Art

[0002] The bus system is a channel for transmitting information between computer components. When a client requests a service from the bus system, the bus system receives the client's request message, processes the request message, and then routes it to the server. The server processes the request and feeds back the processing result to the client.

[0003] Taking the Enterprise Service Bus (ESB) system as an example, the system can provide the most basic connection hub in the network and can constitute an essential element in the enterprise nervous system. At present, in the ESB system, when the client requests a service from the ESB platform, the ESB platform receives the client's request message, parses and encapsulates the request message, and then finds the route corresponding to the request message, thereby routing the request message to the server, and then the server processes the request message and sends the processing result to the client through the ESB platform.

[0004] In the process of implementing the present invention, there are at least the following problems in the prior art:

[0005] The existing bus system is not transparent to the upstream system, and the upstream system needs to be deployed and upgraded before it can be connected to the bus system. The upgrade of the bus system itself also requires the cooperation of upstream and downstream systems. Summary of the invention

[0006] In view of this, an embodiment of the present invention provides a data processing method and device. The method forwards the business request to the sidecar process for processing, isolates the function of the bus system from the native process that implements the business logic, so that the bus system can be upgraded independently, avoiding the impact of the upgrade on the upstream and downstream systems. The method is transparent to the upstream system and can be connected to the bus system without deploying and upgrading the upstream system.

[0007] To achieve the above objective, according to one aspect of an embodiment of the present invention, a data processing method is provided.

[0008] A data processing method according to an embodiment of the present invention includes: obtaining an original business request initiated by a first native process of a service requester, forwarding the original business request to a first sidecar process of the service requester, so that the first sidecar process obtains first metadata of the service requester according to request parameters carried by the original business request; the first sidecar process processes the original business request according to the first metadata and a set first data processing logic to obtain an intermediate business request, and forwards the intermediate business request to a service provider; the first sidecar process receives response information returned by the service provider to the intermediate processing request, and forwards the response information to the first native process.

[0009] Optionally, forwarding the original business request to the first sidecar process of the service requester includes: calling a traffic control interface, modifying the destination address of the routing table of the service requester to the address information corresponding to the first sidecar process of the service requester, so as to forward the original business request to the first sidecar process.

[0010] Optionally, the first metadata includes a first protocol and a first message format allowed by the service requester, and the first data processing logic includes protocol conversion and message conversion; the first sidecar process processes the original business request according to the first metadata and the set first data processing logic, including: the first sidecar process converts the first protocol used to transmit the original business request into a target protocol according to the processing logic corresponding to the protocol conversion; the first sidecar process converts the first message format used to transmit the original business request into a target message format according to the processing logic corresponding to the message conversion.

[0011] Optionally, the first data processing logic also includes any one or more of service monitoring, routing, log collection and fuse.

[0012] Optionally, the first native process and the first sidecar process are deployed on the same machine.

[0013] To achieve the above objective, according to another aspect of an embodiment of the present invention, a data processing method is provided.

[0014] A data processing method according to an embodiment of the present invention includes: calling a second sidecar process of a service provider to obtain an intermediate business request, and obtaining second metadata of the service provider according to request parameters carried by the intermediate business request; the second sidecar process processes the intermediate business request according to the second metadata and a set second data processing logic to obtain a final business request, and forwards the final business request to a second native process of the service provider; the second native process processes the final business request to obtain response information, and sends the response information to the service requester through the second sidecar process.

[0015] Optionally, the calling of the second sidecar process of the service provider to obtain the intermediate business request includes: calling the second sidecar process of the service provider, so that the second sidecar process calls the traffic control interface, modifies the destination address of the routing table of the service provider to the address information corresponding to the second sidecar process, and then obtains the intermediate business request.

[0016] Optionally, the second metadata includes a second protocol and a second message format allowed by the service provider, and the second data processing logic includes protocol conversion and message conversion; the second sidecar process processes the intermediate business request according to the second metadata and the set second data processing logic, including: the second sidecar process converts the target protocol used to transmit the intermediate business request into the second protocol according to the processing logic corresponding to the protocol conversion; the second sidecar process converts the target message format used to transmit the intermediate business request into the second message format according to the processing logic corresponding to the message conversion.

[0017] Optionally, the second data processing logic also includes any one or more of service monitoring, routing, log collection and fuse.

[0018] Optionally, the second native process and the second sidecar process are deployed on the same machine.

[0019] To achieve the above objective, according to another aspect of an embodiment of the present invention, a data processing device is provided.

[0020] A data processing device according to an embodiment of the present invention includes: a request forwarding module, which is used to obtain an original business request initiated by a first native process of a service requester, and forward the original business request to a first sidecar process of the service requester, so that the first sidecar process obtains first metadata of the service requester according to request parameters carried by the original business request; a first request processing module, which is used to process the original business request according to the first metadata and a set first data processing logic by the first sidecar process to obtain an intermediate business request, and forward the intermediate business request to a service provider; and a response forwarding module, which is used to receive, by the first sidecar process, response information returned by the service provider to the intermediate processing request, and forward the response information to the first native process.

[0021] Optionally, the request forwarding module is also used to call the traffic control interface to modify the destination address of the routing table of the service requester to the address information corresponding to the first sidecar process of the service requester, so as to forward the original business request to the first sidecar process.

[0022] Optionally, the first metadata includes a first protocol and a first message format allowed by the service requester, and the first request processing module is further used to convert the first protocol used to transmit the original business request into a target protocol according to a processing logic corresponding to the protocol conversion by the first sidecar process; and convert the first message format used to transmit the original business request into a target message format according to a processing logic corresponding to the message conversion by the first sidecar process.

[0023] Optionally, the first data processing logic also includes any one or more of service monitoring, routing, log collection and fuse.

[0024] Optionally, the first native process and the first sidecar process are deployed on the same machine.

[0025] To achieve the above objective, according to another aspect of an embodiment of the present invention, a data processing device is provided.

[0026] A data processing device according to an embodiment of the present invention includes: a request acquisition module, which is used to call a second sidecar process of a service provider to obtain an intermediate business request, and obtain second metadata of the service provider according to request parameters carried by the intermediate business request; a second request processing module, which is used to process the intermediate business request according to the second metadata and a set second data processing logic by the second sidecar process to obtain a final business request, and forward the final business request to the second native process of the service provider; a response returning module, which is used to process the final business request by the second native process to obtain response information, and send the response information to the service requester through the second sidecar process.

[0027] Optionally, the request acquisition module is also used to call the second sidecar process of the service provider, so that the second sidecar process calls the traffic control interface, modifies the destination address of the routing table of the service provider to the address information corresponding to the second sidecar process, and then obtains the intermediate service request.

[0028] Optionally, the second metadata includes a second protocol and a second message format allowed by the service provider, and the second data processing logic includes protocol conversion and message conversion; the second request processing module is also used for the second sidecar process to convert the target protocol used to transmit the intermediate business request into the second protocol according to the processing logic corresponding to the protocol conversion; and the second sidecar process to convert the target message format used to transmit the intermediate business request into the second message format according to the processing logic corresponding to the message conversion.

[0029] Optionally, the second data processing logic also includes any one or more of service monitoring, routing, log collection and fuse.

[0030] Optionally, the second native process and the second sidecar process are deployed on the same machine.

[0031] To achieve the above objective, according to another aspect of an embodiment of the present invention, an electronic device is provided.

[0032] An electronic device according to an embodiment of the present invention includes: one or more processors; a storage device for storing one or more programs, and when the one or more programs are executed by the one or more processors, the one or more processors implement a data processing method according to an embodiment of the present invention.

[0033] To achieve the above objective, according to another aspect of the embodiments of the present invention, a computer readable medium is provided.

[0034] A computer-readable medium according to an embodiment of the present invention stores a computer program, and when the program is executed by a processor, a data processing method according to an embodiment of the present invention is implemented.

[0035] One embodiment of the above invention has the following advantages or beneficial effects: by forwarding the original business request to the sidecar process of the service requester for processing, the function of the bus system is isolated from the native process that implements the business logic, so that the bus system can be upgraded independently, avoiding the impact of the upgrade on the upstream system, and is transparent to the upstream system, and can be connected to the bus system without deploying and upgrading the upstream system. Through the flow control interface, the direct call of the service requester is realized, solving the problem of bus system expansion.

[0036] Through protocol conversion and message conversion, the protocol used by the service requester is converted to the target protocol, and the message format used by the service requester is converted to the target message format to facilitate data transmission. The sidecar mode is used to implement the data processing logic of the bus system, such as service monitoring, routing, log collection, and fuse, and the corresponding system of the service requester is divided into two processes, avoiding adding code to the system itself and increasing the size of the system. The native process and the sidecar process are deployed on the same machine of the service requester, without adding additional machines on the service requester, saving costs.

[0037] By forwarding the intermediate business request to the service provider's sidecar process for processing, the bus system's functions are isolated from the native process that implements the business logic, so that the bus system can be upgraded independently, avoiding the impact of the upgrade on the downstream system. Through the flow control interface, direct calls from the service provider are realized, solving the problem of bus system expansion. Through protocol conversion and message conversion, the target protocol is converted to the protocol used by the service provider, and the target message format is converted to the message format used by the service provider, which is convenient for the service provider to process.

[0038] The sidecar mode is used to implement the data processing logic of the bus system, such as service monitoring, routing, log collection, and fuse, and the corresponding system of the service provider is divided into two processes, avoiding adding code to the system itself and increasing the size of the system. The native process and the sidecar process are deployed on the same machine of the service provider, eliminating the need to add additional machines on the service provider, saving costs.

[0039] The further effects of the above-mentioned non-conventional optional manner will be described below in conjunction with the specific implementation manner. BRIEF DESCRIPTION OF THE DRAWINGS

[0040] The accompanying drawings are used to better understand the present invention and do not constitute an improper limitation of the present invention.

[0041] Figure 1is a schematic diagram of main steps of a data processing method according to Embodiment 1 of the present invention;

[0042] Figure 2 is a schematic diagram of the main steps of the data processing method according to the second embodiment of the present invention;

[0043] Figure 3 is a schematic diagram of the system architecture of a data processing method according to Embodiment 3 of the present invention;

[0044] Figure 4 Schematic diagram of the relationship between the native process and the sidecar process in Embodiment 3 of the present invention;

[0045] Figure 5 is a schematic diagram of the main flow of the data processing method according to the third embodiment of the present invention;

[0046] Figure 6 1 is a schematic diagram of the conversion process of protocol conversion and message conversion in the data processing method according to the third embodiment of the present invention;

[0047] Figure 7 is a schematic diagram of main modules of a data processing device according to a fourth embodiment of the present invention;

[0048] Figure 8 is a schematic diagram of main modules of a data processing device according to a fifth embodiment of the present invention;

[0049] Fig. 9 is an exemplary system architecture diagram to which embodiments of the present invention may be applied;

[0050] Fig.10 It is a schematic diagram of the structure of a computer device suitable for implementing the electronic device of the embodiment of the present invention. DETAILED DESCRIPTION

[0051] The following is a description of exemplary embodiments of the present invention in conjunction with the accompanying drawings, including various details of the embodiments of the present invention to facilitate understanding, which should be considered as merely exemplary. Therefore, it should be recognized by those of ordinary skill in the art that various changes and modifications may be made to the embodiments described herein without departing from the scope and spirit of the present invention. Similarly, for clarity and conciseness, the description of well-known functions and structures is omitted in the following description.

[0052] The technical terms involved in the embodiments of the present invention are explained below.

[0053] Service mesh: A microservice infrastructure that generally consists of a data plane that is deployed with applications as a network proxy (referred to as a sidecar in this article) and a control plane that manages these sidecars. Service mesh is transparent to applications and non-invasive to code, and does not require applications to modify code or recompile and deploy.

[0054] Sidecar: A separate network proxy process deployed on the application service host, which can add gateway functions to the application without intrusion, including communication message parsing, service registration and discovery, service routing, traffic control, link tracking, etc.

[0055] Service Gateway: It has similar functions to the sidecar, but is deployed centrally and independently from the connected applications.

[0056] WebService: is a platform-independent, low-coupled, self-contained, programmable Web-based application that can use open XML standards to describe, publish, discover, coordinate and configure these applications. It is used to develop distributed interactive applications and can be abbreviated as WS.

[0057] JMS: is the abbreviation of Java Message Service, which means Java Message Service.

[0058] TCP: abbreviation of Transmission Control Protocol, which means transmission control protocol.

[0059] XML: It is the abbreviation of Extensible Markup Language, which means extensible markup language.

[0060] Dubbo: Apache Dubbo is a microservice development framework that provides two major capabilities: RPC communication and microservice governance.

[0061] Http: It is the abbreviation of Hyper Text Transfer Protocol, which means Hypertext Transfer Protocol.

[0062] As described in the background technology, the existing ESB system is an intermediate system responsible for forwarding messages between service requesters and service providers. It can support message routing / addressing, support multiple communication technologies and protocols (such as WebServcie, JMS, TCP), and can process request / response XML messages. And the integration of service requesters and service providers can be achieved through ESB technology.

[0063] However, the existing ESB system is not transparent to the upstream system. The upstream system needs to be deployed and upgraded before it can be connected to the ESB system. The upgrade of the ESB system itself also requires the cooperation of the upstream and downstream systems. In addition, the existing ESB system is generally deployed in a centralized and independent manner, and cannot be horizontally expanded when the concurrency increases, and does not support the RPC service integration framework. In order to solve the above problems, this embodiment provides a new data processing method, which can be applied to an improved ESB system implemented in a distributed architecture, which is described in detail below.

[0064] Figure 1 FIG. 1 is a schematic diagram of the main steps of the data processing method according to the first embodiment of the present invention. Figure 1 As shown, the data processing method of the embodiment of the present invention can be applied to the access adapter of the bus system, and mainly includes the following steps:

[0065] Step S101: Obtain an original service request initiated by a first native process of a service requester, and forward the original service request to a first sidecar process of the service requester, so that the first sidecar process obtains first metadata of the service requester according to request parameters carried by the original service request. The first native process is used to implement the service logic of the service requester, and the first sidecar process is used to implement data processing logic for accessing the bus system for the service requester. The first metadata includes protocol information, message information, etc. allowed to be used by the service requester.

[0066] The service requester calls the first native process to initiate an original business request, which carries request parameters for requesting the service provider to process the request. The access adapter obtains the original business request and forwards it to the first sidecar process of the service requester, which parses the request parameters and then obtains the first metadata of the service requester according to the request parameters.

[0067] Step S102: The first sidecar process processes the original service request according to the first metadata and the set first data processing logic to obtain an intermediate service request, and forwards the intermediate service request to the service provider. After obtaining the first metadata, the first sidecar process processes the original service request according to the first metadata and the set first data processing logic (i.e., the data processing logic for the service requester to access the bus system) to obtain an intermediate service request, and then forwards the intermediate service request to the service provider according to the IP address and port.

[0068] Step S103: The first sidecar process receives the response information returned by the service provider to the intermediate processing request, and forwards the response information to the first native process. The service provider processes the intermediate service request, obtains the response information, and feeds the response information back to the first sidecar process. The first sidecar process receives the response information and forwards the response information to the first native process.

[0069] This embodiment isolates the functions of the bus system from the native process that implements the business logic by forwarding the original business request to the sidecar process of the service requester for processing, so that the bus system can be upgraded independently, avoiding the impact of the upgrade on the upstream system. The bus system is transparent to the upstream system and can be connected to the bus system without deploying and upgrading the upstream system.

[0070] Figure 2 FIG. 1 is a schematic diagram of the main steps of the data processing method according to the second embodiment of the present invention. Figure 2 As shown, the data processing method of the embodiment of the present invention can be applied to the output adapter of the bus system, and mainly includes the following steps:

[0071] Step S201: Call the second sidecar process of the service provider to obtain an intermediate service request, and obtain the second metadata of the service provider according to the request parameters carried by the intermediate service request. The intermediate service request is obtained according to the data processing method described in Example 1, and the second sidecar process is used to implement the data processing logic of the access bus system for the service provider. The second metadata includes protocol information, message information, etc. allowed to be used by the service provider.

[0072] The outbound adapter calls the second sidecar process of the service provider to obtain an intermediate business request, which carries a request parameter for requesting the service provider to process the request. The second sidecar process parses the request parameter and then obtains the second metadata of the service provider according to the request parameter.

[0073] Step S202: The second sidecar process processes the intermediate business request according to the second metadata and the set second data processing logic to obtain a final business request, and forwards the final business request to the second native process of the service provider. After the second sidecar process obtains the second metadata, it processes the intermediate business request according to the second metadata and the set second data processing logic (i.e., the data processing logic for the service provider to implement the access bus system) to obtain a final business request, and then forwards the final business request to the second native process of the service provider.

[0074] Step S203: The second native process processes the final business request, obtains response information, and sends the response information to the service requester through the second sidecar process. The second native process processes the final business request, obtains response information, and feeds the response information back to the second sidecar process, which forwards the response information to the first sidecar process of the service requester.

[0075] This embodiment isolates the functions of the bus system from the native process that implements the business logic by forwarding the intermediate business request to the sidecar process of the service provider for processing, so that the bus system can be upgraded independently, avoiding the impact of the upgrade on the downstream system.

[0076] Figure 3 Schematic diagram of the system architecture of the data processing method according to the third embodiment of the present invention. Figure 3 As shown, the data processing method of the embodiment of the present invention can be implemented by a service requester, a bus system (such as a distributed ESB system) and a service provider. The bus system includes a unified modeling platform, an access adapter and an access adapter, and provides two access modes: a distributed service gateway and a service grid. The service gateway also adopts a distributed architecture and can be expanded according to business capacity and performance requirements.

[0077] The unified modeling platform is used to provide service interface management, service metadata management, configuration of authentication, messages, traffic policies, log query, monitoring management, sidecar management, plug-in adapter management, registration center management, etc. Specifically, it can create the first metadata of the service requester, the second metadata of the service provider, the authorization call relationship, fuse information, current limiting information, etc.

[0078] Among them, the first metadata includes the first protocol, the first message format, the first service instance information and the first address information (including IP address and port) used by the service requester; the second metadata includes the second protocol, the second message format, the second service instance information and the second address information (including IP address and port) used by the service provider.

[0079] In the embodiment, the protocol used by the service requester and the service provider may be WebService, TCP, JMS, Dubbo, Http, etc. The message format used by the service requester and the service provider may be JSON, XML, 8583 message, etc. JSON is the abbreviation of JavaScript Object Notation, which is a lightweight data exchange format; 8583 message is an international standard package format.

[0080] The inbound adapter and the outbound adapter are handled by the sidecar gateway (Phevos-Envoy) component. The sidecar gateway component adopts the sidecar mode and can be used as a companion process for the service requester and the service provider. The sidecar mode deploys different components of a system into different processes to provide isolation and encapsulation, and each component of the system is maintained and updated separately.

[0081] The sidecar mode includes native processes and companion processes. The native process handles the business logic of the upstream and downstream systems, and the sidecar process adds ESB logic for protocol conversion, message conversion, monitoring, logging, routing, and circuit breaking for the upstream and downstream access service buses. By dividing the upstream and downstream systems into two processes, it avoids adding code to the upstream and downstream systems themselves, which increases the size of the system. Moreover, the two processes have different functions and can be upgraded independently, avoiding the impact of ESB system upgrades on the upstream and downstream systems.

[0082] Figure 4 Schematic diagram of the relationship between the native process and the sidecar process in the third embodiment of the present invention. Figure 4 As shown, Figure 3 Take the card system of as an example. The system consists of a native process of Java technology and a sidecar process of C++ technology (this process is a companion process). The native process handles the business logic of the card system, and the sidecar process adds protocol conversion, message conversion, service monitoring, routing, log collection and ESB logic for the card system to access the service bus.

[0083] Based on the above sidecar mode, the access adapter of this embodiment is used to forward the original business request sent by the first native process of the service requester to the corresponding first sidecar process, and the first sidecar process executes the first data processing logic for accessing the ESB system and forwards the processed business request to the service provider. The access adapter is used to call the second sidecar process of the service provider to obtain the processed business request, execute the second data processing logic for accessing the ESB system, and forward the secondary processed business request to the second native process of the service provider for processing. The functions of the access adapter and the access adapter are both implemented by the sidecar gateway component.

[0084] In addition, the distributed ESB system of this embodiment includes a service grid composed of a data plane and a control plane. The data plane is the sidecar gateway component, and the control plane is the unified modeling platform. The unified modeling platform can push the created first metadata and second metadata to the sidecar gateway component. The service grid makes the communication between each service instance smoother, more reliable and faster.

[0085] Figure 5 FIG. 1 is a schematic diagram of the main flow of the data processing method according to Embodiment 3 of the present invention. Figure 5As shown, the data processing method of the embodiment of the present invention mainly includes the following steps:

[0086] Step S501: The access adapter obtains the original business request sent by the first native process of the service requester, and forwards the original business request to the first sidecar process of the service requester. In an embodiment, the original business request sent by the first native process of the service requester can be obtained by binding a listener to the listening port. The protocol used to forward the original business request can be the WebService protocol.

[0087] Preferably, the operating system modifies the destination address of the service requester's routing table to the address information corresponding to the first sidecar process of the service requester by calling the traffic control interface, so as to forward the original business request sent by the first native process of the service requester to the first sidecar process of the service requester.

[0088] Specifically, the NAT routing table of the host is modified through the application program interface (API) of the operating system, and the original service request is redirected to a specific IP / PORT by modifying the IP / PORT (i.e., IP address and port number) in the table. NAT is the abbreviation of Network Address Translation, which means network address translation.

[0089] Step S502: The first sidecar process obtains first metadata of the service requester according to the request parameters carried by the original business request, and executes a first data processing logic for accessing the bus system on the original business request according to the first metadata to obtain an intermediate business request. In an embodiment, the first metadata may include a first protocol and a first message format allowed to be used by the service requester, the first data processing logic may include protocol conversion and message conversion, and the bus system may be an ESB system.

[0090] In actual applications, the implementation process of the first sidecar process processing the original business request can be: the first sidecar process converts the first protocol used to transmit the original business request into the target protocol according to the processing logic corresponding to the protocol conversion; and converts the first message format used to transmit the original business request into the target message format according to the processing logic corresponding to the message conversion.

[0091] The above processing is used to convert the protocol type (such as WebService) of the protocol used by the service requester to transmit the original business request into a protocol type (such as Http) applicable to the ESB system, and convert the message format (such as XML) of the message used by the service requester to transmit the original business request into a message format (such as JSON) applicable to the ESB system.

[0092] In a preferred embodiment, the first data processing logic may also include any one or more of service monitoring, log collection, routing, fuse and traffic strategy. In practical applications, service monitoring can be reported through embedding points, logs are collected asynchronously, and traffic strategy is implemented using token buckets.

[0093] It should be noted that the first metadata is pre-created by the unified modeling platform. Specifically, the first metadata is created for the service requester, wherein the first metadata includes not only the first protocol and the first message format used by the service requester, but also the first service instance information and the first address information.

[0094] Step S503: After the first data processing logic is executed, the first sidecar process forwards the obtained intermediate service request to the service provider. After the first data processing logic is executed, the first sidecar process initiates a request to the corresponding service provider according to the IP address and port number. The protocol used to forward the intermediate service request can be the Http protocol.

[0095] Step S504: Call the second sidecar process of the service provider to obtain the intermediate business request, and obtain the second metadata of the service provider according to the request parameters carried by the intermediate business request, so as to execute the second data processing logic for accessing the bus system on the intermediate business request according to the second metadata to obtain the final business request.

[0096] In an embodiment, the second metadata may include a second protocol and a second message format allowed by the service provider. The second sidecar process of the service provider modifies the destination address of the routing table of the service provider to the address information corresponding to the second sidecar process by calling the flow control interface, thereby obtaining the intermediate service request. In addition, the second data processing logic is the same as the first processing logic, and may also include protocol conversion and message conversion.

[0097] In actual applications, the implementation process of the second sidecar process processing the intermediate business request can be: the second sidecar process converts the target protocol used to transmit the intermediate business request into the second protocol according to the processing logic corresponding to the protocol conversion; and converts the target message format used to transmit the intermediate business request into the second message format according to the processing logic corresponding to the message conversion.

[0098] The above processing is used to convert the protocol type (such as Http) of the protocol used by the ESB system to transmit intermediate business requests into the protocol type applicable to the service provider (such as WebService), and convert the message format (such as JSON) of the message used by the ESB system to transmit intermediate business requests into the message format applicable to the service provider (such as 8583).

[0099] In a preferred embodiment, the second data processing logic may also include any one or more of service monitoring, log collection, routing, fuse and traffic strategy. It should be noted that the second metadata is also pre-created by the unified modeling platform. Specifically, the second metadata is created for the service provider; wherein, in addition to the second protocol and the second message format used by the service provider, the second metadata may also include the second service instance information and the second address information.

[0100] Figure 6 FIG. 1 is a schematic diagram of the conversion process of protocol conversion and message conversion in the data processing method according to the third embodiment of the present invention. Figure 6 As shown, the service requester host can use the RPC protocol to transmit a first message carrying the original service request to the bus system, assuming that the message format of the first message is XML. The bus system calls the first sidecar process, converts the message format of the first message into its applicable JSON format, and forwards the message to the second sidecar process via the Http protocol. The second sidecar process determines that the service provider host accepts the JSON format message sent via the Http protocol, and then forwards the message to the second native process for processing.

[0101] Through protocol conversion and message conversion, the mutual conversion between TCP, WebServcie, RPC protocol and Http protocol in the ESB system, as well as the mutual conversion between XML message, UTF-8 (a variable-length character encoding) message and JSON message are realized.

[0102] Step S505: After the second data processing logic is executed, the second sidecar process forwards the final service request to the second native process of the service provider. Preferably, the second native process and the second sidecar process are deployed on the same machine. In this case, the second sidecar process can forward the final service request to the second native process of the service provider within the service provider host. The protocol used to forward the final service request can be the TCP protocol.

[0103] Step S506: The second native process processes the final business request and returns the obtained response information to the second sidecar process. The second native process processes the final business request according to the business logic, and after the processing is completed, returns the obtained response information to the second sidecar process.

[0104] Step S507: The second sidecar process forwards the response information to the first sidecar process.

[0105] Step S508: The first sidecar process forwards the response information to the first native process. Preferably, the first native process and the first sidecar process are deployed on the same machine, and the first sidecar process can forward the response information to the first native process inside the service requester host.

[0106] In an optional embodiment, a sidecar mode is adopted, a first sidecar process is configured for the service requester, and a second sidecar process is configured for the service provider.

[0107] It can be seen from the data processing method of the embodiment of the present invention that the embodiment realizes point-to-point calls between service requesters and service providers through distributed sidecar gateway components without going through a centralized ESB agent, thus solving the horizontal expansion problem of the existing ESB system. Moreover, by calling the flow control interface and the sidecar technology, the functions and business processes of the ESB system are isolated, so that the ESB system can be upgraded independently, avoiding the impact of the upgrade on the upstream and downstream systems, and is transparent to the upstream system, and can be connected to the ESB system without deploying and upgrading the upstream system.

[0108] Figure 7 Schematic diagram of main modules of a data processing device according to Embodiment 4 of the present invention. Figure 7 As shown, the data processing device 700 of the embodiment of the present invention mainly includes:

[0109] The request forwarding module 701 is used to obtain the original business request initiated by the first native process of the service requester, and forward the original business request to the first sidecar process of the service requester, so that the first sidecar process obtains the first metadata of the service requester according to the request parameters carried by the original business request. The first native process is used to implement the business logic of the service requester, and the first sidecar process is used to implement the data processing logic of the access bus system for the service requester. The first metadata includes protocol information, message information, etc. allowed to be used by the service requester.

[0110] The service requester calls the first native process to initiate an original business request, which carries request parameters for requesting the service provider to process the request. The request forwarding module obtains the original business request and forwards the original business request to the first sidecar process of the service requester, which parses the request parameters and then obtains the first metadata of the service requester according to the request parameters.

[0111] The first request processing module 702 is used for the first sidecar process to process the original service request according to the first metadata and the set first data processing logic, obtain the intermediate service request, and forward the intermediate service request to the service provider. After the first sidecar process obtains the first metadata, it processes the original service request according to the first metadata and the set first data processing logic (that is, the data processing logic for the service requester to access the bus system) to obtain the intermediate service request, and then forwards the intermediate service request to the service provider according to the IP address and port.

[0112] The response forwarding module 703 is configured to receive, by the first sidecar process, the response information returned by the service provider to the intermediate processing request, and forward the response information to the first native process. The first sidecar process receives the response information and forwards the response information to the first native process.

[0113] From the above description, it can be seen that by forwarding the original business request to the sidecar process of the service requester for processing, the functions of the bus system and the native process that implements the business logic are isolated, so that the bus system can be upgraded independently, avoiding the impact of the upgrade on the upstream system. It is transparent to the upstream system and can be connected to the bus system without deploying and upgrading the upstream system.

[0114] Figure 8 Schematic diagram of the main modules of the data processing device according to the fifth embodiment of the present invention. Figure 8 As shown, the data processing device 800 of the embodiment of the present invention mainly includes:

[0115] The request acquisition module 801 is used to call the second sidecar process of the service provider to obtain the intermediate business request, and obtain the second metadata of the service provider according to the request parameters carried by the intermediate business request; wherein the intermediate business request is obtained according to the data processing device of embodiment 4, and the second sidecar process is used to implement the data processing logic of the access bus system for the service provider, and the second metadata includes protocol information, message information, etc. allowed to be used by the service provider.

[0116] The request acquisition module calls the second sidecar process of the service provider to obtain the intermediate business request, which carries request parameters for requesting the service provider to process the request. The second sidecar process parses the request parameters and then obtains the second metadata of the service provider according to the request parameters.

[0117] The second request processing module 802 is used for the second sidecar process to process the intermediate business request according to the second metadata and the set second data processing logic to obtain the final business request, and forward the final business request to the second native process of the service provider. After the second sidecar process obtains the second metadata, it processes the intermediate business request according to the second metadata and the set second data processing logic (that is, the data processing logic for the service provider to implement the access bus system) to obtain the final business request, and then forwards the final business request to the second native process of the service provider.

[0118] The response return module 803 is used for the second native process to process the final business request, obtain response information, and send the response information to the service requester through the second sidecar process. The second native process processes the final business request, obtains response information, and feeds the response information back to the second sidecar process, which forwards the response information to the first sidecar process of the service requester.

[0119] From the above description, it can be seen that by forwarding the intermediate business request to the sidecar process of the service provider for processing, the functions of the bus system are isolated from the native process that implements the business logic, so that the bus system can be upgraded independently, avoiding the impact of the upgrade on the downstream system.

[0120] Fig. 9 An exemplary system architecture 900 is shown to which the data processing method or data processing device according to the embodiment of the present invention can be applied.

[0121] like Fig. 9 As shown, system architecture 900 may include servers 901, 903, 905 and networks 902, 904. Network 902 is used to provide a medium for a communication link between server 901 and server 903. Network 904 is used to provide a medium for a communication link between server 903 and server 905. Networks 902, 904 may include various connection types, such as wired, wireless communication links or optical fiber cables, etc.

[0122] Servers 901, 903, and 905 can all be servers that provide various services. Among them, server 901, as a service requester, requests service from server 903; server 903 forwards the original business request to the sidecar process of the service requester, performs the first data processing logic, and forwards the intermediate business request obtained by the processing to server 905 (i.e., the service provider); the second sidecar process of the service provider obtains the intermediate business request, performs the second data processing logic, forwards the final business request obtained by the processing to the second native process for processing, and feeds back the processing result to server 901.

[0123] It should be noted that the data processing method provided in the embodiment of the present application is generally executed by the server 903, and accordingly, the data processing device is generally set in the server 903.

[0124] It should be understood that Fig. 9 The number of networks and servers in the embodiment is only for illustration. Any number of networks and servers may be provided as required.

[0125] According to an embodiment of the present invention, the present invention also provides an electronic device and a computer-readable medium.

[0126] The electronic device of the present invention comprises: one or more processors; a storage device for storing one or more programs, and when the one or more programs are executed by the one or more processors, the one or more processors implement a data processing method of an embodiment of the present invention.

[0127] The computer-readable medium of the present invention stores a computer program, and when the program is executed by a processor, a data processing method of an embodiment of the present invention is implemented.

[0128] Reference below Fig.10 , which shows a schematic diagram of the structure of a computer system 1000 suitable for implementing an electronic device of an embodiment of the present invention. Fig.10 The electronic device shown is only an example and should not bring any limitation to the functions and scope of use of the embodiments of the present invention.

[0129] like Fig.10 As shown, the computer system 1000 includes a central processing unit (CPU) 1001, which can perform various appropriate actions and processes according to a program stored in a read-only memory (ROM) 1002 or a program loaded from a storage part 1008 into a random access memory (RAM) 1003. In the RAM 1003, various programs and data required for the operation of the computer system 1000 are also stored. The CPU 1001, the ROM 1002, and the RAM 1003 are connected to each other through a bus 1004. An input / output (I / O) interface 1005 is also connected to the bus 1004.

[0130] The following components are connected to the I / O interface 1005: an input section 1006 including a keyboard, a mouse, etc.; an output section 1007 including a cathode ray tube (CRT), a liquid crystal display (LCD), etc., and a speaker, etc.; a storage section 1008 including a hard disk, etc.; and a communication section 1009 including a network interface card such as a LAN card, a modem, etc. The communication section 1009 performs communication processing via a network such as the Internet. A drive 1000 is also connected to the I / O interface 1005 as needed. A removable medium 1001, such as a magnetic disk, an optical disk, a magneto-optical disk, a semiconductor memory, etc., is installed on the drive 1000 as needed, so that a computer program read therefrom is installed into the storage section 1008 as needed.

[0131] In particular, according to the embodiments disclosed in the present invention, the process described in the main step diagram above can be implemented as a computer software program. For example, the embodiments disclosed in the present invention include a computer program product, which includes a computer program carried on a computer-readable medium, and the computer program includes a program code for executing the method shown in the main step diagram. In such an embodiment, the computer program can be downloaded and installed from the network through the communication part 1009, and / or installed from the removable medium 1001. When the computer program is executed by the central processing unit (CPU) 1001, the above-mentioned functions defined in the system of the present invention are executed.

[0132] It should be noted that the computer-readable medium shown in the present invention may be a computer-readable signal medium or a computer-readable storage medium or any combination of the above two. The computer-readable storage medium may be, for example, but not limited to, an electrical, magnetic, optical, electromagnetic, infrared, or semiconductor system, device or device, or any combination of the above. More specific examples of computer-readable storage media may include, but are not limited to: an electrical connection with one or more wires, a portable computer disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the above. In the present invention, a computer-readable storage medium may be any tangible medium containing or storing a program that can be used by or in combination with an instruction execution system, device or device. In the present invention, a computer-readable signal medium may include a data signal propagated in a baseband or as part of a carrier wave, which carries a computer-readable program code. This propagated data signal may take a variety of forms, including but not limited to electromagnetic signals, optical signals, or any suitable combination of the above. Computer-readable signal media may also be any computer-readable medium other than computer-readable storage media, which may send, propagate or transmit a program for use by or in conjunction with an instruction execution system, apparatus or device. The program code contained on the computer-readable medium may be transmitted using any appropriate medium, including but not limited to: wireless, wire, optical cable, RF, etc., or any suitable combination of the above.

[0133] The flow chart and block diagram in the accompanying drawings illustrate the possible architecture, function and operation of the system, method and computer program product according to various embodiments of the present invention. In this regard, each box in the flow chart or block diagram can represent a module, a program segment, or a part of a code, and the above-mentioned module, program segment, or a part of a code contains one or more executable instructions for realizing the specified logical function. It should also be noted that in some alternative implementations, the functions marked in the box can also occur in a different order from the order marked in the accompanying drawings. For example, two boxes represented in succession can actually be executed substantially in parallel, and they can sometimes be executed in the opposite order, depending on the functions involved. It should also be noted that each box in the block diagram or flow chart, and the combination of the boxes in the block diagram or flow chart can be implemented with a dedicated hardware-based system that performs a specified function or operation, or can be implemented with a combination of dedicated hardware and computer instructions.

[0134] The modules involved in the embodiments of the present invention may be implemented in software or in hardware. The modules described may also be set in a processor. For example, they may be described as: a processor includes a request forwarding module, a first request processing module, and a response forwarding module. The names of these modules do not, in some cases, constitute a limitation on the module itself. For example, the request forwarding module may also be described as "a module that obtains the original business request initiated by the first native process of the service requester, and forwards the original business request to the first sidecar process of the service requester, so that the first sidecar process obtains the first metadata of the service requester according to the request parameters carried by the original business request."

[0135] As another aspect, the present invention also provides a computer-readable medium, which may be included in the device described in the above embodiment; or it may exist independently without being assembled into the device. The above computer-readable medium carries one or more programs, and when the above one or more programs are executed by a device, the device includes: obtaining an original business request initiated by a first native process of a service requester, forwarding the original business request to a first sidecar process of the service requester, so that the first sidecar process obtains the first metadata of the service requester according to the request parameters carried by the original business request; the first sidecar process processes the original business request according to the first metadata and the set first data processing logic to obtain an intermediate business request, and forwards the intermediate business request to the service provider; the first sidecar process receives the response information returned by the service provider to the intermediate processing request, and forwards the response information to the first native process.

[0136] According to the technical solution of the embodiment of the present invention, by forwarding the original business request to the sidecar process of the service requester for processing, the functions of the bus system are isolated from the native process that implements the business logic, so that the bus system can be upgraded independently, avoiding the impact of the upgrade on the upstream system. The bus system is transparent to the upstream system and can be connected to the ESB system without deploying and upgrading the upstream system.

[0137] The above product can execute the method provided by the embodiment of the present invention, and has the functional modules and beneficial effects corresponding to the execution method. For technical details not described in detail in this embodiment, please refer to the method provided by the embodiment of the present invention.

[0138] The above specific implementations do not constitute a limitation on the protection scope of the present invention. It should be understood by those skilled in the art that various modifications, combinations, sub-combinations and substitutions may occur depending on design requirements and other factors. Any modification, equivalent substitution and improvement made within the spirit and principle of the present invention shall be included in the protection scope of the present invention.

Claims

1. A data processing method, characterized in that: include: Obtaining an original business request initiated by a first native process of a service requester; wherein the service requester performs a point-to-point call with a service provider based on a distributed sidecar gateway component; Forwarding the original business request to the first sidecar process of the service requester, so that the first sidecar process obtains the first metadata of the service requester according to the request parameters carried by the original business request; wherein the first sidecar process isolates the function of the bus system from the native process; forwarding the original business request to the first sidecar process of the service requester includes: calling a flow control interface, modifying the destination address of the routing table of the service requester to the address information corresponding to the first sidecar process of the service requester, so as to forward the original business request to the first sidecar process; The first sidecar process processes the original business request according to the first metadata and the set first data processing logic to obtain an intermediate business request, and forwards the intermediate business request to the service provider; the first metadata includes the first protocol and the first message format allowed by the service requester, and the first data processing logic includes protocol conversion and message conversion; the first sidecar process processes the original business request according to the first metadata and the set first data processing logic, including: the first sidecar process converts the first protocol used to transmit the original business request into a target protocol according to the processing logic corresponding to the protocol conversion; the first sidecar process converts the first message format used to transmit the original business request into a target message format according to the processing logic corresponding to the message conversion; the protocols used by the service requester and the service provider include any one of WebService, TCP, JMS, Dubbo, and Http; The first sidecar process receives response information returned by the service provider to the intermediate service request, and forwards the response information to the first native process.

2. The method according to claim 1, characterized in that The first data processing logic also includes any one or more of service monitoring, routing, log collection and fuse.

3. The method according to any one of claims 1 to 2, characterized in that: The first native process and the first sidecar process are deployed on the same machine.

4. A data processing method, characterized in that: include: Calling the second sidecar process of the service provider to obtain the intermediate business request; wherein the service provider performs a point-to-point call with the service requester based on the distributed sidecar gateway component; According to the request parameters carried by the intermediate business request, the second metadata of the service provider is obtained; wherein the intermediate business request is obtained according to the method according to any one of claims 1 to 3; wherein the second sidecar process isolates the functions of the bus system from the native process; the second sidecar process of calling the service provider to obtain the intermediate business request comprises: calling the second sidecar process of the service provider, so that the second sidecar process calls the flow control interface, and modifies the destination address of the routing table of the service provider to the address information corresponding to the second sidecar process, thereby obtaining the intermediate business request; The second sidecar process processes the intermediate business request according to the second metadata and the set second data processing logic to obtain a final business request, and forwards the final business request to the second native process of the service provider; the second metadata includes a second protocol and a second message format allowed by the service provider, and the second data processing logic includes protocol conversion and message conversion; the second sidecar process processes the intermediate business request according to the second metadata and the set second data processing logic, including: the second sidecar process converts the target protocol used to transmit the intermediate business request into the second protocol according to the processing logic corresponding to the protocol conversion; the second sidecar process converts the target message format used to transmit the intermediate business request into the second message format according to the processing logic corresponding to the message conversion; the protocol used by the service requester and the service provider includes any one of WebService, TCP, JMS, Dubbo, and Http; The second native process processes the final business request, obtains response information, and sends the response information to the service requester through the second sidecar process.

5. The method according to claim 4, characterized in that The second data processing logic also includes any one or more of service monitoring, routing, log collection and fuse.

6. The method according to any one of claims 4 to 5, characterized in that: The second native process and the second sidecar process are deployed on the same machine.

7. A data processing device, characterized in that: include: A request forwarding module, used to obtain the original business request initiated by the first native process of the service requester; wherein the service requester performs a point-to-point call with the service provider based on a distributed sidecar gateway component; Forwarding the original business request to the first sidecar process of the service requester, so that the first sidecar process obtains the first metadata of the service requester according to the request parameters carried by the original business request; wherein the first sidecar process isolates the function of the bus system from the native process; forwarding the original business request to the first sidecar process of the service requester includes: calling a flow control interface, modifying the destination address of the routing table of the service requester to the address information corresponding to the first sidecar process of the service requester, so as to forward the original business request to the first sidecar process; The first request processing module is used for the first sidecar process to process the original business request according to the first metadata and the set first data processing logic, obtain an intermediate business request, and forward the intermediate business request to the service provider; the first metadata includes the first protocol and the first message format allowed by the service requester, and the first data processing logic includes protocol conversion and message conversion; the first sidecar process processes the original business request according to the first metadata and the set first data processing logic, including: the first sidecar process converts the first protocol used to transmit the original business request into a target protocol according to the processing logic corresponding to the protocol conversion; the first sidecar process converts the first message format used to transmit the original business request into a target message format according to the processing logic corresponding to the message conversion; the protocols used by the service requester and the service provider include any one of WebService, TCP, JMS, Dubbo, and Http; A response forwarding module is used to receive, by the first sidecar process, response information returned by the service provider to the intermediate business request, and forward the response information to the first native process.

8. A data processing device, characterized in that: include: A request acquisition module, used to call the second sidecar process of the service provider to obtain the intermediate business request; wherein the service provider performs a point-to-point call with the service requester based on a distributed sidecar gateway component; According to the request parameters carried by the intermediate business request, the second metadata of the service provider is obtained; wherein the intermediate business request is obtained by the device according to claim 7; wherein the second sidecar process isolates the function of the bus system from the native process; the second sidecar process of calling the service provider to obtain the intermediate business request includes: calling the second sidecar process of the service provider, so that the second sidecar process calls the flow control interface, and modifies the destination address of the routing table of the service provider to the address information corresponding to the second sidecar process, thereby obtaining the intermediate business request; The second request processing module is used for the second sidecar process to process the intermediate business request according to the second metadata and the set second data processing logic, obtain the final business request, and forward the final business request to the second native process of the service provider; the second metadata includes the second protocol and the second message format allowed by the service provider, and the second data processing logic includes protocol conversion and message conversion; the second sidecar process processes the intermediate business request according to the second metadata and the set second data processing logic, including: the second sidecar process converts the target protocol used to transmit the intermediate business request into the second protocol according to the processing logic corresponding to the protocol conversion; the second sidecar process converts the target message format used to transmit the intermediate business request into the second message format according to the processing logic corresponding to the message conversion; the protocol used by the service requester and the service provider includes any one of WebService, TCP, JMS, Dubbo, and Http; A response returning module is used to process the final business request by the second native process, obtain response information, and send the response information to the service requester through the second sidecar process.

9. An electronic device, characterized in that: include: one or more processors; a storage device for storing one or more programs, When the one or more programs are executed by the one or more processors, the one or more processors implement the method according to any one of claims 1 to 6.

10. A computer readable medium having a computer program stored thereon, characterized in that: When the program is executed by a processor, the method according to any one of claims 1 to 6 is implemented.

Images