Data processing method, system and equipment based on proxy middleware and medium

Abstract

The invention provides a data processing method, system and device based on proxy middleware and a medium, and relates to the technical field of data proxy, and the method comprises the following steps: obtaining a business application, and embedding the proxy middleware into the business application; the latest configuration information is read and loaded, and the target API interface of the proxy middleware is determined, so that the expandability and the flexibility of the system are improved; receiving an original HTTP (Hyper Text Transport Protocol) request of the target API interface through the proxy middleware; when the current mode is a first mode, performing data processing on the original HTTP request through the proxy middleware according to a first processing strategy to generate a target data product; through the proxy middleware, the original HTTP request can be intelligently identified and processed, the request is modified and sent to the correct downstream service through the accurate service position information and the downstream service information, the accuracy and efficiency of data processing are effectively improved, then a processed target data product is rapidly obtained, and the user experience is improved. Therefore, the response speed and the user experience are enhanced.

Description

Technical Field

[0001] The present invention relates to the technical field of data proxy, and particularly relates to a data processing method, system, electronic device and readable storage medium based on a proxy middleware. Background Art

[0002] With the development of information technology and the popularization of data middle platforms, data products emerge in an endless stream. In the process of developing data products, a microservices architecture is usually adopted. When building data products, programs often implement the proxy work for the data middle platform through code writing, which leads to an increasingly large number of API interfaces of the data middle platform. When the number of docking interfaces gradually increases and multiple people collaborate, it becomes increasingly difficult to manage data products.

[0003] However, the mainstream API proxy interface management products on the market today have poor integration with the configuration system and front-end applications when applied to the data product development process; the proxy service functions are weak and the functions are single.

[0004] Therefore, it is necessary to propose a data processing method, system, electronic device and readable storage medium based on a proxy middleware. Summary of the Invention

[0005] This specification provides a data processing method, system, electronic device and readable storage medium based on a proxy middleware, which realizes flexible interaction between business applications and API interfaces by embedding the proxy middleware, improves the data processing efficiency and flexibility; this method supports dynamic configuration update to ensure that the proxy middleware always runs based on the latest configuration, enhancing the system's response speed. Through careful determination of service location information and modification of HTTP requests, efficient forwarding and response processing of requests are achieved, which helps to generate high-quality data products. In addition, this method also has data proxy analysis and monitoring functions. By asynchronously sending proxy information and analyzing the results, comprehensive monitoring of data processing is realized, which helps to discover and solve potential problems in a timely manner.

[0006] A data processing method based on a proxy middleware provided by this application adopts the following technical solutions, including:

[0007] Obtain a business application and embed the proxy middleware into the business application;

[0008] Read and load the latest configuration information to determine the target API interface of the proxy middleware;

[0009] Receive the original HTTP request of the target API interface through the proxy middleware;

[0010] When the current mode is the first mode, the proxy middleware processes the original HTTP request according to the first processing strategy to generate a target data product; wherein, determining the service location information corresponding to the original HTTP request; combining the service location information to find the downstream information corresponding to the external downstream service; modifying the original HTTP request based on the downstream information to obtain a target HTTP request; sending the target HTTP request to the external downstream service; generating a target data product according to the response result returned by the external downstream service;

[0011] The proxy middleware returns the target data product to the business application.

[0012] Optionally, the proxy middleware uses the Spring Boot microservice architecture as the underlying framework.

[0013] Optionally, the reading and loading of the latest configuration information to determine the target API interface of the proxy middleware includes:

[0014] Obtaining the initial configuration information;

[0015] Modifying the initial configuration information to obtain the latest configuration information;

[0016] Based on the release instruction, sending the latest configuration information to the configuration management service;

[0017] Reading the latest configuration information of the proxy middleware from the configuration management service;

[0018] Based on the latest configuration information, performing configuration update on the proxy middleware;

[0019] After the configuration update of the proxy middleware is completed, initializing the data API proxy service to determine the target API interface.

[0020] Optionally, the modifying the initial configuration information to obtain the latest configuration information includes:

[0021] Building a test environment;

[0022] If the Mock switch is in the on state, modifying the latest configuration information;

[0023] If the Mock switch is in the off state, reloading the latest configuration information based on the proxy middleware.

[0024] Optionally, the determining the service location information corresponding to the original HTTP request includes:

[0025] Parsing the original HTTP request to extract parameter information;

[0026] Match the parameter information to find the corresponding processing rule;

[0027] Determine the request target corresponding to the original identifier according to the corresponding processing rule;

[0028] Allocate a target control component for the original HTTP request;

[0029] Parse the request target through the target control component, find the corresponding call service, and determine the service location information.

[0030] Optionally, generating a target data product according to the response result returned by the external downstream service includes:

[0031] Send the target HTTP request to the external downstream service through the proxy middleware to obtain the returned response result;

[0032] Process the response result through a second enhancement component to obtain a target data product;

[0033] Optionally, it further includes:

[0034] Analyze the target data product to obtain the analysis result of the data proxy; wherein, asynchronously send proxy information related to the original HTTP request through the proxy middleware; analyze the preset field values in the proxy information to generate the analysis result;

[0035] Monitor the data proxy based on the analysis result.

[0036] A data processing system based on a proxy middleware provided by the present application adopts the following technical solutions, including:

[0037] An embedding module, configured to obtain a business application and embed a proxy middleware into the business application;

[0038] An interface determination module, configured to read and load the latest configuration information to determine the target API interface of the proxy middleware;

[0039] The proxy middleware is configured to receive the original HTTP request of the target API interface; when the current mode is the first mode, perform data processing on the original HTTP request according to the first processing strategy to generate a target data product; return the target data product to the business application.

[0040] The proxy middleware includes:

[0041] A determination sub-module, configured to determine the service location information corresponding to the original HTTP request;

[0042] A search sub-module, configured to search for downstream information corresponding to an external downstream service in combination with the service location information;

[0043] A modification sub-module, configured to modify the original HTTP request based on the downstream information to obtain a target HTTP request;

[0044] A request sending sub-module, configured to send the target HTTP request to the external downstream service;

[0045] A data product generation module, configured to generate a target data product according to a response result returned by the external downstream service.

[0046] Optionally, the proxy middleware uses the Spring Boot microservice architecture as the underlying framework.

[0047] Optionally, the interface determination module includes:

[0048] An initial configuration sub-module, configured to obtain initial configuration information;

[0049] A configuration modification sub-module, configured to modify the initial configuration information to obtain the latest configuration information;

[0050] A publishing sub-module, configured to send the latest configuration information to a configuration management service based on a publishing instruction;

[0051] A reading sub-module, configured to read the latest configuration information of the proxy middleware from the configuration management service;

[0052] A configuration update sub-module, configured to update the configuration of the proxy middleware based on the latest configuration information;

[0053] An interface determination sub-module, configured to initialize a data API proxy service after the configuration update of the proxy middleware is completed, and determine the target API interface.

[0054] Optionally, the configuration modification sub-module includes:

[0055] Build a test environment;

[0056] A modification unit, configured to modify the latest configuration information if the Mock switch is in an on state;

[0057] A reload unit, configured to notify the proxy middleware to reload the latest configuration information if the Mock switch is in an off state.

[0058] Optionally, the determination sub-module includes:

[0059] A parsing unit, configured to parse the original HTTP request and extract parameter information;

[0060] A matching unit, configured to match the parameter information and find corresponding processing rules;

[0061] A target determination unit, configured to determine a request target corresponding to the original identifier according to the corresponding processing rules;

[0062] An allocation unit, configured to allocate a target control component for the original HTTP request;

[0063] A service determination unit, configured to parse the request target through the target control component, find a corresponding invoked service, and determine the service location information.

[0064] Optionally, the data product generation module includes:

[0065] A response return sub-module, configured to send the target HTTP request to the external downstream service through the proxy middleware and obtain the returned response result;

[0066] A second enhancement sub-module, configured to process the response result through a second enhancement component to obtain a target data product;

[0067] Optionally, it further includes: a monitoring module;

[0068] The monitoring module includes:

[0069] An analysis sub-module, configured to analyze the target data product to obtain an analysis result of the data proxy; wherein, proxy information related to the original HTTP request is asynchronously sent through the proxy middleware; preset field values in the proxy information are analyzed to generate the analysis result;

[0070] A monitoring sub-module, configured to monitor the data proxy based on the analysis result.

[0071] This specification also provides an electronic device, wherein the electronic device includes:

[0072] A processor; and,

[0073] A memory storing computer-executable instructions, and the executable instructions, when executed, cause the processor to execute any one of the above methods.

[0074] This specification also provides a computer-readable storage medium, wherein the computer-readable storage medium stores one or more programs, and the one or more programs, when executed by a processor, implement any one of the above methods.

[0075] The present specification also provides a computer program product, wherein the computer program product includes: a computer program / instructions, and when the computer program / instructions are executed by a processor, the above-mentioned method is implemented.

[0076] In the present application, by obtaining a business application, embedding a proxy middleware into the business application; reading and loading the latest configuration information to determine the target API interface of the proxy middleware, the scalability and flexibility of the system are improved; receiving the original HTTP request of the target API interface through the proxy middleware; when the current mode is the first mode, processing the data of the original HTTP request according to the first processing strategy through the proxy middleware to generate a target data product; wherein, determining the service location information corresponding to the original HTTP request; finding the downstream information corresponding to the external downstream service in combination with the service location information; modifying the original HTTP request based on the downstream information to obtain a target HTTP request; sending the target HTTP request to the external downstream service, effectively improving the accuracy and efficiency of data processing; generating a target data product according to the response result returned by the external downstream service; returning the target data product to the business application through the proxy middleware, enabling the business application to quickly obtain the processed target data product, thereby enhancing the response speed and user experience of the application. Description of the Drawings

[0077] Figure 1 It is a schematic diagram of the principle of a data processing method based on a proxy middleware provided by an embodiment of the present specification;

[0078] Figure 2 It is a schematic diagram of the process of a data processing method based on a proxy middleware provided by an embodiment of the present specification;

[0079] Figure 3 It is a schematic diagram of the execution logic of the proxy middleware of a data processing method based on a proxy middleware provided by an embodiment of the present specification;

[0080] Figure 4 It is a schematic diagram of the structure of a data processing system based on a proxy middleware provided by an embodiment of the present specification;

[0081] Figure 5 It is a schematic diagram of the structure of an electronic device provided by an embodiment of the present specification;

[0082] Figure 6 It is a schematic diagram of the principle of a computer-readable medium provided by an embodiment of the present specification. Detailed Embodiments

[0083] The following description is used to disclose the present invention so that those skilled in the art can implement the present invention. The preferred embodiments in the following description are only examples, and those skilled in the art can think of other obvious variations. The basic principles of the present invention defined in the following description can be applied to other embodiments, variations, improvements, equivalent solutions, and other technical solutions without departing from the spirit and scope of the present invention.

[0084] Exemplary embodiments of the present invention will now be described more fully with reference to the accompanying drawings. However, the exemplary embodiments can be implemented in various forms and should not be construed as limited to the embodiments set forth herein. On the contrary, providing these exemplary embodiments enables the present invention to be more comprehensive and complete, and more convenient to fully convey the inventive concept to those skilled in the art. The same reference numerals in the figures denote the same or similar elements, components, or parts, and thus repeated descriptions thereof will be omitted.

[0085] On the premise of conforming to the technical concept of the present invention, the features, structures, characteristics, or other details described in a specific embodiment do not exclude being combined in a suitable manner in one or more other embodiments.

[0086] In the description of specific embodiments, the features, structures, characteristics, or other details described in the present invention are for those skilled in the art to fully understand the embodiments. However, it does not exclude that those skilled in the art can practice the technical solutions of the present invention without one or more of the specific features, structures, characteristics, or other details.

[0087] The flowcharts shown in the accompanying drawings are only illustrative and do not necessarily include all the contents and operations / steps, nor do they necessarily have to be executed in the order described. For example, some operations / steps can be decomposed, while some operations / steps can be combined or partially combined, so the actual execution order may change according to the actual situation.

[0088] The block diagrams shown in the accompanying drawings are only functional entities and do not necessarily correspond to physically independent entities. That is, these functional entities can be implemented in software form, or in one or more hardware modules or integrated circuits, or in different networks and / or processor devices and / or micro-control component devices.

[0089] The term "and / or" or "and / or" includes all combinations of any one or more of the associated listed items.

[0090] Figure 1 A schematic diagram of the principle of a data processing method based on proxy middleware provided for the embodiments of this specification. The method includes:

[0091] S1 Obtain a business application and embed the proxy middleware into the business application;

[0092] S2 Read and load the latest configuration information to determine the target API interface of the proxy middleware;

[0093] S3 Receive the original HTTP request of the target API interface through the proxy middleware;

[0094] S4 When the current mode is the first mode, process the original HTTP request according to the first processing strategy through the proxy middleware to generate a target data product; specifically, determine the service location information corresponding to the original HTTP request; find the downstream information corresponding to the external downstream service in combination with the service location information; modify the original HTTP request based on the downstream information to obtain a target HTTP request; send the target HTTP request to the external downstream service; generate a target data product according to the response result returned by the external downstream service;

[0095] S5 Return the target data product to the business application through the proxy middleware.

[0096] With the development of information technology and the popularization of data middle platforms, data products emerge in an endless stream. In the process of developing data products, a microservices architecture is usually adopted. When building data products, programs often implement the proxy work for the data middle platform through code writing, which leads to an increasingly large number of API interfaces in the data middle platform. When the number of docking interfaces gradually increases and multiple people collaborate, data products become increasingly difficult to manage. Therefore, certain optimization work on configuration management is required.

[0097] Currently, the mainstream API proxy interface management products on the market are all developed based on the scenario of simulating data interfaces, such as YApi, APIPost, etc., to meet the problem that when the backend interface is still under development and cannot be called by the frontend. They can basically meet the requirements directly, but face the following problems:

[0098] (1) Related management products are generally a SAAS product or an independently running product, and cannot be well integrated with the current Java microservices architecture system based on SpringBoot.

[0099] (2) They only implement the function of interface simulation to manage the interfaces. However, due to the relatively weak function of their proxy services themselves, they cannot bear the large amount of traffic in the formal production link.

[0100] (3) Encapsulation for front - end calls cannot be well integrated with front - end applications, and additional implementation using a third - party call module is required. For the enhancement solutions for requests and responses, basically, solutions in JavaScript or Python are used for implementation. After being integrated into the microservice architecture, it will affect the execution performance of the code and is not compatible with the Spring Boot ecosystem itself.

[0101] (4) The implementation is relatively single. It has the ability to proxy data API services but does not have the ability to monitor data API service calls.

[0102] Based on this, the present invention constructs a data - processing method based on a proxy middleware, which is applied to a data proxy service established on the Spring Boot framework, as Figure 2 shown, specifically including:

[0103] S1 Obtain a business application and embed the proxy middleware into the business application;

[0104] S11 Construct the proxy middleware;

[0105] Specifically, with the Spring Boot microservice architecture as the underlying framework, a highly versatile Java SDK that can be embedded into business applications is developed as the proxy middleware. The proxy middleware includes: several control components and several enhancement components.

[0106] The proxy middleware supports interface simulation and has powerful proxy service functions, including dynamic update of API proxy configuration, service discovery, interface authentication, simulation interface switch, data API request - response enhancement component capabilities, and log analysis and recording capabilities, etc.

[0107] The proxy middleware of the present invention is a stable, powerful, extensible Java SDK that accesses the Spring Boot microservice ecosystem and has enterprise - level ecological data API proxy. Therefore, it can bear a large amount of traffic in the formal production process and provides rich management and enhancement functions.

[0108] The present invention splits the configuration and request execution processes. Instead of using the traditional centralized structure, after application integration using the SDK mode, the configuration is cached, reducing the performance degradation caused by database queries.

[0109] S12 Obtain the business application uploaded by the configurator through the first configuration page;

[0110] A configuration web - end is pre - constructed, and the configuration web - end includes, but is not limited to: the first configuration page, the second configuration page, etc.

[0111] The configurator can upload the application program package of the business application through the first configuration page. Based on the upload operation of the configurator on the configuration page, the application program package of the uploaded business application is obtained.

[0112] S13 Embed the proxy middleware into the business application;

[0113] Embed the same proxy middleware into different business applications. Of course, in the face of specific requirement differences, different proxy middleware can also be developed based on different business applications.

[0114] The proxy middleware will be responsible for communicating with the backend service of the business application.

[0115] S14 Build the basic environment.

[0116] Determine the basic environment of the business application (such as operating system, dependency library, etc.) to ensure that the proxy middleware can run properly.

[0117] S2 Read and load the latest configuration information to determine the target API interface of the proxy middleware;

[0118] S21 Obtain the initial configuration information;

[0119] Provide a second configuration page for the configurator to input configuration information.

[0120] In an embodiment of the present specification, the second configuration page includes several configuration windows. Each configuration window corresponds to a configuration item name and a configuration column. Among them, the configuration item name is preset and used to represent the attribute that needs to be configured; the configuration column is for the configurator to input the attribute value corresponding to the configuration item name to form the configuration information.

[0121] The configuration information includes: basic information of the API, preset proxy rules, external downstream service information, etc.

[0122] Take the configuration information input for the first time for the business application as the initial configuration information.

[0123] After the configurator inputs the configuration information corresponding to the business application on the second configuration page for the first time, obtain the configuration information input by the configurator and take it as the initial configuration information.

[0124] S22 Modify the initial configuration information to obtain the latest configuration information;

[0125] S221 Build a test environment;

[0126] Build a test environment to test the effectiveness of the configuration information.

[0127] If the Mock switch is in the on state, modify the latest configuration information;

[0128] When the on instruction of the Mock switch is obtained, it is determined that the Mock switch is in the on state, and the simulation interface editing mode is entered; in the simulation interface editing mode, the response data of the simulation interface input by the configuration personnel is obtained, the configuration information is modified, and the simulation interface is tested.

[0129] After the test is completed, the last modified configuration information is used as the latest configuration information;

[0130] S223 If the Mock switch is in the off state, reload the latest configuration information based on the proxy middleware.

[0131] When the off instruction of the Mock switch is obtained, it is determined that the Mock switch is in the off state. At this time, the proxy middleware is notified to automatically perform polling and reload the latest configuration information.

[0132] In the test environment, the present invention can be invented based on the actual development scenario of the data product, and the pain points of the simultaneous development of its downstream interfaces and external services are used to define the simulation interface. After the simulation interface is developed, it can be switched to the production interface with one key to complete the development work of the data proxy service.

[0133] In the above configuration process, the data proxy configuration provided by the present invention converts the part originally implemented by code into a mode of configuration through the interface, greatly reducing the R & D cost of the data product. The configuration personnel can complete the data proxy capability without developing the corresponding code or developing a small part of the enhanced code string.

[0134] S23 Based on the release instruction, send the latest configuration information to the configuration management service;

[0135] Based on the release instruction, push the latest configuration information to the production environment. Specifically, based on the save operation of the configuration personnel on the configuration web page, a save request is sent to the configuration management service, and the configuration management service obtains the latest configuration information and saves it into a specific database.

[0136] Among them, the configuration management service uses Springboot as the development framework, uses a relational database for data storage, and is compatible with mainstream databases. In an embodiment of the present specification, the databases compatible with the configuration management service include but are not limited to MySQL, Oracle, and DM, to meet the service requirements of different software middleware scenarios such as public cloud and Xinchuang cloud.

[0137] Since the data proxy itself is built on Spring Boot, it can be well integrated with the Java microservice architecture system based on Spring Boot. Based on this, the present invention can solve the problem that the mock products implemented through Node.js cannot be integrated into the microservice architecture system of Spring Boot.

[0138] The configuration management service includes multiple configuration modules. Specifically, the configuration management service includes: a configuration entry and management module, a configuration version control module, and a configuration release and rollback module.

[0139] Among them, the configuration entry and management module is used to provide an interface or interfaces to allow users to enter and manage complex configuration information, such as request addresses, request services, request parameters, request headers, request simulation parameters, and return strings, etc.

[0140] The configuration version control module is used to manage the versions of the configuration information, record the modification history of each configuration, and facilitate tracking and tracing back.

[0141] The configuration release and rollback module is used to support the release operation of the configuration, that is, to push the new or modified configuration information to the production environment; at the same time, it also supports the rollback operation of the configuration to restore the configuration to the previous version when necessary.

[0142] S24 reads the latest configuration information of the proxy middleware from the configuration management service;

[0143] S241 The proxy middleware uses the automatic loading mechanism in Spring Boot and initiates a read request to the configuration management service regularly after the automatic loading mechanism is started;

[0144] S242 After receiving the read request from the proxy middleware, the configuration management service retrieves the corresponding latest configuration information from the database and returns it to the proxy middleware;

[0145] S25 updates the configuration of the proxy middleware based on the latest configuration information;

[0146] Based on the scheduled task, the proxy middleware of each business application reads all the latest published configuration information and refreshes it into the cache of the proxy middleware so that the proxy middleware can communicate correctly with the backend service. Specifically:

[0147] After the proxy middleware reads and receives the latest configuration information through the data proxy management interface, it refreshes the latest configuration information into its own cache.

[0148] In one embodiment of this specification, the proxy middleware first parses the latest configuration information obtained from the configuration management service; based on the parsed configuration information, the proxy middleware establishes a mapping relationship that corresponds the API interfaces that the front-end or client may request (i.e., the interfaces defined in the configuration) to the actual interfaces of the back-end service. After the mapping relationship of the interfaces is established, the proxy middleware initializes the corresponding proxy logic. These logics are responsible for listening to the requests from the front-end or client and forwarding the requests to the correct back-end service according to the mapping relationship.

[0149] By refreshing the cache, the proxy middleware can ensure correct communication with the back-end service using the latest configuration information.

[0150] S26 After the configuration update of the proxy middleware is completed, initialize the data API proxy service and determine the target API interface.

[0151] After the proxy middleware successfully loads the latest configuration information, initialize the data API proxy service and determine a number of API interfaces.

[0152] The API interfaces are publicly provided for external services through a control component (such as a Controller in Spring Boot). When a request interface is accessed, load the API interface and perform proxy for the real service. During this process, the front-end or client can access the back-end service through these publicly provided API interfaces without directly knowing the details or location of the back-end service.

[0153] S3 Receive the original HTTP request of the target API interface through the proxy middleware;

[0154] When a user or business application (client) needs to interact with a certain service, it initiates an original HTTP request through the target API interface provided by the proxy middleware.

[0155] S4 Match the corresponding processing strategy based on the current mode.

[0156] The current mode is used to represent the on or off state of the simulation proxy. Specifically, the current mode can be one of the first mode or the second mode.

[0157] S41 When the current mode is the first mode, process the original HTTP request according to the first processing strategy through the proxy middleware to generate a target data product;

[0158] The first mode is the simulation proxy off mode.

[0159] When the current mode is the simulated proxy closed mode, the downstream interface of the target API interface will be directly called to return data. Both the entire request and the return can implement enhanced components for business customization processing operations to meet different requirements.

[0160] S411 Determine the service location information corresponding to the original HTTP request;

[0161] Receive the original HTTP request initiated by the business application, and determine the request target and the location information of the target service;

[0162] The original HTTP request includes an original identifier, original parameters, etc. The original identifier can be a Uniform Resource Identifier (URI).

[0163] After receiving the original HTTP request, the proxy middleware will identify the target of the request based on the original identifier of the original HTTP request. Specifically:

[0164] S411-1 Parse the original HTTP request to extract parameter information;

[0165] The parameter information includes: request method, request URL, request headers, etc.

[0166] S411-2 Match the parameter information to find the corresponding processing rule;

[0167] According to the preset proxy rules, the proxy middleware matches the extracted parameter information to determine the corresponding processing rule; among them, the proxy middleware loads the preset proxy rules, which define the processing rules corresponding to different URIs. Search for the processing rule corresponding to the original identifier from the preset proxy rules.

[0168] S411-3 Determine the request target corresponding to the original identifier according to the corresponding processing rule;

[0169] The request target can be a specific interface of the backend service, a database operation, a file path, etc., which is not limited here.

[0170] S411-4 Allocate a target control component for the original HTTP request;

[0171] In the proxy middleware, the control component (such as the Controller in Spring Boot) is responsible for receiving HTTP requests and allocating them to the corresponding processing logic according to the type and path of the requests.

[0172] Select a suitable target control component for the original HTTP request according to the working state of the control component. In an embodiment of this specification, the working states include: idle state, busy state, etc. That is, use the control component in the idle state as the target control component.

[0173] S411-5 Parse the request target through the target control component, find the corresponding call service, and determine the service location information.

[0174] After the target control component receives the original HTTP request, parse the original parameters in the original HTTP request; the original parameters include: request URL, request headers, request body, etc.

[0175] Cache the original parameters in the task process for subsequent matching and configuration.

[0176] Match the parameter information in the task process to find the data proxy API configuration corresponding to the parameter information.

[0177] Locate the service location information according to the current service discovery information and the current environment information in the data proxy API configuration.

[0178] Based on the latest configuration information of the proxy middleware, use input parameters such as service discovery name and environment to find the corresponding call service.

[0179] The service discovery information can be the unique identifier of the target service. The environment information includes: development environment, test environment, production environment.

[0180] The present invention provides a stable and extensible Java SDK as a proxy middleware to provide services externally through a unified control component (controller). In this way, the front-end application can interact with the back-end through standard HTTP requests, thus indirectly achieving good integration with the front-end application.

[0181] The front-end also provides an sdk for convenient calls for front-end development to use, but the front-end itself can also be called through the traditional http call method. This patent focuses on the back-end proxy, so it is not the key content and will not be elaborated too much.

[0182] S412 Combine the service location information to find the downstream information corresponding to the external downstream service;

[0183] Call the service discovery module, and according to the service location information and the current service discovery information, find the call IP port of the external downstream service as the downstream information.

[0184] S413 Modify the original HTTP request based on the downstream information to obtain the target HTTP request;

[0185] Before sending an HTTP request to the target service, the SDK may perform some request enhancement operations. Specifically, according to the order of enhancement components, the first enhancement component is determined; the first enhancement component is a pre-request enhancement component used to modify or enhance the original HTTP request.

[0186] Execute the first enhancement component to modify the original HTTP request based on downstream information to obtain the target HTTP request.

[0187] In an embodiment of this specification, modifying the original HTTP request may include adding authentication information, modifying request headers, compressing data, etc., so as to make the HTTP request more compliant with the requirements of the server or improve communication efficiency.

[0188] S414 Send the target HTTP request to the external downstream service;

[0189] S415 Generate a target data product according to the response result returned by the external downstream service;

[0190] After completing the above enhancement operations, the proxy middleware will send the target HTTP request to the corresponding target service. After the target service processes the target HTTP request, it will return the target data product to the proxy middleware. Specifically:

[0191] Send the target HTTP request to the external downstream service through the proxy middleware to obtain the returned response result; wherein, after receiving the target HTTP request, the external downstream service processes the target HTTP request to obtain a response result; the external downstream service returns the response result to the proxy middleware.

[0192] Before the proxy middleware returns the target data product to the business application (client), it may perform some return enhancement operations, such as decrypting data, decompressing data, formatting the return content, etc. These operations ensure that the data returned to the client is correct and easy to understand.

[0193] Therefore, process the response result through the second enhancement component to obtain the target data product;

[0194] Specifically, after obtaining the response content returned by the external downstream service, determine the second enhancement component through the proxy middleware according to the return enhancement component in the configuration. The second enhancement component is a post-request enhancement component.

[0195] Process according to the defined structure to obtain record information as the target data product;

[0196] The recorded information includes: the requested HTTP information, the requested user authentication information, the data proxy service information corresponding to the request, the real downstream request message, and the real downstream request return content.

[0197] S42 When the current mode is the second mode, data processing is performed based on the second policy.

[0198] The second mode is the simulation proxy enabled mode.

[0199] When the current mode is the simulation proxy enabled mode, the configured simulation data is directly returned and used as the target data product.

[0200] S5 The target data product is returned to the business application through the proxy middleware.

[0201] S6 Proxy analysis is performed in combination with the recorded information.

[0202] To facilitate proxy service analysis, the proxy middleware embedded in the business application is integrated with the ELK architecture.

[0203] When the proxy middleware returns the processed result to the business application, the proxy middleware will asynchronously send all the information of this request (including request content, response content, timestamp, etc.) to the log platform for monitoring and debugging to ensure the stability and performance of the service. Among them, asynchronous processing does not affect client request processing.

[0204] S61 Analyze the target data product to obtain the analysis result of the data proxy;

[0205] S611 Asynchronously send proxy information related to the business request through the proxy middleware; store the proxy information in ES;

[0206] Process the traffic information of the data proxy through a preset analysis structure to obtain a processing result;

[0207] Send the processing result to the log platform asynchronously through Kafka and store it in ElasticSearch.

[0208] The preset analysis structure has several fields, including: the requested HTTP information, the requested user authentication information, the data proxy service information corresponding to the request, the real downstream request message, and the real downstream request return content.

[0209] The requested HTTP information includes: request address, request body, request header, response body, response header, etc.

[0210] The gateway will verify the credentials carried by the user and obtain the corresponding user authentication information.

[0211] Request the corresponding data proxy service information to confirm which proxy rule the interaction request finally matches to.

[0212] The real downstream request messages include information such as IP, port, URI, etc.

[0213] S612 analyzes the preset field values in the proxy information through ES to generate the analysis result;

[0214] Through ElasticSearch, analyze according to specific fields such as application name, list of enhanced components executed, proxy service key, proxy service address, IP of the affiliated machine, data proxy interface URI, etc., to form a business report of data proxy as the analysis result.

[0215] After the analysis, output the final details and metric displays. The metric displays include functions such as proxy service call count statistics, proxy interface URI statistics, call exception statistics, etc.

[0216] S62 performs monitoring of data proxy based on the analysis result.

[0217] The present invention uses a mode of separating configuration management and data proxy, and cooperates with the mainstream ELK architecture of logs to achieve the ability of monitoring and analysis, laying a foundation for the page configuration and rapid online iteration of data products, and providing comprehensive support for the call monitoring of data API services.

[0218] As Figure 3 shown, the execution logic of the proxy middleware is briefly described below;

[0219] When a user requests a data product, send the original HTTP request to the request entry of the proxy middleware (proxy SDK); determine whether the proxy middleware contains the latest configuration information; if not, output an abnormal result and hand it over to manual processing; if so, obtain the latest configuration information; after configuration, call the service discovery module to find the call IP and port; then execute the pre-request enhanced component to modify / enhance the original HTTP request to generate a target HTTP request; after the external downstream service receives the target HTTP request, process the target HTTP request to obtain a response result; after executing the post-request enhanced component, return the obtained target data product to the business application; at the same time, asynchronously send logs to the log platform. After analysis, the request is returned to the page for effective monitoring.

[0220] Figure 4 It is a schematic structural diagram of a data processing system based on proxy middleware provided by an embodiment of this specification. The system includes:

[0221] An embedding module 410, configured to obtain a business application and embed the proxy middleware 430 into the business application;

[0222] An interface determination module 420, configured to read and load the latest configuration information and determine the target API interface of the proxy middleware 430;

[0223] The proxy middleware 430 is configured to receive the original HTTP request of the target API interface; when the current mode is the first mode, process the data of the original HTTP request according to the first processing strategy to generate a target data product; and return the target data product to the business application.

[0224] The proxy middleware 430 includes:

[0225] A determination sub-module, configured to determine the service location information corresponding to the original HTTP request;

[0226] A search sub-module, configured to search for the downstream information corresponding to the external downstream service in combination with the service location information;

[0227] A modification sub-module, configured to modify the original HTTP request based on the downstream information to obtain a target HTTP request;

[0228] A request sending sub-module, configured to send the target HTTP request to the external downstream service;

[0229] A data product generation module 440, configured to generate a target data product according to the response result returned by the external downstream service.

[0230] Optionally, the proxy middleware 430 uses the Spring Boot microservice architecture as the underlying framework.

[0231] Optionally, the interface determination module 420 includes:

[0232] An initial configuration sub-module, configured to obtain the initial configuration information;

[0233] A configuration modification sub-module, configured to modify the initial configuration information to obtain the latest configuration information;

[0234] A publishing sub-module, configured to send the latest configuration information to the configuration management service based on a publishing instruction;

[0235] A reading sub-module, configured to read the latest configuration information of the proxy middleware 430 from the configuration management service;

[0236] A configuration update sub-module, configured to update the configuration of the proxy middleware 430 based on the latest configuration information;

[0237] The interface determination sub-module is used to initialize the data API proxy service after the configuration update of the proxy middleware 430 is completed, and determine the target API interface.

[0238] Optionally, the configuration modification sub-module includes:

[0239] Build a test environment;

[0240] The modification unit is used to modify the latest configuration information if the Mock switch is in the on state;

[0241] The reload unit is used to notify the proxy middleware 430 to reload the latest configuration information if the Mock switch is in the off state.

[0242] Optionally, the determination sub-module includes:

[0243] The parsing unit is used to parse the original HTTP request and extract parameter information;

[0244] The matching unit is used to match the parameter information and find the corresponding processing rule;

[0245] The target determination unit is used to determine the request target corresponding to the original identifier according to the corresponding processing rule;

[0246] The allocation unit is used to allocate a target control component for the original HTTP request;

[0247] The service determination unit is used to parse the request target through the target control component, find the corresponding call service, and determine the service location information.

[0248] Optionally, the data product generation module 440 includes:

[0249] The response return sub-module is used to send the target HTTP request to the external downstream service through the proxy middleware 430 and obtain the returned response result;

[0250] The second enhancement sub-module is used to process the response result through the second enhancement component to obtain the target data product;

[0251] Optionally, it further includes: a monitoring module;

[0252] The monitoring module includes:

[0253] An analysis sub-module for analyzing the target data product to obtain an analysis result of the data proxy; wherein, proxy information related to the original HTTP request is asynchronously sent through the proxy middleware 430; preset field values in the proxy information are analyzed to generate the analysis result;

[0254] A monitoring sub-module for monitoring the data proxy based on the analysis result.

[0255] The functions of the system according to the embodiments of the present invention have been described in the above method embodiments. Therefore, for the details not elaborated in the description of this embodiment, reference can be made to the relevant descriptions in the foregoing embodiments, and no further elaboration will be provided here.

[0256] Those skilled in the art should understand that the embodiments of the present invention can be provided as a method, a system, or a computer program product. Therefore, the present invention can take the form of a completely hardware embodiment, a completely software embodiment, or an embodiment combining software and hardware aspects. Moreover, the present invention can take the form of a computer program product implemented on one or more computer-usable storage media (including but not limited to disk storage, CD-ROM, optical storage, etc.) containing computer-usable program code.

[0257] Next, an embodiment of the electronic device of the present invention is described. This electronic device can be regarded as an implementation form of the entity of the above method and device embodiments of the present invention. For the details described in the embodiment of the electronic device of the present invention, they should be regarded as a supplement to the above method or device embodiments; for the details not disclosed in the embodiment of the electronic device of the present invention, reference can be made to the above method or device embodiments to implement.

[0258] Figure 5 It is a schematic structural diagram of an electronic device provided for the embodiments of this specification. Figure 5 The computer device shown is only an example and should not impose any limitations on the functions and usage scope of the embodiments of the present invention.

[0259] As Figure 5 shown, the computer device 500 of this exemplary embodiment is presented in the form of a general-purpose data processing device. The components of the computer device 500 may include but are not limited to: at least one processor 510, at least one memory 520, a network interface 530, a display unit 540, an input component 550, etc.

[0260] Among them, the memory 520 stores a computer-readable program, which can be the source program or the code of a read-only program. The program can be executed by the processor element 510, so that the processor 510 executes the steps of various embodiments of the present invention. For example, the processor 510 can execute the steps as Figure 1 shown.

[0261] The memory 520 may include a readable medium in the form of volatile storage units, such as random access storage units (RAM) and / or cache storage units, and may further include a read-only storage unit (ROM). The memory 520 may also include programs / utilities having a set (at least one) of program modules, such program modules including but not limited to: an operating system, one or more application programs, other program modules, and program data, each or some combination of these examples may include an implementation of a network environment.

[0262] Also included is a bus (not shown) that may represent one or more of several types of bus structures, including a memory bus or memory control component, a peripheral bus, an accelerated graphics port, a processing unit, or a local bus using any of a variety of bus structures.

[0263] The computer device 500 may also communicate with one or more external devices (such as a keyboard, a display, a network device, a Bluetooth device, etc.), such that a user can interact with the computer device 500 via these external devices, and / or such that the computer device 500 can communicate with one or more other data processing devices (such as a router, a modem, etc.). Such communication may be carried out through the network interface 530, and may also be through a network adapter with one or more networks (such as a local area network (LAN), a wide area network (WAN), and / or a public network, such as the Internet). The network adapter may communicate with other modules of the computer device 500 through a bus. It should be understood that although not shown in the figure, other hardware and / or software modules may be used in the computer device 500, including but not limited to: microcode, device drivers, redundant processing units, external disk drive arrays, RAID systems, tape drives, and data backup storage systems, etc.

[0264] Figure 6 is a schematic diagram of an embodiment of a computer-readable medium of the present invention. As Figure 6As shown, the computer program can be stored on one or more computer-readable media. The computer-readable media can be a readable signal medium or a readable storage medium. The readable storage medium can be, for example, but not limited to, an electrical, magnetic, optical, electromagnetic, infrared, or semiconductor system, apparatus, or device, or any combination of the above. More specific examples (a non-exhaustive list) of the readable storage medium include: an electrical connection having one or more wires, a portable disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the above. When the computer program is executed by one or more data processing devices, the computer-readable medium can implement the above method of the present invention.

[0265] Through the description of the above embodiments, those skilled in the art can easily understand that the exemplary embodiments described in the present invention can be implemented by software, or by a combination of software and necessary hardware. Therefore, the technical solutions according to the embodiments of the present invention can be embodied in the form of a software product, which can be stored in a computer-readable storage medium (which can be a CD-ROM, a USB flash drive, a mobile hard disk, etc.) or on a network, including several instructions to enable a data processing device (which can be a personal computer, a server, or a network device, etc.) to execute the above method according to the present invention.

[0266] The computer-readable storage medium can include a data signal propagated in a baseband or as part of a carrier wave, in which the readable program code is carried. Such a propagated data signal can take various forms, including but not limited to electromagnetic signals, optical signals, or any suitable combination of the above. The readable storage medium can also be any readable medium other than the readable storage medium, which can send, propagate, or transmit a program for use by or in conjunction with an instruction execution system, apparatus, or device. The program code contained on the readable storage medium can be transmitted by any appropriate medium, including but not limited to wireless, wired, optical fiber, RF, etc., or any suitable combination of the above.

[0267] The program code for performing the operations of the present invention can be written in any combination of one or more programming languages, including object-oriented programming languages such as Java, C++, etc., and also including conventional procedural programming languages such as the "C" language or similar programming languages. The program code can be executed entirely on the user's computing device, partially on the user's device, executed as a stand-alone software package, partially on the user's computing device and partially on a remote computing device, or entirely on a remote computing device or server. In the case of a remote computing device, the remote computing device can be connected to the user's computing device through any type of network, including a local area network (LAN) or a wide area network (WAN), or can be connected to an external computing device (e.g., by connecting through the Internet using an Internet service provider).

[0268] In summary, the present invention can be implemented by a method, apparatus, electronic device, or computer-readable medium for executing a computer program. Some or all of the functions of the present invention can be implemented using a general-purpose data processing device such as a microprocessor or a digital signal processor (DSP) in practice.

[0269] The specific embodiments described above further elaborate on the purpose, technical solution, and beneficial effects of the present invention. It should be understood that the present invention is not inherently related to any specific computer, virtual device, or electronic device, and various general-purpose devices can also implement the present invention. The above are only specific embodiments of the present invention and are not used to limit the present invention. Any modifications, equivalent replacements, improvements, etc. made within the spirit and principle of the present invention shall be included within the protection scope of the present invention.

Claims

1. A data processing method based on proxy middleware, characterized in that: include: Obtaining a business application, and embedding the proxy middleware into the business application; Read and load the latest configuration information to determine the target API interface of the proxy middleware; Receive the original HTTP request of the target API interface through the proxy middleware; When the current mode is the first mode, the proxy middleware performs data processing on the original HTTP request according to the first processing strategy to generate a target data product; wherein, the service location information corresponding to the original HTTP request is determined; the downstream information corresponding to the external downstream service is searched in combination with the service location information; the original HTTP request is modified based on the downstream information to obtain a target HTTP request; the target HTTP request is sent to the external downstream service; and the target data product is generated according to the response result returned by the external downstream service; The target data product is returned to the business application through the proxy middleware.

2. A data processing method based on proxy middleware as claimed in claim 1, characterized in that: The proxy middleware uses the SpringBoot microservice architecture as the underlying framework.

3. A data processing method based on proxy middleware as claimed in claim 1, characterized in that: The reading and loading of the latest configuration information and determining the target API interface of the proxy middleware include: Get initial configuration information; Modifying the initial configuration information to obtain the latest configuration information; Based on the release instruction, sending the latest configuration information to the configuration management service; Reading the latest configuration information of the proxy middleware from the configuration management service; Based on the latest configuration information, update the configuration of the proxy middleware; After the proxy middleware configuration is updated, the data API proxy service is initialized to determine the target API interface.

4. A data processing method based on proxy middleware as claimed in claim 3, characterized in that: The modifying of the initial configuration information to obtain the latest configuration information includes: Build a test environment; If the Mock switch is on, modify the latest configuration information; If the Mock switch is in the off state, the latest configuration information is reloaded based on the proxy middleware.

5. A data processing method based on proxy middleware as claimed in claim 1, characterized in that: The determining the service location information corresponding to the original HTTP request includes: Parsing the original HTTP request to extract parameter information; Match the parameter information and search for corresponding processing rules; Determine the request target corresponding to the original identifier according to the corresponding processing rule; Assigning a target control component to the original HTTP request; The target control component parses the request target, searches for the corresponding calling service, and determines the service location information.

6. A data processing method based on proxy middleware as claimed in claim 1, characterized in that: Generating a target data product according to a response result returned by the external downstream service includes: Sending the target HTTP request to the external downstream service through the proxy middleware, and obtaining the returned response result; The response result is processed by the second enhancement component to obtain a target data product.

7. A data processing method based on proxy middleware as claimed in claim 1, characterized in that: Also includes: Analyze the target data product to obtain the analysis result of the data proxy; wherein the proxy information related to the original HTTP request is asynchronously sent through the proxy middleware; analyze the preset field value in the proxy information to generate the analysis result; The data agent is monitored based on the analysis result.

8. A data processing system based on proxy middleware, characterized in that: include: An embedding module, used to obtain a business application and embed the proxy middleware into the business application; An interface determination module, used to read and load the latest configuration information and determine the target API interface of the proxy middleware; The proxy middleware is used to receive the original HTTP request of the target API interface; when the current mode is the first mode, the original HTTP request is processed according to the first processing strategy to generate the target data product; The target data product is returned to the business application. The proxy middleware includes: A determination submodule, used to determine the service location information corresponding to the original HTTP request; A search submodule, used to search for downstream information corresponding to an external downstream service in combination with the service location information; A modification submodule, used for modifying the original HTTP request based on the downstream information to obtain a target HTTP request; A request sending submodule, used for sending the target HTTP request to the external downstream service; The data product generation module is used to generate a target data product according to the response result returned by the external downstream service.

9. An electronic device, wherein: The electronic device includes: processor; and, A memory storing computer executable instructions which, when executed, cause the processor to perform a method according to any one of claims 1-7.

10. A computer-readable storage medium, wherein: The computer-readable storage medium stores one or more programs, and when the one or more programs are executed by a processor, the method of any one of claims 1 to 7 is implemented.

Images