SCA data retrieval method and device based on dynamic proxy

Abstract

The present invention discloses a SCA data retrieval method and device based on dynamic proxy, wherein the retrieval method includes: transferring data source information into an SCA knowledge base for storage; setting a dynamic proxy entry to proxy all interfaces currently receiving retrieval services; selecting the optimal data source information in the current Java container and SCA knowledge base according to the retrieval service received by the dynamic proxy entry; calling the corresponding data source according to the selected optimal data source information; searching the called data source according to the retrieval service, and returning the retrieval result. The SCA data retrieval method based on dynamic proxy of the present invention realizes the load distribution of each request at the business level through dynamic proxy, and can selectively switch multiple data sources, quickly adapt to the user's on-site needs, effectively improve the disaster tolerance and load performance of the SCA knowledge base, and enhance the overall robustness of SCA.

Description

Technical Field

[0001] The present invention relates to the technical field of SCA data retrieval, and in particular to a SCA data retrieval method and device based on dynamic proxy. Background Art

[0002] In the existing SCA operation and maintenance process, some customers have the need for multi-machine deployment. The common multi-machine deployment solution on the market is achieved through distributed deployment, which requires changing the single-machine table structure of all Clickhouses to a distributed table structure. This will undoubtedly bring too much extra workload, and the risk of data failure caused by the change operation is high.

[0003] Currently, most distributed deployments for the storage layer on the market rely on third-party components such as zookeeper and nginx. Third-party components are used to load balance related table creation and change them to distributed table structures. In addition, data needs to be segmented, saved, and backed up. The process is cumbersome and cannot be dynamically adjusted for complex user sites, resulting in low controllability. Summary of the invention

[0004] The purpose of the present invention is to provide an SCA data retrieval method and device based on dynamic proxy, which realizes the load distribution of each request at the business level through dynamic proxy, and can selectively switch multiple data sources to quickly adapt to the user's on-site needs, effectively improve the disaster tolerance and load performance of the SCA knowledge base, and enhance the overall robustness of SCA.

[0005] In order to achieve the above object, the present invention discloses a SCA data retrieval method based on dynamic proxy, which comprises:

[0006] Transfer data source information to the SCA knowledge base for storage;

[0007] Set up a dynamic proxy entry to proxy all current interfaces that receive retrieval services;

[0008] Selecting the optimal data source information in the current Java container and the SCA knowledge base according to the search service received by the dynamic proxy entry;

[0009] Call the corresponding data source according to the selected optimal data source information;

[0010] The called data source is searched according to the search service, and the search result is returned.

[0011] Furthermore, after the “searching the called data source according to the search service and returning the search result”, it also includes:

[0012] Scan all current data sources and determine their survivability;

[0013] Perform data upgrade on all data sources that are judged to be alive.

[0014] Furthermore, the search information of each of the search services includes a search data source name, a proxy object and input parameters.

[0015] Further, the retrieval information of the retrieval service includes a retrieval data source name, and the "selecting the optimal data source information in the current Java container and the SCA knowledge base according to the retrieval service received by the dynamic proxy entry" includes:

[0016] According to the retrieved data source name, it is determined whether the corresponding data source information in the current Java container is alive. If it is alive, it is used as the optimal data source information. If it is not alive, the data source information in the current Java container is cleared, and then the data source information in the SCA knowledge base is retrieved, and the optimal data source information is selected.

[0017] Furthermore, the “selecting the best data source information” includes:

[0018] Performing a survival check on the corresponding data source based on the retrieved data source information;

[0019] Perform load check on data sources that pass the survival check;

[0020] The data source information corresponding to the data source with the least load in the load verification is used as the optimal data source information.

[0021] Further, the search information of the search service includes input parameters, and the “searching the called data source according to the search service” includes:

[0022] Determine whether the input parameter is a batch data retrieval;

[0023] If it is not a batch data search, the called data source is searched according to the input parameters;

[0024] If it is a batch data search, the input parameters are split, and data search is performed in batches according to the split input parameters. Then, multiple search data obtained by the batch search are merged and used as the search result.

[0025] Further, the “determining whether the input parameter is a batch data search” includes:

[0026] It is determined whether the number of retrieved items of the input parameter is greater than a set threshold. If so, it is determined to be a batch data retrieval.

[0027] In order to achieve the above object, the present invention discloses a SCA data retrieval device based on dynamic proxy, which comprises:

[0028] The storage module is used to transfer the data source information to the SCA knowledge base for storage;

[0029] The setting module is used to set the dynamic proxy entry to proxy all interfaces currently receiving retrieval services;

[0030] A selection module, used to select the optimal data source information in the current Java container and the SCA knowledge base according to the search service received by the dynamic proxy entry;

[0031] A calling module, used to call the corresponding data source according to the selected optimal data source information;

[0032] The search module is used to search the called data source according to the search service and return the search result.

[0033] In this application, the load distribution of each data retrieval request is realized at the business level through dynamic proxy. First, the data source information needs to be passed into the SCA knowledge base for storage, and then a dynamic proxy entry is set to proxy all the interfaces currently receiving the retrieval business, so as to select the best data source information in the current Java container and SCA knowledge base according to the received retrieval business, and call the corresponding data source, and then search the data source according to the retrieval business, so as to return the retrieval results to the caller. Through dynamic proxy and preferential switching of multiple data sources, it can quickly adapt to the user's on-site needs, effectively improve the disaster tolerance and load performance of the SCA knowledge base, and enhance the overall robustness of SCA. BRIEF DESCRIPTION OF THE DRAWINGS

[0034] Figure 1 The figure is a flow chart of an SCA data retrieval method based on dynamic proxy according to an embodiment of the present invention.

[0035] Figure 2 It is a module diagram of an SCA data retrieval device based on dynamic proxy according to an embodiment of the present invention.

[0036] Figure 3 4 is a system diagram of an electronic device according to an embodiment of the present invention. DETAILED DESCRIPTION

[0037] In order to explain the technical content, structural features, achieved objectives and effects of the present invention in detail, the following is a detailed description in conjunction with the implementation methods and the accompanying drawings.

[0038] Embodiment 1

[0039] See also Figure 1The present invention discloses a SCA data retrieval method based on dynamic proxy, which comprises:

[0040] 101. The data source information is transferred to the SCA knowledge base for storage;

[0041] It can be understood that the data source information includes at least one data source parameter configuration, and multiple data source configuration parsing is added to the SCA-SDK initialization parameters so that when the caller calls the initialization function, the data source parameter configuration is dynamically passed in for storage at the SDK data retrieval level encapsulated by the SCA knowledge base. If the customer deploys multiple databases at the same time, multiple data source information needs to be passed in to facilitate calling it for retrieval after the data source is switched later, and it is not limited to this.

[0042] 102. Set a dynamic proxy entry to proxy all interfaces currently receiving retrieval services;

[0043] By unifying the proxy of business interfaces, the need to modify all interfaces is avoided, which effectively improves the security of SCA. In addition, dynamic proxies can be extracted separately as public methods, which are convenient for other projects to use directly. It is easy to use and helps improve work efficiency.

[0044] Furthermore, the search information of each search service includes the search data source name, the proxy object and the input parameters.

[0045] It is understandable that a dynamic proxy entry is defined to proxy all business interfaces for data retrieval in the current SDK to achieve isolation at the business level. Specifically, the name of the data source is retrieved as the name of the main method to be executed. The set dynamic proxy will also record the execution time of each method to facilitate the later troubleshooting of performance issues, and is not limited to this.

[0046] 103. Select the optimal data source information in the current Java container and SCA knowledge base according to the search service received by the dynamic proxy entry;

[0047] Furthermore, the retrieval information of the retrieval service includes the retrieval data source name, and "selecting the optimal data source information in the current Java container and SCA knowledge base according to the retrieval service received by the dynamic proxy entry" includes:

[0048] 1031. Determine whether the corresponding data source information in the current Java container is alive based on the retrieved data source name. If so, use it as the optimal data source information. If not, clear the data source information in the current Java container, then retrieve the data source information in the SCA knowledge base, and select the optimal data source information.

[0049] Without changing the database table structure, etc., multiple data sources can be switched and load detected through components such as Mybatis-flex, effectively improving the disaster tolerance and robustness of SCA.

[0050] Furthermore, “selecting the best data source information” includes:

[0051] 1032. Perform a survival check on the corresponding data source according to the retrieved data source information;

[0052] 1033. Perform load check on the data source that has passed the survival check;

[0053] 1034. The data source information corresponding to the data source with the least load in the load verification is used as the optimal data source information.

[0054] Before each actual method is executed, the corresponding data source is tested for survival and performance, which is beneficial to improving the overall load performance and disaster tolerance of SCA.

[0055] It can be understood that before executing the actual method, it will be determined whether the data source information in the current container is alive. If the data source information is alive, the corresponding method will be directly executed to call the corresponding data source for retrieval. If the data source information is not alive, the existing data source information in the current Java container will be cleared using the DataSourceKey.clear method in Mybatis-flex, and then the relevant preset data source parameter configuration during initialization will be retrieved. The data source will be checked for survival based on the incoming data source information, as well as the number of data source background task executions (load check), and then the optimal data source information will be selected for return, so that the corresponding data source can be called for data retrieval based on the input parameters in the future, and this is not limited to this.

[0056] 104. Calling a corresponding data source according to the selected optimal data source information;

[0057] 105. Search the called data source according to the search service and return the search result.

[0058] Furthermore, the search information of the search service includes input parameters, and “searching the called data source according to the search service” includes:

[0059] 1051. Determine whether the input parameter is for batch data retrieval;

[0060] If it is not a batch data retrieval, the called data source is retrieved according to the input parameters;

[0061] If it is a batch data search, the input parameters are split, and data search is performed in batches according to the split input parameters. Then, multiple search data obtained by the batch search are merged and used as the search result.

[0062] Furthermore, “determining whether the input parameter is a batch data retrieval” includes:

[0063] 1052. Determine whether the number of retrieved items of the input parameter is greater than a set threshold. If so, determine it as a batch data retrieval.

[0064] It can be understood that whether it is a batch data retrieval is determined by judging whether the number of retrieval items in the input parameter of the current method to be executed exceeds the set threshold. If the input parameter is a batch data retrieval, the request parameter is segmented according to the set threshold, and data is retrieved in batches using multiple threads. After the retrieval is completed, the results are merged and returned to the caller.

[0065] Furthermore, after “searching the called data source according to the search service and returning the search result”, it also includes:

[0066] 106. Scan all current data sources and determine their corresponding viability;

[0067] 107. Upgrade the data of all data sources that are judged to be alive.

[0068] It can be understood that operations 106 and 107 are only performed when the SCA knowledge base is upgraded, and do not need to be performed during normal data retrieval.

[0069] In this application, the load distribution of each data retrieval request is realized at the business level through dynamic proxy. First, the data source information needs to be passed into the SCA knowledge base for storage, and then a dynamic proxy entry is set to proxy all the interfaces currently receiving the retrieval business, so as to select the best data source information in the current Java container and SCA knowledge base according to the received retrieval business, and call the corresponding data source, and then search the data source according to the retrieval business, so as to return the retrieval results to the caller. Through dynamic proxy and preferential switching of multiple data sources, it can quickly adapt to the user's on-site needs, effectively improve the disaster tolerance and load performance of the SCA knowledge base, and enhance the overall robustness of SCA.

[0070] Embodiment 2

[0071] See also Figure 1 and Figure 2 The present invention discloses a SCA data retrieval device based on dynamic proxy, which comprises:

[0072] Storage module 201, used to transfer data source information to the SCA knowledge base for storage;

[0073] A setting module 202, used to set a dynamic proxy entry to proxy all interfaces currently receiving retrieval services;

[0074] The selection module 203 is used to select the optimal data source information in the current Java container and SCA knowledge base according to the search service received by the dynamic proxy entry;

[0075] A calling module 204 is used to call a corresponding data source according to the selected optimal data source information;

[0076] The search module 205 is used to search the called data source according to the search service and return the search result.

[0077] Embodiment 3

[0078] See also Figure 1 and Figure 3 , the present invention discloses an electronic device, comprising:

[0079] One or more processors 301;

[0080] The one or more memories 302 are used to store one or more programs. When the one or more programs are executed by the processor, the processor implements the aforementioned SCA data retrieval method based on dynamic proxy.

[0081] Embodiment 4

[0082] The embodiment of the present application discloses a computer-readable storage medium on which a program is stored. When the program is executed by a processor, the aforementioned SCA data retrieval method based on dynamic proxy is implemented.

[0083] Embodiment 5

[0084] The embodiment of the present application discloses a computer program product or a computer program, which includes computer instructions stored in a computer-readable storage medium. A processor of an electronic device reads the computer instructions from the computer-readable storage medium, and the processor executes the computer instructions, so that the electronic device executes the above-mentioned SCA data retrieval method based on a dynamic agent.

[0085] It should be understood that in the embodiments of the present application, the processor referred to may be a central processing unit (CPU), and the processor may also be other general-purpose processors, digital signal processors (DSP), application-specific integrated circuits (ASIC), field-programmable gate arrays (FPGA) or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc. A general-purpose processor may be a microprocessor or the processor may also be any conventional processor, etc.

[0086] Those skilled in the art can understand that all or part of the processes in the above-mentioned embodiments can be implemented by hardware related to computer program instructions, and the program can be stored in a computer-readable storage medium. When the program is executed, it can include the processes of the embodiments of the above-mentioned methods. The storage medium can be a disk, an optical disk, a read-only memory (ROM) or a random access memory (RAM).

[0087] The above disclosure is only the preferred embodiment of the present invention, which certainly cannot be used to limit the scope of rights of the present invention. Therefore, equivalent changes made according to the scope of the patent application of the present invention are still within the scope covered by the present invention.

Claims

1. A SCA data retrieval method based on dynamic proxy, characterized in that: include: Transfer data source information to the SCA knowledge base for storage; Set up a dynamic proxy entry to proxy all current interfaces that receive retrieval services; Selecting the optimal data source information in the current Java container and the SCA knowledge base according to the search service received by the dynamic proxy entry; Call the corresponding data source according to the selected optimal data source information; The called data source is searched according to the search service, and the search result is returned.

2. The SCA data retrieval method based on dynamic proxy according to claim 1 is characterized in that: After the "searching the called data source according to the search service and returning the search result", it also includes: Scan all current data sources and determine their survivability; Perform data upgrade on all data sources that are judged to be alive.

3. The SCA data retrieval method based on dynamic proxy according to claim 1 is characterized in that: The retrieval information of each retrieval service includes a retrieval data source name, a proxy object and input parameters.

4. The SCA data retrieval method based on dynamic proxy according to claim 1 is characterized in that: The retrieval information of the retrieval service includes the retrieval data source name, and the "selecting the current Java container and the optimal data source information in the SCA knowledge base according to the retrieval service received by the dynamic proxy entry" includes: According to the retrieved data source name, it is determined whether the corresponding data source information in the current Java container is alive. If it is alive, it is used as the optimal data source information. If it is not alive, the data source information in the current Java container is cleared, and then the data source information in the SCA knowledge base is retrieved, and the optimal data source information is selected.

5. The SCA data retrieval method based on dynamic proxy according to claim 4 is characterized in that: The “selecting the best data source information” includes: Performing a survival check on the corresponding data source based on the retrieved data source information; Perform load check on data sources that pass the survival check; The data source information corresponding to the data source with the least load in the load verification is used as the optimal data source information.

6. The SCA data retrieval method based on dynamic proxy according to claim 1 is characterized in that: The retrieval information of the retrieval service includes input parameters, and the "searching the called data source according to the retrieval service" includes: Determine whether the input parameter is a batch data retrieval; If it is not a batch data search, the called data source is searched according to the input parameters; If it is a batch data search, the input parameters are split, and data search is performed in batches according to the split input parameters. Then, multiple search data obtained by the batch search are merged and used as the search result.

7. The SCA data retrieval method based on dynamic proxy according to claim 6 is characterized in that: The “determining whether the input parameter is a batch data retrieval” includes: It is determined whether the number of retrieved items of the input parameter is greater than a set threshold. If so, it is determined to be a batch data retrieval.

8. A SCA data retrieval device based on dynamic proxy, characterized in that: include: The storage module is used to transfer the data source information to the SCA knowledge base for storage; The setting module is used to set the dynamic proxy entry to proxy all interfaces currently receiving retrieval services; A selection module, used to select the optimal data source information in the current Java container and the SCA knowledge base according to the search service received by the dynamic proxy entry; A calling module, used to call the corresponding data source according to the selected optimal data source information; The search module is used to search the called data source according to the search service and return the search result.

9. An electronic device, characterized in that: include: one or more processors; One or more memories are used to store one or more programs. When one or more of the programs are executed by the processor, the processor implements the dynamic proxy-based SCA data retrieval method as described in any one of claims 1 to 7.

10. A computer-readable storage medium having a program stored thereon, characterized in that: When the program is executed by a processor, the SCA data retrieval method based on dynamic proxy as described in any one of claims 1 to 7 is implemented.

Images