Methods, devices, equipment and storage media for accelerating cross-server data calls

Abstract

This application belongs to the field of R&D optimization technology and is applied to the R&D optimization process in the financial industry. It relates to a method, apparatus, device, and storage medium for accelerating cross-server data retrieval. The method includes using the Spark search engine as a retrieval engine, retrieving insurance business data required by each insurance branch to complete its business execution logic from the data source table of the financial data warehouse based on the retrieval fields input by the target user; inputting the required business data into the corresponding target data sub-table and sending it to a preset second data server. By introducing the Spark search engine, Lexica keyword search engine, and AI2SQL platform automatic generation components, the method automatically generates external forms corresponding to each insurance branch, using the Hive public data warehouse data of the insurance head office as the source data. This improves the efficiency of data retrieval from the financial data warehouse and also enables data separation on a per-branch basis.

Description

Technical Field

[0001] This application relates to the field of R&D optimization technology and is applied to the R&D optimization process in the financial industry. In particular, it relates to a method, device, equipment and storage medium for accelerating data calls between servers. Background Technology

[0002] With the development of the financial industry, financial data warehouses often contain multiple summary reports. To save on the resource consumption of financial data warehouses, taking insurance business as an example, insurance head offices and branches often share a single financial data warehouse. If the head office and all branches establish direct access relationships with this financial data warehouse, it will lead to significant resource consumption for data access.

[0003] Currently, to address the issue of high resource consumption in data retrieval caused by insurance headquarters and all branches establishing direct communication with financial data warehouses, existing technologies utilize Hive data warehouses to maintain insurance business data at the insurance headquarters. Based on whether access control is required for insurance business data, partitioned and non-partitioned tables are designed. Data flow views suitable for each insurance branch are then designed based on these partitioned and non-partitioned tables, guiding each branch to retrieve relevant insurance business data from the Hive data warehouse. While this method improves data retrieval speed to some extent, it still fails to solve the problem of branches directly accessing relevant business data from the financial data warehouse. Therefore, achieving data separation for insurance business operations and improving the efficiency of data retrieval by each branch has become a pressing technical challenge. Summary of the Invention

[0004] The purpose of this application is to propose a method, apparatus, device and storage medium for accelerating cross-server data calls, so as to improve the efficiency of insurance branch companies in calling insurance business data, and at the same time, realize the data separation of business data corresponding to different insurance branch companies.

[0005] To address the aforementioned technical problems, this application provides a method for accelerating cross-server data calls, employing the following technical solution:

[0006] A method for accelerating cross-server data calls includes the following steps:

[0007] Obtain each non-partitioned table and each partitioned table pre-built in the target Hive data warehouse as a data source table, wherein the target Hive data warehouse is set in a first data server, and the first data server is used to cache the data source tables;

[0008] Run a pre-deployed Spark search engine;

[0009] Using the business execution logic of each branch as the computation logic and the Spark search engine as the retrieval engine, the business data required by each branch to complete its business execution logic is obtained from the data source table based on the retrieval fields input by the target user.

[0010] Data tables are constructed for each branch company. The business data required by each branch company to complete its business execution logic is entered into the target data table. The target data table is a data table that has a one-to-one correspondence with each branch company.

[0011] The target data tables corresponding to each branch are sent to a preset second data server, which is deployed in each branch to cache the corresponding target data tables.

[0012] Furthermore, before executing the step of using the business execution logic of each branch as the computational logic and the Spark search engine as the retrieval engine, the method further includes:

[0013] According to the preset component first call interface, the pre-deployed Lexica prompt word search engine is called, wherein the Lexica prompt word search engine is trained using the GPT-3 autoregressive language model;

[0014] Run the pre-deployed Lexica suggestion search engine;

[0015] Based on the preset second component call interface, the pre-deployed AI2SQL platform is called to automatically generate components;

[0016] Run the pre-deployed AI2SQL auto-generation component.

[0017] Furthermore, before executing the step of invoking the pre-deployed Lexicon search engine according to the preset component first call interface, the method further includes:

[0018] The Lexica prompt word search engine was trained in advance using data from the data source table as training data and the GPT-3 autoregressive language model as the training model.

[0019] Furthermore, the step of training the Lexica suggestion word search engine using the data in the data source table as training data and the GPT-3 autoregressive language model as the training model specifically includes:

[0020] Step 301: Input the training data into the GPT-3 autoregressive language model to perform autoregressive training and obtain the model output results;

[0021] Step 302: Calculate the similarity between the model output result and the preset model output dictionary according to the preset similarity algorithm;

[0022] Step 303: Determine whether the similarity meets the preset similarity threshold;

[0023] Step 304: If the similarity meets the preset similarity threshold, the Lexica prompt word search engine training is complete and training is terminated;

[0024] Step 305: If the similarity does not meet the preset similarity threshold, adjust the regression parameters of the GPT-3 autoregressive language model and repeat steps 301 to 305.

[0025] Furthermore, the step of retrieving the business data required by each branch office to complete its business execution logic from the data source table based on the search fields input by the target user specifically includes:

[0026] Obtain the search fields input by the target user;

[0027] Based on the Lexica suggestion word search engine, the relevant word sequence corresponding to the search field is recommended and displayed on the target display interface;

[0028] In response to the operation instruction executed by the target user on the relevant word sequence, the relevant words of the search field selected by the target user are obtained according to the operation instruction;

[0029] The search field and related words are taken together as search keywords and sent to the Spark search engine for search calculation to obtain the search results.

[0030] Based on the search results, the business data required by each branch company to complete its business execution logic is obtained from the data source table.

[0031] Furthermore, before executing the step of automatically generating components by calling the pre-deployed AI2SQL platform according to the preset second component calling interface, the method further includes:

[0032] Obtain the SQL statement auto-generation calling interface provided by the AI2SQL platform and use it as the second calling interface;

[0033] After executing the step of sending the search field and related words as search keywords to the Spark search engine for search calculation and obtaining the search results, the method further includes:

[0034] Based on the search results and the second calling interface, an SQL query statement is automatically generated. The SQL query statement is the SQL query statement corresponding to the business data required by each branch company to complete the business execution logic from the data source table.

[0035] Furthermore, the step of obtaining the business data required by each branch office to complete its business execution logic from the data source table specifically includes:

[0036] Obtain the address information of the data source table and set it as the data query address for the SQL query statement corresponding to each branch company;

[0037] Obtain the address information of the target data tables used by each branch company, and set them as the data output address of the SQL query statements corresponding to each branch company;

[0038] Execute the SQL query statements corresponding to each branch office and obtain the query results through the data query address;

[0039] The query results are sent to the data sub-tables constructed for each branch company based on the data output address.

[0040] To address the aforementioned technical problems, this application also provides a cross-server data retrieval acceleration device, which employs the following technical solution:

[0041] A device for accelerating cross-server data retrieval includes:

[0042] The data source table acquisition module is used to acquire each non-partitioned table and each partitioned table pre-built in the target Hive data warehouse as data source tables, wherein the target Hive data warehouse is set in the first data server, and the first data server is used to cache the data source tables;

[0043] The computation engine runtime module is used to run the pre-deployed Spark search computation engine;

[0044] The data retrieval module is used to retrieve the business data required by each branch company to complete its business execution logic from the data source table, based on the search fields input by the target user, using the business execution logic of each branch company as the calculation logic and the Spark search engine as the retrieval engine.

[0045] The data entry and table partitioning module is used to build data partitioning tables for each branch company. The business data required by each branch company to complete the business execution logic is entered into the target data partitioning table. The target data partitioning table is a data partitioning table that has a one-to-one correspondence with each branch company.

[0046] The data sharding and caching module is used to send the target data shards corresponding to each branch company to a preset second data server. The second data server is deployed in each branch company to cache the corresponding target data shards.

[0047] To address the aforementioned technical problems, this application also provides a computer device that employs the following technical solution:

[0048] A computer device includes a memory and a processor, wherein the memory stores computer-readable instructions, and the processor executes the computer-readable instructions to implement the steps of the cross-server data call acceleration method described above.

[0049] To address the aforementioned technical problems, this application also provides a computer-readable storage medium, employing the technical solution described below:

[0050] A computer-readable storage medium storing computer-readable instructions, which, when executed by a processor, implement the steps of the cross-server data retrieval acceleration method described above.

[0051] Compared with the prior art, the embodiments of this application have the following main advantages:

[0052] The method for accelerating cross-server data calls described in this application involves: acquiring data source tables from the target Hive data warehouse; running a pre-deployed Spark search engine; using the business execution logic of each insurance branch as the computational logic and the Spark search engine as the retrieval engine; retrieving the insurance business data required by each insurance branch to complete its business execution logic from the data source table based on the retrieval fields input by the target user; constructing data partition tables for each insurance branch; and inputting the corresponding business data required by each insurance branch to the target data partition tables; and sending the target data partition tables corresponding to each insurance branch to a preset second data server. By introducing the Spark search engine, Lexica keyword search engine, and AI2SQL platform automatic generation components, the method automatically generates external forms corresponding to each insurance branch using the Hive public data warehouse data of the insurance head office as the source data, improving the efficiency of data calls and retrieval from the financial data warehouse, and also achieving data separation for each insurance branch. Attached Figure Description

[0053] To more clearly illustrate the solutions in this application, the accompanying drawings used in the description of the embodiments of this application will be briefly introduced below. Obviously, the accompanying drawings described below are some embodiments of this application. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0054] Figure 1 This is an exemplary system architecture diagram to which this application can be applied;

[0055] Figure 2 A flowchart of an embodiment of the method for accelerating cross-server data calls according to this application;

[0056] Figure 3 A flowchart of an embodiment of training the Lexica suggestion word search engine based on the training data provided in this application;

[0057] Figure 4 yes Figure 2 A flowchart of a specific embodiment of step 203 shown;

[0058] Figure 5 yes Figure 4 A flowchart of a specific embodiment of step 405 shown;

[0059] Figure 6 A schematic diagram of a structural embodiment of the cross-server data call acceleration device according to this application;

[0060] Figure 7 yes Figure 6 A schematic diagram of a specific embodiment of module 603 is shown;

[0061] Figure 8 A schematic diagram of a specific embodiment of the Lexica prompt word search engine training module according to this application;

[0062] Figure 9 A schematic diagram of a specific embodiment of the AI2SQL platform automatic generation module according to this application;

[0063] Figure 10 A schematic diagram of the structure of an embodiment of the computer device according to this application. Detailed Implementation

[0064] Unless otherwise defined, all technical and scientific terms used herein have the same meaning as commonly understood by one of ordinary skill in the art to which this application pertains; the terminology used herein in the specification of the application is for the purpose of describing particular embodiments only and is not intended to be limiting of the application; the terms "comprising" and "having," and any variations thereof, in the specification, claims, and foregoing drawings of this application, are intended to cover non-exclusive inclusion. The terms "first," "second," etc., in the specification, claims, or foregoing drawings of this application are used to distinguish different objects, not to describe a particular order.

[0065] In this document, the term "embodiment" means that a particular feature, structure, or characteristic described in connection with an embodiment may be included in at least one embodiment of this application. The appearance of this phrase in various places throughout the specification does not necessarily refer to the same embodiment, nor is it a separate or alternative embodiment mutually exclusive with other embodiments. It will be explicitly and implicitly understood by those skilled in the art that the embodiments described herein can be combined with other embodiments.

[0066] To enable those skilled in the art to better understand the present application, the technical solutions in the embodiments of the present application will be clearly and completely described below with reference to the accompanying drawings.

[0067] like Figure 1 As shown, system architecture 100 may include terminal devices 101, 102, and 103, a network 104, and a server 105. Network 104 serves as the medium for providing communication links between terminal devices 101, 102, and 103 and server 105. Network 104 may include various connection types, such as wired or wireless communication links, or fiber optic cables, etc.

[0068] Users can use terminal devices 101, 102, and 103 to interact with server 105 via network 104 to receive or send messages, etc. Various communication client applications can be installed on terminal devices 101, 102, and 103, such as web browser applications, shopping applications, search applications, instant messaging tools, email clients, social media platform software, etc.

[0069] Terminal devices 101, 102, and 103 can be various electronic devices with displays and support web browsing, including but not limited to smartphones, tablets, e-book readers, MP3 players (Moving Picture Experts Group Audio Layer III), MP4 players (Moving Picture Experts Group Audio Layer IV), laptops, and desktop computers, etc.

[0070] Server 105 can be a server that provides various services, such as a backend server that supports the pages displayed on terminal devices 101, 102, and 103.

[0071] It should be noted that the cross-server data call acceleration method provided in this application embodiment is generally executed by the server / terminal device, and correspondingly, the cross-server data call acceleration device is generally set in the server / terminal device.

[0072] It should be understood that Figure 1 The number of terminal devices, networks, and servers shown is merely illustrative. Depending on implementation needs, any number of terminal devices, networks, and servers can be included.

[0073] To facilitate understanding, we will first introduce some relevant components in the embodiments of this application: Hive data warehouse, Spark search engine, GPT-3 autoregressive language model, Lexica prompt word search engine, and AI2SQL platform automatic generation component.

[0074] The Hive data warehouse is not only a data warehouse used for data storage, but also includes the Hive search engine. However, its search performance is far inferior to that of the Spark search engine. This leads to low data retrieval efficiency when using the Hive search engine to search and retrieve data maintained in the Hive data warehouse.

[0075] GPT-3 is an autoregressive language model with 175 billion parameters developed by OpenAI. It generates text that simulates human language through deep learning. Because it overcomes the shortcomings of traditional neural network technology in solving problems such as polysemy, it has very high learning efficiency and content output accuracy.

[0076] Lexica is a keyword suggestion search engine tool that uses GPT-3 as its analysis and recommendation model. It is used for keyword and related word prediction and recommendation suggestions, which can improve both the efficiency and accuracy of data acquisition.

[0077] The AI2SQL platform is based on the GPT-3 autoregressive language model and the Codex model series. Its purpose is to help both engineers and non-engineers easily write efficient and error-free SQL query statements without any knowledge of SQL, thereby reducing the error rate of SQL query statements and improving query efficiency and accuracy.

[0078] Continue to refer to Figure 2The diagram illustrates a flowchart of an embodiment of the cross-server data call acceleration method according to this application. The cross-server data call acceleration method includes the following steps:

[0079] Step 201: Obtain the pre-built non-partitioned tables and partitioned tables in the target Hive data warehouse as data source tables. The target Hive data warehouse is set up in the first data server, and the first data server is used to cache the data source tables.

[0080] Taking a property insurance company as an example, the various non-partitioned tables and partitioned tables pre-built in the target Hive data warehouse are essentially the original non-partitioned tables and partitioned tables in the property insurance company before the target Hive data warehouse was upgraded.

[0081] In this embodiment, the first data server is used to cache the data source table. The data source table can be a data source table related to insurance business in the financial industry. The first data server can also be understood as a physical hardware server that caches data in the target Hive data warehouse, or a virtual data platform that caches data in the target Hive data warehouse.

[0082] Step 202: Run the pre-deployed Spark search engine.

[0083] In this embodiment, the Spark search engine is used to replace the Hive search engine built into the target Hive data warehouse. Subsequently, the search operation is performed only through the Spark search engine, which improves the efficiency of data retrieval and retrieval from the data warehouse.

[0084] Step 203: Using the business execution logic of each branch as the computation logic and the Spark search engine as the retrieval engine, obtain the business data required by each branch to complete its business execution logic from the data source table according to the retrieval fields input by the target user.

[0085] In this embodiment, before executing the step of using the business execution logic of each branch as the computational logic and the Spark search engine as the retrieval engine, the method further includes: calling a pre-deployed Lexica suggestion word search engine according to a preset first component call interface, wherein the Lexica suggestion word search engine is trained using a GPT-3 autoregressive language model; running the pre-deployed Lexica suggestion word search engine; calling a pre-deployed AI2SQL platform automatic generation component according to a preset second component call interface; and running the pre-deployed AI2SQL automatic generation component.

[0086] In this embodiment, the preset first component call interface is used to call the Lexica suggestion word search engine, which can be the call address corresponding to starting the Lexica suggestion word search engine; the preset second component call interface is used to call the AI2SQL platform automatically generated component, which can be the call interface provided by the component in the AI2SQL platform that automatically generates SQL query statements.

[0087] In this embodiment, before data retrieval and processing by the Spark search engine, a Lexica suggestion search engine and an AI2SQL platform auto-generation component are pre-introduced. The Lexica suggestion search engine recommends more precise search terms to the target user when they input a search term. For example, if the target user wants to input "great wall" but only inputs "great wa," and the target data in the target Hive data warehouse is "great wall," the Lexica suggestion search engine completes the search with "great wall" and displays it on a pre-defined interface for the user to select, further improving the efficiency of data retrieval and processing from the data warehouse. Furthermore, the AI2SQL platform auto-generation component automatically constructs SQL query statements based on the target data's position in the partitioned and non-partitioned tables of the target Hive data warehouse after the Spark search engine retrieves the target data, further improving the efficiency of data retrieval and processing from the data warehouse.

[0088] In this embodiment, the target user can be a program developer, whose development requirement is to transfer the data in the target Hive data warehouse to a pre-set second data server in each property insurance branch.

[0089] In this embodiment, before executing the step of calling the pre-deployed Lexica prompt word search engine according to the preset component first call interface, the method further includes: pre-training the Lexica prompt word search engine using data in the data source table as training data and the GPT-3 autoregressive language model as the training model.

[0090] By using the data in the data source table as training data and the GPT-3 autoregressive language model as the training model to train the Lexica suggestion search engine, it can be ensured that the suggestion range of the Lexica suggestion search engine is limited to the data in the data source table. This not only limits the suggestion range of the suggestion search engine and avoids blindly displaying irrelevant data, but also ensures the accuracy of the data suggestion range without interference from other irrelevant data, further improving the efficiency of data retrieval and access from the data warehouse.

[0091] Continue to refer to Figure 3 The flowchart illustrates an embodiment of training a Lexica suggestion word search engine based on the training data provided in this application, including:

[0092] Step 301: Input the training data into the GPT-3 autoregressive language model to perform autoregressive training and obtain the model output results;

[0093] Step 302: Calculate the similarity between the model output result and the preset model output dictionary according to the preset similarity algorithm;

[0094] In this embodiment, the similarity algorithm can be any algorithm used to compare the data similarity between two datasets, such as Euclidean distance algorithm, cosine similarity algorithm, and Jaccard similarity algorithm.

[0095] Step 303: Determine whether the similarity meets the preset similarity threshold;

[0096] Step 304: If the similarity meets the preset similarity threshold, the Lexica prompt word search engine training is complete and training is terminated;

[0097] Step 305: If the similarity does not meet the preset similarity threshold, adjust the regression parameters of the GPT-3 autoregressive language model and repeat steps 301 to 305.

[0098] By using the data in the data source table as training data and the GPT-3 autoregressive language model as the training model to train the Lexica suggestion search engine, it can be ensured that the suggestion range of the Lexica suggestion search engine is limited to the data in the data source table. This not only limits the suggestion range of the suggestion search engine and avoids blindly displaying irrelevant data, but also ensures the accuracy of the data suggestion range without interference from other irrelevant data, further improving the efficiency of data retrieval and access from the data warehouse.

[0099] By employing an autoregressive training method and adjusting the regression parameters of the GPT-3 autoregressive language model, the training success of the Lexica prompt word search engine was ensured.

[0100] Continue to refer to Figure 4 , Figure 4 yes Figure 2 A flowchart of a specific embodiment of step 203 shown includes:

[0101] Step 401: Obtain the search fields input by the target user;

[0102] Step 402: Based on the Lexica suggestion word search engine, recommend and display the relevant word sequence corresponding to the search field on the target display interface;

[0103] Step 403: In response to the operation instruction executed by the target user on the related word sequence, obtain the related words of the search field selected by the target user according to the operation instruction;

[0104] Step 404: The search field and related words of the search field are sent together as search keywords to the Spark search engine for search calculation and the search results are obtained.

[0105] Step 405: Based on the search calculation results, obtain the business data required by each branch company to complete the business execution logic from the data source table.

[0106] In this embodiment, before executing the step of calling the pre-deployed AI2SQL platform to automatically generate components according to the preset component second calling interface, the method further includes: obtaining the SQL statement automatic generation calling interface provided by the AI2SQL platform as the second calling interface.

[0107] In this embodiment, after executing the step of sending the search field and related words of the search field as search keywords to the Spark search engine for search calculation and obtaining the search calculation result, the method further includes: automatically generating an SQL query statement based on the search calculation result and the second calling interface, wherein the SQL query statement is an SQL query statement corresponding to the business data required by each branch company to complete the business execution logic from the data source table.

[0108] Continue to refer to Figure 5 , Figure 5 yes Figure 4 A flowchart of a specific embodiment of step 405 shown includes:

[0109] Step 501: Obtain the address information of the data source table and set it as the data query address of the SQL query statement corresponding to each branch company;

[0110] Step 502: Obtain the address information of the target data partition table used by each branch company, and set it as the data output address of the SQL query statement corresponding to each branch company;

[0111] Step 503: Execute the SQL query statements corresponding to each branch company and obtain the query results through the data query address;

[0112] Step 504: Send the query results to the data sub-tables constructed for each branch company according to the data output address.

[0113] The AI2SQL platform helps target users easily write efficient and error-free SQL query statements even without SQL knowledge, reducing the error rate of SQL query statement writing and improving query efficiency and accuracy. Due to the efficient and error-free SQL query statements, the efficiency of data retrieval from the data warehouse is further improved.

[0114] Step 204: Construct data sub-tables for each branch company, and enter the business data required by each branch company when completing the business execution logic into the target data sub-table. The target data sub-table represents a data sub-table with a one-to-one correspondence with each branch company.

[0115] Taking property insurance as an example, by constructing data sub-tables for each property insurance branch, the financial business data required by each branch to complete its business execution logic is entered into the target data sub-table. This allows each branch to access data and run programs without needing to directly call and retrieve data from the financial data warehouse if the financial data warehouse has not been updated. This reduces the direct data call process between the first data server and each property insurance branch, thereby reducing the resource processing consumption of the first data server.

[0116] Step 205: Send the target data tables corresponding to each branch company to a preset second data server, wherein the second data server is deployed in each branch company to cache the corresponding target data tables.

[0117] By sending the target data tables corresponding to each property insurance branch to the second data server deployed within each property insurance branch, when each property insurance branch uses data and runs programs, if there are no data updates in the financial data warehouse, it only needs to retrieve the data from the target data tables corresponding to each property insurance branch, without having to directly call and retrieve data from the financial data warehouse. This reduces the direct data call process between the first data server and each property insurance branch, and correspondingly reduces the resource processing consumption of the first data server.

[0118] This application obtains data source tables from the target Hive data warehouse; runs a pre-deployed Spark search engine; uses the business execution logic of each insurance branch as the computational logic and the Spark search engine as the retrieval engine; and retrieves the insurance business data required by each insurance branch to complete its business execution logic from the data source tables based on the retrieval fields input by the target user. It then constructs data partition tables for each insurance branch, entering the corresponding business data required by each branch into the target data partition tables; and finally sends the target data partition tables corresponding to each insurance branch to a pre-set second data server. By introducing the Spark search engine, Lexica keyword search engine, and AI2SQL platform automatic generation components, it automatically generates external forms corresponding to each insurance branch using the Hive public data warehouse data of the insurance head office as the source data. This improves the efficiency of data retrieval and retrieval from the financial data warehouse and also achieves data separation for each insurance branch.

[0119] The embodiments of this application can acquire and process relevant data based on artificial intelligence technology. Artificial intelligence (AI) refers to the theories, methods, technologies, and application systems that use digital computers or machines controlled by digital computers to simulate, extend, and expand human intelligence, perceive the environment, acquire knowledge, and use that knowledge to obtain optimal results.

[0120] Foundational technologies for artificial intelligence generally include sensors, dedicated AI chips, cloud computing, distributed storage, big data processing, operating / interactive systems, and mechatronics. AI software technologies mainly encompass computer vision, robotics, biometrics, speech processing, natural language processing, and machine learning / deep learning.

[0121] In this embodiment, by introducing the Spark search engine, the Lexicon prompt word search engine, and the AI2SQL platform automatic generation component, the system automatically generates external forms corresponding to each insurance branch using the Hive public data warehouse data of the insurance head office as the source data. This improves the efficiency of data retrieval and access from the public financial data warehouse, while also enabling data separation on a per-insurance-branch basis.

[0122] Further reference Figure 6 As a response to the above Figure 2 The implementation of the method shown in this application provides an embodiment of a cross-server data retrieval acceleration device, which is similar to... Figure 2 Corresponding to the method embodiments shown, this device can be specifically applied to various electronic devices.

[0123] like Figure 6 As shown, the cross-server data retrieval acceleration device 600 described in this embodiment includes: a data source table acquisition module 601, a computing engine operation module 602, a data retrieval acquisition module 603, a data entry and table partitioning module 604, and a data partitioning and caching module 605. Wherein:

[0124] The data source table acquisition module 601 is used to acquire each non-partitioned table and each partitioned table pre-built in the target Hive data warehouse as data source tables, wherein the target Hive data warehouse is set in the first data server, and the first data server is used to cache the data source tables.

[0125] The computation engine running module 602 is used to run the pre-deployed Spark search computation engine;

[0126] The data retrieval module 603 is used to retrieve the business data required by each branch company to complete its business execution logic from the data source table, based on the search fields input by the target user, using the business execution logic of each branch company as the calculation logic and the Spark search engine as the retrieval engine.

[0127] The data entry sub-table module 604 is used to construct data sub-tables for each branch company, and to enter the business data required by each branch company when completing the business execution logic into the target data sub-table. The target data sub-table represents a data sub-table that has a one-to-one correspondence with each branch company.

[0128] The data sharding caching module 605 is used to send the target data shards corresponding to each branch company to a preset second data server, wherein the second data server is deployed in each branch company to cache the corresponding target data shards.

[0129] Continue to refer to Figure 7 , Figure 7 yes Figure 6 The schematic diagram shows a specific embodiment of module 603, wherein the data retrieval and acquisition module 603 includes a retrieval field input submodule 701, a prompt word display submodule 702, a prompt word selection submodule 703, a search calculation submodule 704, and a data acquisition submodule 705, wherein:

[0130] The retrieval field input submodule 701 is used to obtain the retrieval field input by the target user;

[0131] The prompt word display submodule 702 is used to recommend and display the relevant word sequence corresponding to the search field on the target display interface according to the Lexica prompt word search engine.

[0132] The prompt word selection submodule 703 is used to respond to the operation instruction executed by the target user on the related word sequence, and to obtain the related words of the search field selected by the target user according to the operation instruction;

[0133] The search operation submodule 704 is used to send the search field and related words of the search field as search keywords to the Spark search operation engine for search operation and to obtain the search operation results.

[0134] The data acquisition submodule 705 is used to obtain, based on the search calculation results, the business data required by each branch company to complete its business execution logic from the data source table.

[0135] In some specific embodiments of this application, the cross-server data retrieval acceleration device 600 further includes a Lexica suggestion word search engine training module, which will be discussed further below. Figure 8 , Figure 8 This is a schematic diagram of a specific embodiment of the Lexica prompt word search engine training module according to this application. The Lexica prompt word search engine training module includes a training data input submodule 801, a similarity calculation submodule 802, a comparison judgment submodule 803, a training completion judgment submodule 804, and a regression parameter adjustment submodule 805, wherein:

[0136] The training data input submodule 801 is used to input the training data into the GPT-3 autoregressive language model, perform autoregressive training, and obtain the model output results;

[0137] The similarity calculation submodule 802 is used to calculate the similarity between the model output result and the preset model output dictionary according to the preset similarity algorithm.

[0138] The comparison and judgment submodule 803 is used to determine whether the similarity meets the preset similarity threshold;

[0139] The training completion judgment submodule 804 is used to determine that if the similarity meets the preset similarity threshold, the Lexica prompt word search engine training is complete and training is terminated.

[0140] The regression parameter adjustment submodule 805 is used to adjust the regression parameters of the GPT-3 autoregressive language model and repeat the above training steps if the similarity does not meet the preset similarity threshold.

[0141] In some specific embodiments of this application, the cross-server data retrieval acceleration device 600 further includes an AI2SQL platform automatic generation module. (Continue to refer to...) Figure 9 , Figure 9 This is a schematic diagram of a specific embodiment of the AI2SQL platform automatic generation module according to this application. The AI2SQL platform automatic generation module includes an interface acquisition submodule 901 and a query statement generation submodule 902, wherein:

[0142] The interface acquisition submodule 901 is used to obtain the SQL statement automatic generation call interface provided by the AI2SQL platform as the second call interface;

[0143] The query statement generation submodule 902 is used to automatically generate SQL query statements based on the search calculation results and the second calling interface. The SQL query statements are SQL query statements corresponding to the business data required by each branch company to complete the business execution logic from the data source table.

[0144] This application obtains data source tables from the target Hive data warehouse; runs a pre-deployed Spark search engine; uses the business execution logic of each insurance branch as the computational logic and the Spark search engine as the retrieval engine; and retrieves the insurance business data required by each insurance branch to complete its business execution logic from the data source tables based on the retrieval fields input by the target user. It then constructs data partition tables for each insurance branch, entering the corresponding business data required by each branch into the target data partition tables; and finally sends the target data partition tables corresponding to each insurance branch to a pre-set second data server. By introducing the Spark search engine, Lexica keyword search engine, and AI2SQL platform automatic generation components, it automatically generates external forms corresponding to each insurance branch using the Hive public data warehouse data of the insurance head office as the source data. This improves the efficiency of data retrieval and retrieval from the financial data warehouse and also achieves data separation for each insurance branch.

[0145] Those skilled in the art will understand that all or part of the processes in the methods of the above embodiments can be implemented by instructing related hardware through computer-readable instructions. These computer-readable instructions can be stored in a computer-readable storage medium. When the program is executed, it can include the processes of the embodiments of the methods described above. The aforementioned storage medium can be a non-volatile storage medium such as a magnetic disk, optical disk, or read-only memory (ROM), or random access memory (RAM).

[0146] It should be understood that although the steps in the flowcharts of the accompanying figures are shown sequentially as indicated by the arrows, these steps are not necessarily executed in the order indicated by the arrows. Unless explicitly stated herein, there is no strict order restriction on the execution of these steps, and they can be executed in other orders. Moreover, at least some steps in the flowcharts of the accompanying figures may include multiple sub-steps or multiple stages. These sub-steps or stages are not necessarily completed at the same time, but can be executed at different times, and their execution order is not necessarily sequential, but can be performed alternately or in turn with other steps or at least some of the sub-steps or stages of other steps.

[0147] To address the aforementioned technical problems, embodiments of this application also provide a computer device. Please refer to [link / reference needed]. Figure 10 , Figure 10 This is a basic structural block diagram of the computer device in this embodiment.

[0148] The computer device 10 includes a memory 10a, a processor 10b, and a network interface 10c that are interconnected via a system bus. It should be noted that only the computer device 10 with components 10a-10c is shown in the figure; however, it should be understood that it is not required to implement all the shown components, and more or fewer components can be implemented alternatively. Those skilled in the art will understand that the computer device described here is a device capable of automatically performing numerical calculations and / or information processing according to pre-set or stored instructions, and its hardware includes, but is not limited to, microprocessors, application-specific integrated circuits (ASICs), field-programmable gate arrays (FPGAs), digital signal processors (DSPs), embedded devices, etc.

[0149] The computer device can be a desktop computer, laptop, handheld computer, or cloud server, etc. The computer device can interact with the user via a keyboard, mouse, remote control, touchpad, or voice control.

[0150] The memory 10a includes at least one type of readable storage medium, including flash memory, hard disk, multimedia card, card-type memory (e.g., SD or DX memory), random access memory (RAM), static random access memory (SRAM), read-only memory (ROM), electrically erasable programmable read-only memory (EEPROM), programmable read-only memory (PROM), magnetic memory, magnetic disk, optical disk, etc. In some embodiments, the memory 10a may be an internal storage unit of the computer device 10, such as the hard disk or memory of the computer device 10. In other embodiments, the memory 10a may also be an external storage device of the computer device 10, such as a plug-in hard disk, smart media card (SMC), secure digital (SD) card, flash card, etc., equipped on the computer device 10. Of course, the memory 10a may also include both the internal storage unit and the external storage device of the computer device 10. In this embodiment, the memory 10a is typically used to store the operating system and various application software installed on the computer device 10, such as computer-readable instructions for methods to accelerate data calls between servers. Furthermore, the memory 10a can also be used to temporarily store various types of data that have already been output or will be output.

[0151] In some embodiments, the processor 10b may be a central processing unit (CPU), controller, microcontroller, microprocessor, or other data processing chip. The processor 10b is typically used to control the overall operation of the computer device 10. In this embodiment, the processor 10b is used to execute computer-readable instructions stored in the memory 10a or to process data, for example, to execute computer-readable instructions for the cross-server data call acceleration method.

[0152] The network interface 10c may include a wireless network interface or a wired network interface, which is typically used to establish communication connections between the computer device 10 and other electronic devices.

[0153] The computer equipment proposed in this embodiment belongs to the field of R&D optimization technology and is applied in the R&D optimization process of the financial industry. This application obtains data source tables from the target Hive data warehouse; runs a pre-deployed Spark search engine; uses the business execution logic of each insurance branch as the calculation logic and the Spark search engine as the retrieval engine; based on the retrieval fields input by the target user, it obtains the insurance business data required by each insurance branch to complete its business execution logic from the data source table; constructs data sub-tables for each insurance branch, and enters the corresponding business data required by each insurance branch to complete its business execution logic into the target data sub-tables; and sends the target data sub-tables corresponding to each insurance branch to a preset second data server. By introducing the Spark search engine, Lexica suggestion search engine, and AI2SQL platform automatic generation components, it automatically generates external forms corresponding to each insurance branch using the Hive public data warehouse data of the insurance head office as the source data, improving the efficiency of data retrieval from the financial data warehouse, and also achieving data separation for each insurance branch.

[0154] This application also provides another embodiment, namely, a computer-readable storage medium storing computer-readable instructions that can be executed by a processor to cause the processor to perform the steps of the cross-server data call acceleration method described above.

[0155] The computer-readable storage medium proposed in this embodiment belongs to the field of R&D optimization technology and is applied in the R&D optimization process of the financial industry. This application obtains data source tables from the target Hive data warehouse; runs a pre-deployed Spark search engine; uses the business execution logic of each insurance branch as the calculation logic and the Spark search engine as the retrieval engine; based on the retrieval fields input by the target user, it obtains the insurance business data required by each insurance branch to complete its business execution logic from the data source table; constructs data sub-tables for each insurance branch, and enters the corresponding business data required by each insurance branch to complete its business execution logic into the target data sub-tables; and sends the target data sub-tables corresponding to each insurance branch to a preset second data server. By introducing the Spark search engine, Lexica suggestion search engine, and AI2SQL platform automatic generation components, it automatically generates external forms corresponding to each insurance branch using the Hive public data warehouse data of the insurance head office as the source data, improving the efficiency of data retrieval and access from the financial data warehouse, while also achieving data separation for each insurance branch.

[0156] Through the above description of the embodiments, those skilled in the art can clearly understand that the methods of the above embodiments can be implemented by means of software plus necessary general-purpose hardware platforms. Of course, they can also be implemented by hardware, but in many cases the former is a better implementation method. Based on this understanding, the technical solution of this application, in essence, or the part that contributes to the prior art, can be embodied in the form of a software product. This computer software product is stored in a storage medium (such as ROM / RAM, magnetic disk, optical disk) and includes several instructions to cause a terminal device (which may be a mobile phone, computer, server, air conditioner, or network device, etc.) to execute the methods described in the various embodiments of this application.

[0157] Obviously, the embodiments described above are only some embodiments of this application, not all embodiments. The accompanying drawings show preferred embodiments of this application, but do not limit the patent scope of this application. This application can be implemented in many different forms; rather, the purpose of providing these embodiments is to provide a more thorough and comprehensive understanding of the disclosure of this application. Although this application has been described in detail with reference to the foregoing embodiments, those skilled in the art can still modify the technical solutions described in the foregoing specific embodiments, or make equivalent substitutions for some of the technical features. Any equivalent structures made using the content of this application's specification and drawings, directly or indirectly applied to other related technical fields, are similarly within the scope of patent protection of this application.

Claims

1. A method for accelerating cross-server data retrieval, characterized in that, Includes the following steps: Obtain each non-partitioned table and each partitioned table pre-built in the target Hive data warehouse as a data source table, wherein the target Hive data warehouse is set in a first data server, and the first data server is used to cache the data source tables; Run a pre-deployed Spark search engine; Using the business execution logic of each branch as the computational logic and the Spark search engine as the retrieval engine, the method retrieves the business data required by each branch to complete its business execution logic from the data source table based on the retrieval fields input by the target user. Prior to executing the step of using the business execution logic of each branch as the computational logic and the Spark search engine as the retrieval engine, the method further includes: According to the preset component first call interface, the pre-deployed Lexica prompt word search engine is called, wherein the Lexica prompt word search engine is trained using the GPT-3 autoregressive language model; Run the pre-deployed Lexica suggestion search engine; Based on the preset second component call interface, the pre-deployed AI2SQL platform is called to automatically generate components; Run the pre-deployed AI2SQL auto-generation component; Data tables are constructed for each branch company. The business data required by each branch company to complete its business execution logic is entered into the target data table. The target data table is a data table that has a one-to-one correspondence with each branch company. The target data tables corresponding to each branch are sent to a preset second data server, which is deployed in each branch to cache the corresponding target data tables.

2. The method of claim 1, wherein, Before executing the step of invoking the pre-deployed Lexica suggestion search engine according to the preset component first call interface, the method further includes: The Lexica prompt word search engine was trained in advance using data from the data source table as training data and the GPT-3 autoregressive language model as the training model.

3. The method of claim 2, wherein, The steps of training the Lexica suggestion word search engine using data from the data source table as training data and the GPT-3 autoregressive language model as the training model specifically include: Step 301: Input the training data into the GPT-3 autoregressive language model to perform autoregressive training and obtain the model output results; Step 302: Calculate the similarity between the model output result and the preset model output dictionary according to the preset similarity algorithm; Step 303: Determine whether the similarity meets the preset similarity threshold; Step 304: If the similarity meets the preset similarity threshold, the Lexica prompt word search engine training is complete and training is terminated; Step 305: If the similarity does not meet the preset similarity threshold, adjust the regression parameters of the GPT-3 autoregressive language model and repeat steps 301 to 305.

4. The method of claim 1 to 3, wherein, The step of retrieving the business data required by each branch office to complete its business execution logic from the data source table based on the search fields input by the target user specifically includes: Obtain the search fields input by the target user; Based on the Lexica suggestion word search engine, the relevant word sequence corresponding to the search field is recommended and displayed on the target display interface; In response to the operation instruction executed by the target user on the relevant word sequence, the relevant words of the search field selected by the target user are obtained according to the operation instruction; The search field and related words are taken together as search keywords and sent to the Spark search engine for search calculation to obtain the search results. Based on the search results, the business data required by each branch company to complete its business execution logic is obtained from the data source table.

5. The method of claim 4, wherein, Before executing the step of automatically generating components by calling the pre-deployed AI2SQL platform according to the preset component second calling interface, the method further includes: Obtain the SQL statement auto-generation calling interface provided by the AI2SQL platform and use it as the second calling interface; After executing the step of sending the search field and related words as search keywords to the Spark search engine for search calculation and obtaining the search results, the method further includes: Based on the search results and the second calling interface, an SQL query statement is automatically generated. The SQL query statement is the SQL query statement corresponding to the business data required by each branch company to complete the business execution logic from the data source table.

6. The method of claim 5, wherein, The step of obtaining the business data required by each branch office to complete its business execution logic from the data source table specifically includes: Obtain the address information of the data source table and set it as the data query address for the SQL query statement corresponding to each branch company; Obtain the address information of the target data tables used by each branch company, and set them as the data output address of the SQL query statements corresponding to each branch company; Execute the SQL query statements corresponding to each branch office and obtain the query results through the data query address; The query results are sent to the data sub-tables constructed for each branch company based on the data output address.

7. A device for accelerating cross-server data retrieval, characterized in that, include: The data source table acquisition module is used to acquire each non-partitioned table and each partitioned table pre-built in the target Hive data warehouse as data source tables, wherein the target Hive data warehouse is set in the first data server, and the first data server is used to cache the data source tables; The computation engine runtime module is used to run the pre-deployed Spark search computation engine; The data retrieval module is used to retrieve the business data required by each branch office to complete its business execution logic from the data source table, based on the search fields input by the target user, using the business execution logic of each branch office as the computational logic and the Spark search engine as the retrieval engine. Prior to executing the computational logic based on the business execution logic of each branch office and the Spark search engine as the retrieval engine, the module further includes: According to the preset component first call interface, the pre-deployed Lexica prompt word search engine is called, wherein the Lexica prompt word search engine is trained using the GPT-3 autoregressive language model; Run the pre-deployed Lexica suggestion search engine; Based on the preset second component call interface, the pre-deployed AI2SQL platform is called to automatically generate components; Run the pre-deployed AI2SQL auto-generation component; The data entry and table partitioning module is used to build data partitioning tables for each branch company. The business data required by each branch company to complete the business execution logic is entered into the target data partitioning table. The target data partitioning table is a data partitioning table that has a one-to-one correspondence with each branch company. The data sharding and caching module is used to send the target data shards corresponding to each branch company to a preset second data server. The second data server is deployed in each branch company to cache the corresponding target data shards.

8. A computer device comprising a memory and a processor, the memory storing computer-readable instructions, wherein the processor, when executing the computer-readable instructions, implements the steps of the method for accelerating cross-server data retrieval as described in any one of claims 1 to 6.

9. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores computer-readable instructions, which, when executed by a processor, implement the steps of the cross-server data retrieval acceleration method as described in any one of claims 1 to 6.

Images