A data query retrieval method and apparatus

Abstract

The present application relates to the technical field of data query retrieval, and particularly relates to a data query retrieval method and device. First, a client develops a consumption function FCy(x, c) according to a remote service specification, wherein x represents an input parameter of the consumption function FC, c represents a context of the consumption function execution, and y represents a consumption function execution result. Then, the client calls a remote service, and transmits the input parameter x and the consumption function FC to a server through a network. Finally, the client receives the execution result y returned from the server. The method can meet various variable and uncertain data retrieval requirements, and improves the stability of a data retrieval server program. The method is suitable for application scenarios such as data centers and data asset management, and is convenient to popularize and use.

Description

Technical Field

[0001] This invention relates to the field of data query and retrieval technology, and in particular to a method and device for data query and retrieval. Background Technology

[0002] Cross-application system data retrieval, analysis, and processing technologies generally involve an application system (client) sending a data query request to another application system (server), the server then performing the query, analysis, and processing according to the client's requirements to obtain the processing result, and finally returning the processing result to the client.

[0003] In the data processing described above, the server-side program must be deeply integrated into the client's business logic; otherwise, it cannot produce results that meet the client's needs. This means that for every different data requirement model the client has, the server must develop a corresponding set of data processing logic. In short, as client requirements change, the server-side data processing logic must also change. Although the server can provide plugins to extend its service capabilities and address adaptability issues, even so, the workload of organizing and coordinating development and testing on both ends is substantial, making it difficult to meet the unpredictable nature of data retrieval needs that may change at any time. Summary of the Invention

[0004] In view of the shortcomings of the existing technology, the purpose of this invention is to provide a method and device for data query and retrieval, which can not only meet the needs of data retrieval that is constantly updated and changing, but also significantly reduce the operation and maintenance costs of the server.

[0005] To achieve the above objectives, the present invention provides the following technical solution:

[0006] A data query and retrieval method includes:

[0007] The client develops a consumer function FCy(x,c) according to the remote service specification, where: x represents the input parameter of the consumer function FC, c represents the execution context of the consumer function, the context provides the services and resources required for the consumer function to run, and y represents the execution result of the consumer function;

[0008] The client calls a remote service, transmitting the input parameter x and the consumption function FC to the server via the network;

[0009] The client receives the execution result y returned from the server.

[0010] Furthermore, the process of transmitting the input parameter x and the consumption function FC to the server via the network includes: binary encoding the input parameter x and the consumption function FC, and sending the encoded byte array to the server via the network.

[0011] Furthermore, the process of binary encoding the consumption function FC includes:

[0012] When Java is used as the client-side development language, the lambda technology of JDK 8 and above is used to implement the consumer function. The remote service proxy of the client serializes the consumer function to obtain a binary encoded array.

[0013] The source code for the consumer function and the client source code are mixed and coexist in the same Java file;

[0014] The implementation of the consumption function using JDK 8 and above lambda technology includes:

[0015] Define a new lambda expression interface. The characteristics of the new lambda expression interface are: an interface annotated with @FunctionalInterface; the interface has only one method; and the interface is derived from the java.io.Serializable interface.

[0016] A data query and retrieval method includes:

[0017] The server-side models the raw data to obtain a data model;

[0018] The server-side develops a data loading service based on the data model;

[0019] The server-side remote service receives input parameters x and a consumption function FCy(x,c) from the client, where FC represents the consumption function, x represents the input parameters of the consumption function FC, c represents the execution context of the consumption function, the context provides the services and resources required for the consumption function to run, and y represents the execution result of the consumption function.

[0020] The server-side constructs the environment and context required for the execution of the consumer function (c).

[0021] The server calls the consumption function FCy(x,c). The consumption function uses the input parameter x and the services and resources provided by the context c to retrieve, analyze and process the data, and obtain the execution result y of the consumption function.

[0022] The server returns the execution result y to the client.

[0023] Furthermore, the process by which the server-side remote service receives the input parameter x and the consumption function FCy(x,c) from the client includes:

[0024] The remote server receives the input parameter x and the binary encoded array of the consumption function FCy(x,c) sent by the client.

[0025] The server decodes the binary encoded array to restore the input parameter x and the consumption function FCy(x,c);

[0026] The server loads the consumption function FCy(x,c).

[0027] Furthermore, before returning the execution result y to the client, the process also includes unloading the execution environment and context c of the consumer function.

[0028] Furthermore, the loading consumption function FCy(x,c) includes:

[0029] If the received consumer function is written in a scripting language, the server loads the consumer function through the script engine;

[0030] If the received consumption function is written in a non-scripting language, it is provided to the server as a string source code. The server uses a compiler to compile the source code in real time, generate executable code, and then loads it.

[0031] If the received consumer function is written in a non-scripting language, the consumer function is serialized into binary encoding, and the server uses deserialization to restore and load the consumer function.

[0032] Furthermore, the process of the server using deserialization to restore and load the consumer function includes:

[0033] When Java is used as a server-side development language:

[0034] S1. Parse the class name of the consumption function from the binary array;

[0035] a) Load classes by the name of the consumer function class;

[0036] b) If no system exception is thrown, proceed to step S2;

[0037] c) The system throws a class not found exception. Check if the class name in the exception is the same as the class name uploaded by the client last time. If they are the same, proceed to step S3.

[0038] d) Save the current state, request the client to upload the class file for the missing class, and reclaim the current processing thread;

[0039] e) Receive class files uploaded by the client and save them to the local file system;

[0040] f) Allocate processing threads, restore the current state, and execute step a);

[0041] S2. Successfully loaded the consumer function class, continue with subsequent processing;

[0042] S3. If the consumer function class or other classes that the consumer function depends on cannot be loaded, an exception is thrown, and the call processing is aborted.

[0043] Furthermore, it also includes:

[0044] The server creates a security sandbox. The following steps are implemented within the created security sandbox:

[0045] Receives the input parameter x and the consumption function FCy(x,c) from the client;

[0046] Construct the environment and context c required for the consumption function to execute;

[0047] The execution result is obtained by calling the consumption function FCy(x,c) with input parameter x and context c;

[0048] Unload the execution environment and context c of the consumer function;

[0049] Return the execution result y to the client.

[0050] A computer device includes a memory and a processor, the memory storing program instructions, the processor executing the program instructions to perform any of the methods described above.

[0051] Compared with the prior art, the present invention has the following beneficial effects:

[0052] This invention employs a client-side approach where a consumer function is developed and executed on the server side. This allows for the flexible organization and utilization of various services and resources provided by the server based on data retrieval needs. The server does not need to concern itself with the execution logic of the consumer function (i.e., the client's retrieval requirements); it only needs to return the execution result to the client. Following this processing strategy, the server can establish a fixed set of processing logic, which, combined with the use of the consumer function, satisfies the client's diverse data retrieval, analysis, and processing needs. This improves data retrieval service capabilities, maintains server-side program stability, reduces the workload of coordinating the development, debugging, and deployment of both ends of the system, and lowers server-side operation and maintenance costs. This invention is applicable to data centers, data asset management, and other application scenarios, facilitating widespread adoption. Attached Figure Description

[0053] Figure 1 This is an execution logic diagram of one embodiment of the data query and retrieval described in this invention;

[0054] Figure 2 This is an execution logic diagram of another embodiment of the data query and retrieval described in this invention;

[0055] Figure 3 This is a screenshot of the Java Lambda technology described in this invention, which combines consumer functions and client code.

[0056] Figure 4 This is the logic diagram of the class loading algorithm of the present invention;

[0057] Figure 5 This is a sequence diagram of the data query and retrieval execution of the present invention. Detailed Implementation

[0058] The present invention will be further described below with reference to the accompanying drawings and specific embodiments. Terms such as “upper,” “inner,” “middle,” “left,” “right,” and “one” used in this specification are merely for clarity of description and are not intended to limit the scope of the invention. Changes or adjustments to their relative relationships, without substantially altering the technical content, should also be considered within the scope of the invention.

[0059] The data query and retrieval method described in this invention involves two ends: a client and a server, such as... Figure 5 As shown below, the specific data query and retrieval methods will be described from the perspectives of both the client and the server.

[0060] From the client's perspective, the data query and retrieval method of the present invention includes the following steps:

[0061] S1. The client develops a consumer function FCy(x,c) according to the remote service specification, where: x represents the input parameter of the consumer function FC, c represents the execution context of the consumer function, the context is used to provide authorized local services and resources to the consumer function, which is provided by the server, and y represents the execution result of the consumer function.

[0062] S2. The client calls a remote service, transmitting the input parameter x and the consumption function FC to the server via the network. The specific processing includes:

[0063] The input parameter x and the consumption function FC are binary encoded, and the encoded byte array is sent to the server over the network.

[0064] The process of binary encoding the consumption function FC includes:

[0065] When Java is used as the client-side development language, the lambda technique of JDK8 (and above) is used to implement the consumer function. The remote service proxy of the client serializes the consumer function to obtain a binary encoded array. The source code of the consumer function and the client source code are mixed and coexist in the same Java file, which reduces the workload of separate compilation and packaging, improves the convenience of code debugging, and enhances the readability and manageability of the source code.

[0066] The implementation of the consumption function using JDK 8 and above lambda technology includes:

[0067] Define a new lambda expression interface. The characteristics of the new lambda expression interface are: an interface annotated with @FunctionalInterface; the interface has only one method; and the interface is derived from the java.io.Serializable interface.

[0068] The lambda expressions of this invention differ from existing technologies. The JDK provides a series of lambda expression interfaces in the java.util.function package. Compared with ordinary interfaces, lambda expression interfaces have the following two typical characteristics:

[0069] Feature 1: When an interface class is annotated with @FunctionalInterface, the compiler recognizes the interface as a lambda expression interface based on the @FunctionalInterface annotation.

[0070] Feature 2: An interface has one and only one method, indicating that the interface is used as a function.

[0071] Closures written using the aforementioned lambda expression interface, when compiled, generate anonymous inner classes that are not serializable. The lambda expression interface in this invention adds a new feature to existing technologies:

[0072] Feature 3: The interface is derived from the java.io.Serializable interface.

[0073] The new feature will result in anonymous inner classes generated after closure compilation being serializable and able to serve as carriers for consumer functions, thus expanding the scope of lambda expressions. Combined with the concise nature of lambda expressions, this innovative approach seamlessly integrates server-side execution code (consumer functions) and client-side code, presenting a unified interface. Figure 3 The effect shown.

[0074] S3. The client receives the execution result y returned from the server.

[0075] From the server-side perspective, the data query and retrieval method of this invention includes the following steps:

[0076] S1. The original data is provided by the server, and the server models the original data to obtain the data model.

[0077] S2. The server develops the services needed to load data according to the data model.

[0078] S3. The server remote service receives the input parameter x and the consumption function FCy(x,c) from the client, where FC represents the consumption function, x represents the input parameter of the consumption function FC, c represents the execution context of the consumption function, the context is used to provide the consumption function with authorized local services and resources, and provide the services required for the consumption function to run, including data services, cache services, temporary file services, etc.; y represents the execution result of the consumption function.

[0079] The processing steps of the server-side remote service receiving the input parameter x and the consumption function FCy(x,c) from the client include:

[0080] The remote server receives the input parameter x and the binary encoded array of the consumption function FCy(x,c) sent by the client.

[0081] The server decodes the binary code to restore the input parameter x and the consumption function FCy(x,c);

[0082] Load the consumption function FCy(x,c).

[0083] The loading consumption function FCy(x,c) includes:

[0084] If the received consumer function is written in a scripting language, the server loads the consumer function through the script engine;

[0085] If the received consumer function is written in a non-scripting language, it is provided to the server as a string source code. The server uses a compiler to compile the source code in real time, generate executable code, and then loads it.

[0086] If the received consumer function is written in a non-scripting language, the consumer function is serialized into binary encoding, and the server uses deserialization to restore and load the consumer function.

[0087] Consumption functions developed using the Java language, such as Figure 4 As shown, the server uses a class loading algorithm to implement the deserialization loading and consumption function, including:

[0088] Step 1: Parse the class name of the consumption function from the binary array;

[0089] a) Load classes by the name of the consumer function class;

[0090] b) If no system exception is thrown, proceed to step 2;

[0091] c) The system throws a class not found exception. Check if the class name in the exception is the same as the class name uploaded by the client last time. If they are the same, proceed to step 3.

[0092] d) Save the current state, require the client to upload the class file for the missing class, and reclaim the current processing thread; this will allow execution to continue after the interruption is completed, and reclaim the thread to avoid consuming resources.

[0093] e) Receive class files uploaded by the client and save them to the local file system;

[0094] f) Allocate a processing thread, restore the current state, and execute step a). At this point, another thread can be used to continue executing the task that was interrupted.

[0095] Step 2: Successfully loaded the consumer function class, continue with subsequent processing;

[0096] Step 3: If the consumer function class or other classes that the consumer function depends on cannot be loaded, an exception is thrown, and the call processing is aborted.

[0097] The class loading method of this invention differs from existing technologies. Existing technologies typically load classes from local Jar files or download Jar files from a network server node (center) before loading. The main component transmitted is the Jar file, and the Jar filename is fixed and known at compile time. The class loading process is synchronous; if a class is not found, a program exception is thrown and execution is terminated. In contrast, the classes in this invention originate from a peer (client). The location of the client is uncertain, and the classes to be loaded are consumer function classes and their importing classes, which cannot be determined at compile time. The main component transmitted is the .class file, which is managed locally as a Jar directory. The class loading process is asynchronous. Only when the class loader cannot find the required class in the local file system will it request the necessary .class file from the peer, saving the current state and reclaiming current thread resources. The class loading process resumes only after the peer provides the required class over the network. This class loading method saves network bandwidth resources, has a more flexible processing mechanism, and is suitable for loading consumer function classes and other referenced classes during remote service calls.

[0098] S4. The server-side constructs the environment and context required for the execution of the consumer function.

[0099] S5. The server calls the consumption function FCy(x,c). The consumption function uses the input parameter x and the services and resources provided by the context c to retrieve, analyze and process the data, and obtain the execution result y of the consumption function.

[0100] S6. The server unloads the execution environment and context c of the consumer function. The existence cycle of the consumer function is only between loading and unloading, saving server resources.

[0101] S6. The server returns the execution result y to the client.

[0102] The data query and retrieval method of the present invention further includes: the server performing security audit or code audit on the input parameters and consumption functions, and performing out-of-domain data security audit on the calculation results returned to the client, so as to enhance the security of the system and data.

[0103] The data query and retrieval method described in this invention can be applied in data asset trading scenarios, adhering to the principle of "data not leaving the domain," with the loading and execution of consumption functions placed within a security sandbox. Security sandbox technology is a security mechanism used to isolate running programs, aiming to restrict access permissions for untrusted processes or untrusted code, preventing them from posing threats to the host and system, and ensuring system and data security. Therefore, the data query and retrieval method described in this invention also includes creating a security sandbox, such as... Figure 2 As shown; when using Java as the development language, Java's built-in security sandbox technology can be used.

[0104] The following steps are performed in the security sandbox:

[0105] The server-side remote service receives the input parameter x and the consumption function FCy(x,c) from the client;

[0106] The server-side constructs the environment and context required for the execution of the consumer function (c).

[0107] The server calls the consumption function FCy(x,c) with input parameter x and context c to obtain the execution result;

[0108] The execution environment and context c of the server-side unloading consumer function;

[0109] The server returns the execution result y to the client.

[0110] A computer device includes a memory and a processor, the memory storing program instructions, and the processor executing the program instructions to perform the data query and retrieval method described above from a client perspective or a server perspective.

[0111] The data query and retrieval method of this invention can be visualized using a self-service kitchen in daily life. The self-service kitchen (server) provides customers (clients) with ingredients (data), cooking utensils (context / service), and a kitchen (environment / safety sandbox) for food preparation. Customers bring their own chefs (consumer functions), submit their food requests (input parameters) to the chefs (consumer functions), and after disinfection and other processes (audit), the chefs (consumer functions) enter the kitchen (environment / safety sandbox) to prepare the food. Finally, the food (execution result y) is delivered to the customer (client) through a server (audit). The advantage of a self-service kitchen is that the self-service kitchen (server) only needs to prepare the ingredients (data) and cooking utensils (context / service), and any request (retrieval request) from the customer (client) can be fulfilled.

[0112] This invention divides the data query and retrieval process into a personalized part (consumption function) and a common part (services, resources, data). The personalized part is developed by the client, while the services and data developed by the server are characterized by universality and a high degree of standardization. This method can be used to meet various changing and uncertain data retrieval needs, improving the stability of the data retrieval server program. At the same time, this division of labor saves the costs of communication and joint debugging between the data retrieval parties, shortens the delivery time of the client system, and saves the operation and maintenance costs of the server system. It is applicable to various application scenarios such as data centers and data asset management, and is easy to promote and use.

[0113] This method, designed for the Java programming language, improves the use of Lambda expressions and the asynchronous class loading algorithm. It embeds the server-side execution of consumer functions into the client-side code, creating a seamless integration that is not only easier to read and understand but also facilitates debugging, packaging, and deployment, thus improving software development efficiency. Furthermore, by combining Java's built-in security sandbox technology, it achieves a balance between convenience and security.

[0114] This method, when applied to the data asset trading process, establishes standards for data products based on public services and data. This facilitates the development of a pricing system for data products, promotes large-scale transactions, and fosters a data trading ecosystem.

[0115] The embodiments of the present invention are not limited thereto. Based on the above description of the present invention, and using common technical knowledge and conventional means in the field, the present invention can be modified, replaced or combined in various other forms without departing from the basic technical idea of ​​the present invention, and all such modifications, replacements or combinations fall within the scope of protection of the present invention.

Claims

1. A method for data query and retrieval, characterized in that, include: The server-side models the raw data to obtain a data model; The server-side develops a data loading service based on the data model; The server-side remote service receives input parameters x and a consumption function FCy(x,c) from the client, where FC represents the consumption function, x represents the input parameters of the consumption function FC, c represents the execution context of the consumption function, the context provides the services and resources required for the consumption function to run, and y represents the execution result of the consumption function. The server-side constructs the environment and context required for the execution of the consumer function (c). The server calls the consumption function FCy(x,c). The consumption function uses the input parameter x and the services and resources provided by the context c to retrieve, analyze and process the data, and obtain the execution result y of the consumption function. The server returns the execution result y to the client; The process of the server-side remote service receiving the input parameter x and the consumption function FCy(x,c) from the client includes: The remote server receives the input parameter x and the binary encoded array of the consumption function FCy(x,c) sent by the client. The server decodes the binary encoded array to restore the input parameter x and the consumption function FCy(x,c); The server loads the consumption function FCy(x,c); The loading consumption function FCy(x,c) includes: If the received consumption function is written in a non-scripting language, it is provided to the server as a string source code. The server uses a compiler to compile the source code in real time, generate executable code, and then loads it. If the received consumer function is written in a non-scripting language, the consumer function is serialized into binary encoding, and the server uses deserialization to restore and load the consumer function. The server-side process of restoring and loading the consumer function using deserialization includes: When Java is used as a server-side development language: S1. Parse the class name of the consumption function from the binary array; a) Load classes by the name of the consumer function class; b) If no system exception is thrown, proceed to step S2; c) The system throws a class not found exception. Check if the class name in the exception is the same as the class name uploaded by the client last time. If they are the same, proceed to step S3. d) Save the current state, request the client to upload the class file for the missing class, and reclaim the current processing thread; e) Receive class files uploaded by the client and save them to the local file system; f) Allocate processing threads, restore the current state, and execute step a); S2. Successfully loaded the consumer function class, continue with subsequent processing; S3. If the consumer function class or other classes that the consumer function depends on cannot be loaded, an exception is thrown, and the call processing is aborted.

2. The data query and retrieval method according to claim 1, characterized in that, Before returning the execution result y to the client, the process also includes: unloading the execution environment and context c of the consumer function.

3. The data query and retrieval method according to claim 1, characterized in that: The loading consumption function FCy(x,c) also includes: If the received consumer function is written in a scripting language, the server loads the consumer function through the script engine.

4. The data query and retrieval method according to claim 1, characterized in that, Also includes: The server creates a security sandbox. The following steps are implemented within the created security sandbox: Receives the input parameter x and the consumption function FCy(x,c) from the client; Construct the environment and context c required for the consumption function to execute; The execution result is obtained by calling the consumption function FCy(x,c) with input parameter x and context c; Unload the execution environment and context c of the consumer function; Return the execution result y to the client.

5. A method for data query and retrieval, characterized in that, include: The client develops a consumer function FCy(x,c) according to the remote service specification, where: x represents the input parameter of the consumer function FC, c represents the execution context of the consumer function, the context provides the services and resources required for the consumer function to run, and y represents the execution result of the consumer function; The source code of the consumer function and the client source code are mixed and coexisted in the same Java file, reducing the workload of separate compilation and packaging. The client calls a remote service, transmitting the input parameter x and the consumption function FC to the server via the network; according to the server's requirements, the client uploads the class file to the server. The client receives the execution result y returned from the server; The process of transmitting the input parameter x and the consumption function FC to the server via the network includes: binary encoding the input parameter x and the consumption function FC, and sending the encoded byte array to the server via the network; The process of binary encoding the consumption function FC includes: When Java is used as the client-side development language, the lambda technology of JDK 8 and above is used to implement the consumer function. The remote service proxy of the client serializes the consumer function to obtain a binary encoded array. The implementation of the consumption function using JDK 8 and above lambda technology includes: Define a new lambda expression interface. The characteristics of the new lambda expression interface are: an interface annotated with @FunctionalInterface; the interface has only one method; and the interface is derived from the java.io.Serializable interface.

6. A computer device, characterized in that, It includes a memory and a processor, the memory storing program instructions, and the processor executing the program instructions to perform the method according to any one of claims 1 to 5.

Images