Data query and retrieval method and equipment

Abstract

The invention relates to the technical field of data query and retrieval, in particular to a data query and retrieval method and equipment. Firstly, a client develops a consumption function FCy (x, c) according to a remote service specification, x represents an input parameter of the consumption function FC, c represents a context executed by the consumption function, and y represents an execution result of the consumption function. Then, the client calls a remote service, and transmits the input parameter x and the consumption function FC to the server through a network; and finally, the client receives an execution result y returned by the server. According to the method, various variable and uncertain data retrieval requirements can be met, and the stability of a data retrieval server program is improved. The method is suitable for data centers, data asset management and other application scenes, and is convenient to popularize and use.

Description

Technical Field

[0001] The present invention relates to the technical field of data query and retrieval, and in particular to a method and device for data query and retrieval. Background Art

[0002] Cross-application data retrieval, analysis, and processing technology generally involves an application system (client) sending a data query and retrieval request to another application system (server), and then the server completes the query, analysis, and processing according to the client's requirements to obtain the processing results, and finally returns the processing results to the client.

[0003] In the data processing process described above, the server program must be deeply involved in the client's business logic, otherwise it will not be possible to obtain results that meet customer needs. This means that the server must develop as many data processing logics as there are data demand models on the client. In short, when the client's needs change, the server's data processing logic must change accordingly. Although the server can provide plug-ins to expand service capabilities to solve adaptability problems, even so, the workload of organizing and coordinating the development and testing of both ends is large, and it is difficult to meet the uncertain characteristics of data retrieval needs that may change at any time. Summary of the invention

[0004] In view of the deficiencies in the prior art, the purpose of the present invention is to provide a method and device for data query and retrieval, which can not only meet the needs of data retrieval being updated at any time, but also significantly reduce the operation and maintenance costs of the server.

[0005] To achieve the above object, the present invention is implemented through the following technical solutions:

[0006] A method for data query and retrieval, comprising:

[0007] The client develops the consumption function FCy(x,c) according to the remote service specification, where x represents the input parameter of the consumption function FC, c represents the context of the consumption function execution, and the context provides the services and resources required for the consumption function to run, and y represents the result of the consumption function execution;

[0008] The client calls the remote service and transmits the input parameter x and the consumption function FC to the server through the network;

[0009] The client receives the execution result y returned from the server.

[0010] Furthermore, the process of transmitting the input parameter x and the consumption function FC to the server through the network includes: binary encoding the input parameter x and the consumption function FC, and sending the encoded byte array to the server through the network.

[0011] Furthermore, the process of binary encoding the consumption function FC includes:

[0012] When Java is used as the client development language, the lambda technology of JDK8 and above is used to implement the consumption function. The client's remote service agent serializes the consumption function to obtain a binary-encoded array;

[0013] The consumer function source code and the client source code are mixed and stored in the same Java file;

[0014] The lambda technology using JDK8 and above to implement the consumption function includes:

[0015] Define a new lambda expression interface. The characteristics of the new lambda expression interface are: an interface annotated with @FunctionalInterface; the interface has only one method; the interface is derived from the java.io.Serializable interface.

[0016] A method for data query and retrieval, comprising:

[0017] The server models the original data to obtain a data model;

[0018] The server develops a service for loading data according to the data model;

[0019] The server-side remote service receives the input parameter x and the consumption function FCy(x,c) from the client, where FC represents the consumption function, x represents the input parameter of the consumption function FC, c represents the context of the consumption function execution, and the context provides the services and resources required for the consumption function to run. y represents the result of the consumption function execution.

[0020] The server builds the environment and context c required for the execution of the consumer function;

[0021] The server calls the consumption function FCy(x,c). The consumption function uses the input parameter x and the services and resources provided by the context c to retrieve, analyze and process the data and obtain the execution result y of the consumption function.

[0022] The server returns the execution result y to the client.

[0023] Furthermore, the processing process of the server-side remote service receiving the input parameter x and the consumption function FCy(x,c) from the client includes:

[0024] The server-side remote service receives the input parameter x and the binary encoding array of the consumption function FCy(x,c) sent by the client;

[0025] The server decodes the binary encoding array and restores the input parameter x and the consumption function FCy(x,c);

[0026] The server loads the consumption function FCy(x,c).

[0027] Furthermore, before returning the execution result y to the client, the method further includes: uninstalling the execution environment and context c of the consumption function.

[0028] Furthermore, the load consumption function FCy(x,c) comprises:

[0029] If the received consumption function is written in a scripting language, the server loads the consumption function through the script engine;

[0030] If the received consumption function is written in a non-scripting language and provided to the server in the form of string source code, the server uses a compiler to compile the source code in real time, generates executable code and then loads it;

[0031] If the received consumption function is written in a non-scripting language, the consumption function is serialized and binary-encoded, and the server uses deserialization to restore and load the consumption function.

[0032] Furthermore, the server uses deserialization to restore and load the consumption function, and the processing includes:

[0033] When Java is used as the server development language:

[0034] S1. Parse the class name of the consumer function from the binary array;

[0035] a) Load the class according to the consumer function class name;

[0036] b) If no exception is thrown by the system, go to step S2;

[0037] c) The system throws a class not found exception, and checks whether the class name in the exception is consistent with the class name uploaded by the client last time. If they are consistent, execute step S3;

[0038] d) Save the scene, ask the client to upload the class file of the class that was not found, and recycle the current processing thread;

[0039] e) Receive the class file uploaded by the client and save it to the local file system;

[0040] f) Allocate processing threads, restore the scene, and execute step a);

[0041] S2. The consumption function class is loaded successfully and the subsequent processing continues;

[0042] S3. Failed to load the consumer function class or other classes that the consumer function depends on, throw an exception, and terminate the call processing.

[0043] Furthermore, it also includes:

[0044] The server creates a security sandbox. The following steps are implemented in the created security sandbox:

[0045] Receive the input parameter x and consumption function FCy(x,c) from the client;

[0046] Build the environment and context c required for the execution of the consumer function;

[0047] Call the consumption function FCy(x,c) with the input parameter x and context c to get the execution result;

[0048] Unload the execution environment and context c of the consumer function;

[0049] Return the execution result y to the client.

[0050] A computer device includes a memory and a processor, wherein the memory stores program instructions, and the processor runs the program instructions to execute any one of the above methods.

[0051] Compared with the prior art, the present invention has the following beneficial effects:

[0052] The present invention adopts the method of developing consumption functions by the client, running the consumption functions on the server, and flexibly organizing and using various services and resources provided by the server according to data retrieval requirements. The server does not need to pay attention to the execution logic of the consumption function (i.e., the retrieval requirements of the client), and only needs to return the execution results of the consumption function to the client. According to the above processing strategy, the server can form a set of fixed processing logic, combined with the use of consumption functions, to meet the client's variable data retrieval, analysis and processing requirements. Thereby improving the data retrieval service capability, maintaining the stability of the server program, reducing the workload of organizing and coordinating the development, debugging and deployment of the systems on both ends, and reducing the operation and maintenance costs of the server. The present invention can be applied to application scenarios such as data centers and data asset management, and is easy to promote and use. BRIEF DESCRIPTION OF THE DRAWINGS

[0053] Figure 1 It is an execution logic diagram of an embodiment of the data query and retrieval of the present invention;

[0054] Figure 2 It is an execution logic diagram of another embodiment of the data query and retrieval of the present invention;

[0055] Figure 3 It is a screenshot of the Java Lambda technology mixed consumption function and client code described in the present invention.

[0056] Figure 4 It is the class loading algorithm logic diagram of the present invention;

[0057] Figure 5 It is a data query retrieval execution sequence diagram of the present invention. DETAILED DESCRIPTION

[0058] The present invention will be further described below in conjunction with the accompanying drawings and specific implementation methods. The terms such as "upper", "inner", "middle", "left", "right" and "one" cited in this specification are only for the convenience of description, and are not used to limit the scope of the present invention. The change or adjustment of their relative relationship shall also be regarded as the scope of the present invention without substantially changing the technical content.

[0059] The data query and retrieval method described in the present invention involves two ends, namely the client and the server. Figure 5 As shown, the following will describe the specific data query and retrieval methods from the perspectives of the client and the server respectively.

[0060] From the perspective of the client, the data query and retrieval method of the present invention includes the following steps:

[0061] S1. The client develops a consumption function FCy(x,c) according to the remote service specification, where x represents the input parameter of the consumption function FC, c represents the context of the consumption function execution, which is used to provide the consumption function with authorized local services and resources provided by the server, and y represents the execution result of the consumption function.

[0062] S2. The client calls the remote service and transmits the input parameter x and the consumption function FC to the server through the network. The specific processing process includes:

[0063] Binary encode the input parameter x and the consumption function FC, and send the encoded byte array to the server through the network.

[0064] The process of binary encoding the consumption function FC includes:

[0065] When Java is used as the client development language, the lambda technology of JDK8 (and above) is used to implement the consumption function. The client's remote service agent serializes the consumption function to obtain a binary-encoded array; the source code of the consumption function and the client source code are mixed and coexist in the same Java file, reducing the workload of separate compilation and packaging, improving the convenience of code debugging, and improving the readability and manageability of the source code.

[0066] The lambda technology using JDK8 and above to implement the consumption function includes:

[0067] Define a new lambda expression interface. The characteristics of the new lambda expression interface are: an interface annotated with @FunctionalInterface; the interface has only one method; the interface is derived from the java.io.Serializable interface.

[0068] The lambda expression of the present invention is different from the prior art. JDK provides a series of lambda expression interfaces in the java.util.function package. Compared with ordinary interfaces, the lambda expression interface has the following two typical features:

[0069] Feature 1: The interface class is annotated with @FunctionalInterface, and the compiler recognizes the interface as a lambda expression interface based on the @FunctionalInterface annotation.

[0070] Feature 2: The interface has one and only one method, which means that the interface is used as a function.

[0071] The anonymous inner class generated after the closure written according to the above lambda expression interface is compiled does not have the serialization capability. The lambda expression interface in the present invention adds a new feature to the prior art:

[0072] Feature 3: The interface is derived from the java.io.Serializable interface.

[0073] The newly added features will cause the anonymous inner class generated after the closure is compiled to have serialization capabilities and can be used as the carrier of the consumer function, thereby expanding the scope of use of lambda expressions; combined with the concise writing characteristics of lambda expressions, the server-side execution code (consumer function) and the client code are innovatively seamlessly bonded together, presenting a Figure 3 The effect shown.

[0074] S3. The client receives the execution result y returned from the server.

[0075] From the perspective of the server, the data query and retrieval method of the present invention includes the following steps:

[0076] S1. The original data is provided by the server, and the server models the original data to obtain a data model.

[0077] S2. The server develops the services required to load data according to the data model.

[0078] S3. The server-side remote service receives the input parameter x and the consumption function FCy(x,c) from the client, where FC represents the consumption function, x represents the input parameter of the consumption function FC; c represents the context of the execution of the consumption function, which is used to provide the consumption function with authorized local services and resources, and provide the services required for the operation of the consumption function. The context includes data services, cache services, temporary file services, etc.; y represents the execution result of the consumption function.

[0079] The server-side remote service receives the input parameter x from the client and the processing process of the consumption function FCy(x,c) includes:

[0080] The server-side remote service receives the input parameter x and the binary encoding array of the consumption function FCy(x,c) sent by the client;

[0081] The server decodes the binary code and restores the input parameter x and the consumption function FCy(x,c);

[0082] Load the consumption function FCy(x,c).

[0083] The load consumption function FCy(x,c) comprises:

[0084] If the received consumption function is written in a scripting language, the server loads the consumption function through the script engine;

[0085] If the received consumption function is written in a non-scripting language and provided to the server as a string source code, the server will use a compiler to compile the source code in real time and load the generated executable code;

[0086] If the received consumption function is written in a non-scripting language, the consumption function is serialized and binary-encoded, and the server uses deserialization to restore and load the consumption function.

[0087] The consumption function developed in Java language, such as Figure 4 As shown, the server uses a class loading algorithm to implement deserialization loading consumption functions, including:

[0088] Step 1. Parse the class name of the consumer function from the binary array;

[0089] a) Load the class according to the consumer function class name;

[0090] b) If no exception is thrown by the system, go to step 2;

[0091] c) The system throws a class not found exception, and checks whether the class name in the exception is consistent with the class name uploaded by the client last time. If they are consistent, proceed to step 3;

[0092] d) Save the scene, require the client to upload the class file of the class that was not found, and recycle the current processing thread so that it can continue to execute after the interrupt processing is completed, and recycle the thread to avoid occupying resources;

[0093] e) Receive the class file uploaded by the client and save it to the local file system;

[0094] f) Allocate a processing thread, restore the scene, and execute step a), at which time another thread can be used to continue executing the task before the interruption;

[0095] Step 2: Successfully load the consumption function class and continue with subsequent processing;

[0096] Step 3: Fail to load the consumption function class or other classes that the consumption function depends on, throw an exception, and terminate the call processing.

[0097] The class loading of the present invention is different from the prior art. The prior art generally loads classes from local Jar files, or downloads Jar files from network server nodes (centers) and then loads them. The main body of the transmission is the Jar file, and the name of the downloaded Jar file is fixed and known when the program is compiled; the class loading process is synchronous, and when the class cannot be found, the program exception is directly thrown and the execution is suspended. The class of the present invention comes from the peer (client), the location of the client is uncertain, and the class to be loaded is the consumption function class and the introduction class of the class, which cannot be determined when the program is compiled; the main body of the transmission is the .class class file, and the .class class file is managed locally in the Jar directory mode; the class loading process is asynchronous, and when the class loader cannot find the class to be loaded in the local file system, it will ask the peer to provide the required .class class file, and save the scene at the same time, recycle the current thread resources, and resume the class loading process until the peer provides the required class through the network. The class loading of the present invention can save network bandwidth resources, and the processing mechanism is more flexible, which is suitable for realizing the loading of consumption function classes and other reference classes in the remote service call process.

[0098] S4. The server builds the environment and context c required for the execution of the consumer function.

[0099] S5. The server calls the consumption function FCy(x,c). The consumption function uses the input parameter x and the services and resources provided by the context c to retrieve, analyze and process the data to obtain the execution result y of the consumption function.

[0100] S6. The server unloads the execution environment and context c of the consumer function. The existence cycle of the consumer function is only between loading and unloading, saving server resources.

[0101] S6. The server returns the execution result y to the client.

[0102] The data query and retrieval method described in the present invention also includes: the server performs a security audit or code audit on the input parameters and consumption functions, and performs an out-of-domain data security audit on the calculation results returned to the client to enhance the security of the system and data.

[0103] The data query and retrieval method described in the present invention can be applied in data asset transaction scenarios, following the principle of "data does not leave the domain", and the consumption function loading and execution are placed in a security sandbox. Security sandbox technology is a security mechanism used to isolate running programs. Its purpose is to limit the access rights of untrusted processes or untrusted code at runtime to prevent them from posing threats to the host and system, and ensure system and data security. Therefore, the data query and retrieval method described in the present invention also includes creating a security sandbox, such as Figure 2 As shown; when Java is used as the development language, Java's built-in security sandbox technology can be used.

[0104] The following steps are implemented in the security sandbox:

[0105] The server-side remote service receives the input parameter x and the consumption function FCy(x,c) from the client;

[0106] The server builds the environment and context c required for the execution of the consumer function;

[0107] The server calls the consumer function FCy(x,c) with the input parameter x and context c to get the execution result;

[0108] The server unloads the execution environment and context c of the consumer function;

[0109] The server returns the execution result y to the client.

[0110] A computer device includes a memory and a processor, wherein the memory stores program instructions, and the processor runs the program instructions to execute the data query and retrieval method described above from a client perspective or a server perspective.

[0111] The data query and retrieval method of the present invention can be understood figuratively using a self-service kitchen in daily life. The self-service kitchen (server) provides customers (clients) with a number of ingredients (data), kitchen utensils (context / service), and kitchen (environment / safety sandbox) for processing dishes. Customers bring their own chefs (consumption functions), and customers make dish requirements (input parameters) to the chefs (consumption functions). After disinfection and other processes (audit), the chefs (consumption functions) enter the kitchen (environment / safety sandbox) and process the dishes in the kitchen (environment / safety sandbox). The final dish (execution result y) is delivered to the customer (client) by the waiter (audit). The benefit of a self-service kitchen is that the self-service kitchen (server) only needs to prepare ingredients (data) and kitchen utensils (context / service), and any requirements (retrieval requirements) of customers (clients) for dishes can be met.

[0112] The present invention divides the data query and retrieval process into a personalized part (consumption function) and a public part (services, resources, data); the personalized part is developed by the client, and the services and data developed by the server are universal and highly standardized. This method can be used to meet various changing and uncertain data retrieval needs and improve the stability of the data retrieval server program; at the same time, such a division of labor saves the demand communication and joint debugging costs of both parties in data retrieval, shortens the delivery period of the client system, and saves the operation and maintenance costs of the server system; it is suitable for various application scenarios such as data centers and data asset management, and is easy to promote and use.

[0113] This method aims at the Java development language, improves the usage of Lambda expression technology and the class asynchronous loading algorithm; realizes the embedding of the consumption function executed by the server into the client code, which is not only easy to read and understand, but also more convenient for debugging, packaging and deployment, thereby improving the efficiency of software development; in addition, combined with Java's built-in security sandbox technology, it can take into account the unity of convenience and security at the same time.

[0114] This method is used in the data asset transaction process to build the standards of data products based on public services and data, which is conducive to the formulation of the price system of data products, the promotion of large-scale transactions, and the formation of a data transaction ecosystem.

[0115] The implementation methods of the present invention are not limited to this. According to the above content of the present invention, using common technical knowledge and customary means in the field, without departing from the above basic technical ideas of the present invention, the present invention can also make other various forms of modification, replacement or combination, all of which fall within the scope of protection of the present invention.

Claims

1. A method for data query and retrieval, characterized in that: include: The client develops the consumption function FCy(x,c) according to the remote service specification, where x represents the input parameter of the consumption function FC, c represents the context of the consumption function execution, and the context provides the services and resources required for the consumption function to run, and y represents the result of the consumption function execution; The client calls the remote service and transmits the input parameter x and the consumption function FC to the server through the network; The client receives the execution result y returned from the server.

2. A data query and retrieval method according to claim 1, characterized in that: The process of transmitting the input parameter x and the consumption function FC to the server through the network includes: binary encoding the input parameter x and the consumption function FC, and sending the encoded byte array to the server through the network.

3. A method for data query and retrieval according to claim 2, characterized in that: The process of binary encoding the consumption function FC includes: When Java is used as the client development language, the lambda technology of JDK8 and above is used to implement the consumption function. The client's remote service agent serializes the consumption function to obtain a binary-encoded array; The consumer function source code and the client source code are mixed and stored in the same Java file; The lambda technology using JDK8 and above to implement the consumption function includes: Define a new lambda expression interface. The characteristics of the new lambda expression interface are: an interface annotated with @FunctionalInterface; the interface has only one method; the interface is derived from the java.io.Serializable interface.

4. A method for data query and retrieval, characterized in that: include: The server models the original data to obtain a data model; The server develops a service for loading data according to the data model; The server-side remote service receives the input parameter x and the consumption function FCy(x,c) from the client, where FC represents the consumption function, x represents the input parameter of the consumption function FC, c represents the context of the consumption function execution, and the context provides the services and resources required for the consumption function to run. y represents the result of the consumption function execution. The server builds the environment and context c required for the execution of the consumer function; The server calls the consumption function FCy(x,c). The consumption function uses the input parameter x and the services and resources provided by the context c to retrieve, analyze and process the data and obtain the execution result y of the consumption function. The server returns the execution result y to the client.

5. A data query and retrieval method according to claim 4, characterized in that: The processing process of the server-side remote service receiving the input parameter x and the consumption function FCy(x,c) from the client includes: The server-side remote service receives the input parameter x and the binary encoding array of the consumption function FCy(x,c) sent by the client; The server decodes the binary encoding array and restores the input parameter x and the consumption function FCy(x,c); The server loads the consumption function FCy(x,c).

6. A method for data query and retrieval according to claim 5, characterized in that: Before returning the execution result y to the client, the method also includes: uninstalling the execution environment and context c of the consumption function.

7. A method for data query and retrieval according to claim 5, characterized in that: The load consumption function FCy(x,c) comprises: If the received consumption function is written in a scripting language, the server loads the consumption function through the script engine; If the received consumption function is written in a non-scripting language and provided to the server in the form of string source code, the server uses a compiler to compile the source code in real time, generates executable code and then loads it; If the received consumption function is written in a non-scripting language, the consumption function is serialized and binary-encoded, and the server uses deserialization to restore and load the consumption function.

8. A method for data query and retrieval according to claim 7, characterized in that: The processing process of the server using deserialization to restore and load the consumption function includes: When Java is used as the server development language: S1. Parse the class name of the consumer function from the binary array; a) Load the class according to the consumer function class name; b) If no exception is thrown by the system, go to step S2; c) The system throws a class not found exception, and checks whether the class name in the exception is consistent with the class name uploaded by the client last time. If they are consistent, execute step S3; d) Save the scene, ask the client to upload the class file of the class that was not found, and recycle the current processing thread; e) Receive the class file uploaded by the client and save it to the local file system; f) Allocate processing threads, restore the scene, and execute step a); S2. The consumption function class is loaded successfully and the subsequent processing continues; S3. Failed to load the consumer function class or other classes that the consumer function depends on, throw an exception, and terminate the call processing.

9. A method for data query and retrieval according to claim 6, characterized in that: Also includes: The server creates a security sandbox. The following steps are implemented in the created security sandbox: Receive the input parameter x and consumption function FCy(x,c) from the client; Build the environment and context c required for the execution of the consumer function; Call the consumption function FCy(x,c) with the input parameter x and context c to get the execution result; Unload the execution environment and context c of the consumer function; Return the execution result y to the client.

10. A computer device, characterized in that: The method comprises a memory and a processor, wherein the memory stores program instructions, and the processor runs the program instructions to execute any one of the methods of claims 1 to 9.

Images