Object storage retrieval optimization method and system based on function calculation

Abstract

The embodiment of the invention provides an object storage and retrieval optimization method and system based on function computing, the method is applied to the object storage and retrieval optimization system based on function computing, the system comprises an object storage access gateway, an object gateway, a function computing module and a retrieval acceleration module which are connected in sequence, and the object gateway is further connected with a storage engine; the method comprises the following steps: an object storage access gateway obtains a retrieval request input by a user and determines a request type; if the retrieval request is of the first type, the object storage access gateway sends the retrieval request to an object gateway, the object gateway executes and completes operation corresponding to the retrieval request in a storage engine, and after the operation is completed, event information is generated through an event trigger pre-configured on the object gateway; the function calculation module operates a preset function operator according to the event information to obtain a calculation result of the function operator; and the retrieval acceleration module adds the calculation result of the function operator to a pre-obtained initial table to obtain final table data.

Description

Technical Field

[0001] The present invention relates to the field of novel information technology, and in particular to an object storage retrieval optimization method and system based on function calculation. Background Art

[0002] Object storage has the characteristics of highly reliable services, flexible scalability, and flat data management, and has quickly become the most popular choice for unstructured massive data storage. The flat data management method enhances the scalability of the system and improves data access efficiency, but at the same time it exposes poor retrieval capabilities. Currently, mainstream object storage only supports retrieval based on object name prefixes, and cannot meet complex analytical query requests such as range retrieval, sorting, aggregation, and association. This is very unfriendly to users who need to index, analyze, and mine massive data.

[0003] At present, the main approaches to complex retrieval requirements of object storage are: 1. First traverse all metadata, and then count, sort, and analyze the metadata. This approach is inefficient and has poor relevance to historical data. It is essentially an offline retrieval; 2. Modify the object storage gateway and write the relevant metadata information to the relational database at the same time as the data is actually written into the storage engine. This approach is highly invasive to the data reading and writing process, and will more or less affect the efficiency and stability of the original process; at the same time, this is a fixed metadata written as a retrieval condition, which cannot meet the flexible and changeable retrieval needs of users in different storage scenarios. Summary of the invention

[0004] The embodiment of the present invention provides an object storage retrieval optimization method and system based on function computing to solve the problems of poor object storage retrieval methods and inefficient complex analysis retrieval in the prior art.

[0005] The embodiment of the present invention discloses an object storage retrieval optimization method based on function computing, and the method is applied to an object storage retrieval optimization system based on function computing, wherein the system comprises: an object storage access gateway, an object gateway, a function computing module, and a retrieval acceleration module connected in sequence, and the object gateway is also connected to a storage engine; the method comprises:

[0006] The object storage access gateway obtains the search request input by the user and determines the request type of the search request;

[0007] If the retrieval request is of the first type, the object storage access gateway sends the retrieval request to the object gateway, and the object gateway executes and completes the operation corresponding to the retrieval request in the storage engine, and after the operation is completed, generates event information corresponding to the retrieval request through an event trigger pre-configured on the object gateway;

[0008] The function calculation module runs a preset function operator according to the event information to obtain a calculation result of the function operator;

[0009] The retrieval acceleration module adds the calculation result of the function operator to the pre-acquired initial table to obtain the final table data.

[0010] Preferably, the method further comprises:

[0011] If the retrieval request is of the second type, the object storage access gateway sends the retrieval request to the retrieval acceleration module. The retrieval acceleration module determines whether there is a pre-acquired initial table. If there is a pre-acquired initial table, the operation corresponding to the retrieval request is executed through TiDB to obtain the result data.

[0012] Preferably,

[0013] The retrieval request is an HTTP request;

[0014] Accordingly,

[0015] If the first query parameter in the search request is not Analytical, the search request type is determined to be the first type;

[0016] If the first query parameter in the search request is Analytical, the search request type is determined to be the second type.

[0017] Preferably,

[0018] The event information corresponding to the search request includes: operation action, operation object, and operation object attribute;

[0019] The operation action is a specific action of the operation corresponding to the search request;

[0020] The operation object is the data object involved in performing the operation action;

[0021] The operation object attributes include metadata of the operation object.

[0022] Preferably, the function calculation module runs a preset function operator according to the event information to obtain a calculation result of the function operator, which specifically includes:

[0023] It is determined whether the operation action in the event information is a file writing operation. If so, a preset function operator is run to obtain a calculation result of the function operator.

[0024] Preferably,

[0025] The function operator is used to determine the search field according to the operation object attribute in the event information and the preset search logic, and use the search field determined by the function operator as the calculation result of the function operator;

[0026] The search logic is one of the first search logic, the second search logic, and the third search logic;

[0027] The first search logic is: using the operation object attribute in the event information as a search field;

[0028] The second search logic is: using the operation object attributes and predefined metadata and / or predefined tags in the event information as search fields;

[0029] The third search logic is: using the operation object attributes and object attribute data in the event information as search fields;

[0030] The object attribute data is obtained by processing the operation object in the event information through a designated platform connected to the function calculation module.

[0031] Preferably,

[0032] The initial table is obtained after Index Proxy in the retrieval acceleration module receives the preset table structure configuration information and initializes the table structure on TiDB;

[0033] The table structure configuration information includes: the data model and the attributes of the search field and the index configured for the specified search field;

[0034] The data model is a table, including a table name, field names, and field constraints.

[0035] On the other hand, this embodiment also provides an object storage retrieval optimization system based on function computing, the system comprising:

[0036] The management console is used to receive the function information selected by the user in the management console and the function code entered by the user, and obtain the function operator according to the function information and the function code; it is also used to obtain the table structure configuration information entered by the user in the management console, and send the table structure configuration information to the configuration center;

[0037] The configuration center is used to configure event triggers in the object gateway; and is also used to send the table structure configuration information to the retrieval acceleration module;

[0038] The retrieval acceleration module is used to obtain the initial table through Index Proxy and TiDB according to the table structure configuration information;

[0039] The object storage access gateway is used to obtain a search request input by a user, determine a request type of the search request, and send the search request to the object gateway when the search request is of the first type;

[0040] The retrieval request is an HTTP request; if the first query parameter in the retrieval request is not Analytical, the retrieval request type is determined to be the first type;

[0041] The object gateway is used to execute and complete the operation corresponding to the retrieval request on the storage engine, and after the operation is completed, generate event information corresponding to the retrieval request through an event trigger on the object gateway;

[0042] A function calculation module, used to run the function operator according to the event information to obtain a calculation result of the function operator;

[0043] The retrieval acceleration module is also used to add the calculation results of the function operator to the initial table to obtain final table data.

[0044] Preferably,

[0045] The object storage access gateway is further configured to send the retrieval request to the retrieval acceleration module when the retrieval request is of the second type;

[0046] If the first query parameter in the search request is Analytical, the search request type is determined to be the second type;

[0047] The retrieval acceleration module is also used to determine whether an initial table exists. If an initial table exists, the operation corresponding to the retrieval request is executed through TiDB to obtain result data corresponding to the retrieval request.

[0048] Preferably,

[0049] The function running information includes: virtual hardware resources of the function running environment, function language running environment, and initial concurrent parameters;

[0050] The table structure configuration information includes: the data model and the attributes of the search field and the index configured for the specified search field; the data model is a table, including the table name, field name, and field constraints.

[0051] The embodiments of the present invention include the following advantages:

[0052] A method and system for optimizing object storage retrieval based on function computing in this embodiment automatically triggers function computing according to the type of retrieval request after the retrieval request is completed, thereby reducing the intermediate waiting time between the retrieval request and the function operator calculation and improving the response speed. In addition, a method for optimizing object storage retrieval based on function computing in this embodiment can avoid unnecessary full scan operations by judging the type of retrieval request through the object storage access gateway, thereby reducing unnecessary computing consumption and data transmission burden. The function computing module executes the function operator according to the type of retrieval request and supports customized data processing methods, so that the object storage retrieval optimization system based on function computing provided in this embodiment can flexibly adapt to different query scenarios. The retrieval acceleration module uses the results of function computing to dynamically update the initial table data to form the final table data. This caching and preprocessing mechanism effectively reduces repeated calculations, is particularly suitable for high-frequency query scenarios, and can significantly improve retrieval performance.

[0053] The object storage retrieval optimization method based on function computing in this embodiment uses the function computing module to provide users with customizable retrieval conditions. Users can flexibly define retrieval logic, expand the online retrieval capability of object storage, and improve the retrieval efficiency in complex query scenarios.

[0054] The object storage retrieval optimization method based on function computing of the present embodiment utilizes the event-driven mechanism of object storage, and triggers the retrieval calculation module after the data reading and writing process is completed by listening to the event trigger pre-configured on the object gateway. It not only enriches the retrieval capability, but also reduces the intrusiveness to the original data process, ensuring stability and high availability. In addition, the object storage retrieval optimization method based on function computing of the present embodiment utilizes the object storage access gateway as a unified retrieval request entry, parses and classifies the retrieval request input by the user, quickly determines the request type (such as the first type and the second type of retrieval request), and distributes different types of retrieval requests to the most appropriate module (such as the storage engine or the retrieval acceleration module), which reduces the traversal pressure of subsequent modules (especially the object gateway) and reduces the waiting time and processing time of the request. BRIEF DESCRIPTION OF THE DRAWINGS

[0055] Figure 1 It is a flow chart of an object storage retrieval optimization method based on function computing provided in an embodiment of the present invention;

[0056] Figure 2 It is a schematic diagram of the object storage retrieval optimization system based on function computing provided in an embodiment of the present invention when processing a file write request;

[0057] Figure 3 Schematic diagram of a function computing module in an object storage retrieval optimization system based on function computing provided in an embodiment of the present invention;

[0058] Figure 4 is a schematic diagram of the object storage retrieval optimization system based on function computing provided in an embodiment of the present invention when processing a file read request;

[0059] Figure 5 is a schematic diagram of an object storage retrieval optimization system based on function computing provided in an embodiment of the present invention when processing a query request;

[0060] Figure 6 is a schematic diagram of the structure of an object storage retrieval optimization system based on function computing provided in an embodiment of the present invention;

[0061] Figure 7 It is a flowchart of retrieval request classification and processing of an object storage retrieval optimization system based on function computing provided in an embodiment of the present invention. DETAILED DESCRIPTION

[0062] In order to make the above-mentioned objects, features and advantages of the present invention more obvious and easy to understand, the present invention is further described in detail below with reference to the accompanying drawings and specific embodiments.

[0063] Function Compute is an elastic computing service based on the Serverless architecture and a specific implementation of FAAS (Function as a Service). The platform allows users to write, run, and expand functions without having to manage underlying servers and infrastructure. Users only need to upload business code and set trigger conditions. The platform will dynamically allocate computing resources based on the actual request volume and complete automatic scheduling and scaling.

[0064] HTAP is a new type of data processing architecture that can support both online transaction processing (OLTP) and online analytical processing (OLAP). Traditional databases are usually divided into two types: OLTP and OLAP. OLTP systems are used to process high-concurrency, low-latency transaction operations, such as order processing and inventory updates; while OLAP systems are used for large-scale data analysis and queries, such as report generation and trend analysis. The HTAP architecture combines the advantages of both, and can process transaction requests and complete analytical queries in the same system.

[0065] Reference Figure 1 This embodiment provides an object storage retrieval optimization method based on function computing, and the method is applied to an object storage retrieval optimization system based on function computing. The system includes: an object storage access gateway, an object gateway, a function computing module, and a retrieval acceleration module connected in sequence. The object gateway is also connected to a storage engine; the method includes:

[0066] S1. The object storage access gateway obtains a search request input by a user and determines a request type of the search request;

[0067] The retrieval request is an HTTP request; it should be noted that the HTTP request is a standard protocol for communication between the client and the server. It is based on a stateless communication protocol, which means that each request is independent and the server does not store any information about the previous request. HTTP requests are used to obtain resources from the server, send data, or submit forms to the server. An HTTP request includes multiple components, such as: request line (Request Line), request header (Request Headers), query parameters (Query Parameters), and request body (Request Body). The query parameters in the HTTP request are data located after the "?" in the URL and consist of key-value pairs. Multiple query parameters are separated by the & symbol. In this embodiment, the first query parameter in the HTTP request is used to determine the type of request.

[0068] Accordingly, in this embodiment, if the first query parameter in the search request is not Analytical, the search request type is determined to be the first type; if the first query parameter in the search request is Analytical, the search request type is determined to be the second type.

[0069] In this embodiment, different types of search requests are distinguished by the first query parameter in the search request, so that the system can select different optimization strategies in subsequent processing. Through classification, the system can respond to different types of requests more accurately and avoid unnecessary processing or redundant operations. In addition, in this embodiment, the request type is determined by the first query parameter of the HTTP request, without traversing all parameters, and the request type can be quickly identified.

[0070] S2, such as Figure 2 As shown, if the retrieval request is of the first type, the object storage access gateway sends the retrieval request to the object gateway, and the object gateway executes and completes the operation corresponding to the retrieval request in the storage engine, and after the operation is completed, generates event information corresponding to the retrieval request through an event trigger pre-configured on the object gateway;

[0071] The event information corresponding to the search request includes: operation action, operation object, and operation object attribute;

[0072] The operation action is a specific action of the operation corresponding to the search request;

[0073] The operation object is a data object involved in executing the operation action; for example, in an object storage system, the operation object may be a specific file, directory, database table, etc.

[0074] The operation object attributes include metadata of the operation object. The metadata in this embodiment includes: basic information of the object: such as file size, file format, creation time, modification time, etc.

[0075] In this embodiment, the event trigger is pre-configured in the object gateway to monitor the operation results of the storage engine. Once the storage engine completes the operation, the event trigger automatically generates event information corresponding to the operation. The event trigger can capture storage operations in real time without additional manual intervention during use. Through the pre-configured event trigger, the results of the storage engine operation can be associated with the metadata, which provides detailed context information for subsequent retrieval acceleration and optimization.

[0076] S3. The function calculation module runs a preset function operator according to the event information to obtain a calculation result of the function operator; specifically, the function calculation module determines whether the operation action in the event information is a file writing operation, and if so, runs a preset function operator to obtain a calculation result of the function operator.

[0077] It should be noted that the function computing module is a component based on the Serverless architecture, which is used to handle asynchronous computing tasks. It can dynamically run function operators based on triggered events without pre-deploying servers. Specifically, the function computing module is a Serverless computing platform based on the Kubernetes cluster, which provides standardized programming interfaces and basic operating environments, supports users to implement custom function operators in different programming languages, and adjusts retrieval logic according to business needs. Through the elastic scaling capabilities of Kubernetes, resource allocation can be automatically adjusted to respond to requests with optimal performance. At the same time, the integrated monitoring, alarm, and logging functions provide users with powerful observability and management tools, simplify operation and maintenance operations, and allow users to focus on the development of business logic without having to worry about the management of the underlying infrastructure.

[0078] In this embodiment, the event information is generated by the event trigger after the file write operation is completed, and the function calculation module will determine the operation action in the event information. The function operator is a predefined calculation function used to determine the search field based on the operation object attributes and search logic in the event information. These search fields will be used for subsequent data query and analysis.

[0079] The function operator is used to determine the search field according to the operation object attribute in the event information and the preset search logic, and use the search field determined by the function operator as the calculation result of the function operator;

[0080] The search logic is one of the first search logic, the second search logic, and the third search logic;

[0081] The first search logic is: using the operation object attribute in the event information as a search field;

[0082] The second search logic is: using the operation object attributes and predefined metadata and / or predefined tags in the event information as search fields;

[0083] The third search logic is: using the operation object attributes and object attribute data in the event information as search fields;

[0084] The object attribute data is obtained by processing the operation object in the event information through a designated platform connected to the function calculation module.

[0085] In this embodiment, the object storage system realizes automatic processing from file writing to retrieval field determination through the combination of event-driven mechanism and function computing module, which can dynamically adapt to different types of requests and greatly improve the query efficiency and user experience of the system.

[0086] For an example, see Figure 3 After the user uploads, overwrites, or deletes an object through the first type of search request, the event trigger immediately triggers the event-driven mechanism. The event-driven mechanism calls the function computing module and passes event information, including the object's operation attributes (such as object name, size, etc.).

[0087] Then, according to the pre-set search logic, the function operator extracts relevant attributes from the event information and generates search fields. There are three types of search logic in this embodiment:

[0088] First search logic: directly use the operation object attributes in the event information as search fields. This usually includes natively supported attributes such as size (file size), createTime (creation time), etag (entity tag), storageClass (storage class), etc.

[0089] Second search logic: In addition to using the operation object attributes, user-defined metadata and / or tags are also extracted as search fields. This part supports user-defined configurations, such as user-defined metadata and object tags. For example, in addition to metadata (such as size, createTime, etc.), users can define additional metadata key-value pairs as needed to describe specific attributes or application scenarios of files. For example: x-amz-meta-project:AI-Research——Indicates that this file belongs to the "AI-Research" project. x-amz-meta-owner:John Doe——Identifies the owner of the file as John Doe. User-defined metadata is used for file classification, identification, and retrieval in business scenarios. User-defined metadata or tags, these data are added to object storage through user configuration or API calls.

[0090] The third retrieval logic: Based on the extraction of the attributes of the operation object, further use the external computing power platform or AI platform to analyze the object and extract additional object attribute data, such as image feature values ​​or AI recognition labels.

[0091] From the above content, we can see that there are three sources of search fields: Native: built-in attributes of the object storage system, including size (file size), createTime (creation time), etag (entity tag), storageClass (storage class) and other basic information. Self-defining: user-defined metadata and tags, which are added to the object storage through user configuration or API calls. External: attributes calculated by external platforms, such as image feature values ​​extracted by AI platforms.

[0092] S4. The retrieval acceleration module adds the calculation result of the function operator to the pre-acquired initial table to obtain final table data.

[0093] The initial table is obtained after Index Proxy in the retrieval acceleration module receives the preset table structure configuration information and initializes the table structure on TiDB;

[0094] The table structure configuration information includes: the data model and the attributes of the search field and the index configured for the specified search field;

[0095] The data model is a table, including a table name, field names, and field constraints.

[0096] It should be noted that Index Proxy is a submodule in the retrieval acceleration module, which is mainly responsible for interacting with the database and managing indexes and table structures. Index Proxy will initialize the table structure in the database according to the configuration information pre-set by the user. TiDB is a distributed database that can handle two different types of data tasks at the same time, such as: OLTP (transaction processing): that is, processing daily database operations, such as file uploads, modifications and other requests. TiDB processes these operations very quickly and can respond to user requests in real time. OLAP (analytical processing): supports complex data analysis tasks, such as finding, filtering and counting large numbers of files or data. TiDB can perform multi-dimensional analytical queries and is suitable for processing complex searches that require a combination of multiple conditions. Therefore, TiDB can not only quickly process routine operations, but also perform in-depth data analysis. These two capabilities just meet the retrieval requirements of object storage systems.

[0097] Table structure initialization means creating a database table in TiDB and defining the table's fields and indexes according to the preset structure configuration. This step is to ensure that the structure of the database table can meet subsequent insertion and retrieval requirements. Specifically, the initial table is created in TiDB according to the set structure after the Index Proxy submodule receives the table structure configuration information. This initial table is the basis for subsequent retrieval acceleration and data insertion.

[0098] In this embodiment, by adding the calculation results to the search acceleration table, the speed of subsequent search operations can be significantly improved. This is because the acceleration table has optimized the search field and accelerated the search process through indexing.

[0099] For an example, see Figure 2 , suppose a user sends an HTTP request through a client to write a picture file sample.jpg into the object storage system (this request is a file write request). At this time, the object storage access gateway obtains the HTTP request entered by the user, and by checking that the first query parameter in the URL of the HTTP request is not Analytical, the object storage access gateway recognizes that the HTTP request is a first type request;

[0100] Then, the object storage access gateway receives the user's upload request and sends the image file sample.jpg and the request information to the object gateway. The object gateway processes the write operation of the image file sample.jpg and stores the image file sample.jpg in the storage engine. After the file is written, the event trigger set on the object gateway is activated and an event information is generated. This event information includes: operation action: file write; operation object: sample.jpg; operation object attributes include: file size: 5MB file format: jpg; creation time: xxxx year-xx month-xx day.

[0101] After receiving the event information, the function computing module will determine whether the operation is a file writing operation. Because it is a file writing operation, it will execute a preset function operator.

[0102] Assuming that the first search logic is used in the function operator at this time, then after the function operator is completed, the following search fields will be obtained: file size: 5MB file format: jpg; creation time: xxxx-xx-xx, these search fields are used as the calculation results of the function operator.

[0103] Finally, the retrieval acceleration module inserts the calculation results of the function operator into an initial table to obtain the final table data.

[0104] For another example, see Figure 4 , suppose the user sends an HTTP request (a file read request) through the client to read a picture file sample.jpg from the object storage system. The following is a detailed description of the read operation retrieval request process:

[0105] The HTTP request sent by the user is to read the file sample.jpg from the object storage system.

[0106] The object storage access gateway first obtains the HTTP request entered by the user and checks the first query parameter in the URL of the request.

[0107] Assuming that the first query parameter of the request is not "Analytical", the object storage access gateway will recognize that this is a first type HTTP request (ie, a non-analytical request).

[0108] The object storage access gateway receives the user's file read request and sends the request information to the object gateway.

[0109] The object gateway processes the file read request, obtains the corresponding image file sample.jpg from the storage engine, and returns it to the user.

[0110] In a specific embodiment, the method further comprises:

[0111] S5. If the retrieval request is of the second type, the object storage access gateway sends the retrieval request to the retrieval acceleration module. The retrieval acceleration module determines whether there is a pre-acquired initial table. If there is a pre-acquired initial table, the operation corresponding to the retrieval request is executed through TiDB to obtain the result data.

[0112] In step S5, the second type of retrieval request is processed. If the object storage access gateway recognizes that the retrieval request is of the second type (that is, the first query parameter is "Analytical"), it will directly use the initial table prepared in advance in the retrieval acceleration module to avoid repeated calculations, thereby speeding up the response speed.

[0113] For an example, see Figure 5 , suppose a user sends an HTTP request (this request is a query request, that is, the first query parameter in this HTTP request is "Analytical"), requesting an analytical query for image files in the object storage system. The user wants to perform advanced query and analysis operations based on metadata fields such as file size and file format, for example, to find all files with a size of more than 10MB and a jpg format.

[0114] The HTTP request entered by the user is as follows:

[0115] GET / api / v1 / search? type=Analytical&file_size_gt=10MB&file_format=jpg

[0116] Wherein, type=Analytical: indicates that this is a second type of search request;

[0117] file_size_gt=10MB:indicates that the query condition is that the file size is greater than 10MB.

[0118] file_format=jpg: indicates that the query condition is that the file format is jpg;

[0119] At this point, the specific process of step S5 includes:

[0120] Step 1: The object storage access gateway parses the HTTP request input by the user and detects that the first query parameter is "Analytical", thus determining that this is a second type of retrieval request.

[0121] Step 2: The object storage access gateway sends the retrieval request directly to the retrieval acceleration module.

[0122] Step 3: After receiving the HTTP request, the retrieval acceleration module first checks whether there is a pre-created and initialized initial table. The initial table is created based on the previous table structure configuration information and is usually used to store and index retrieval fields (such as file size, file format, creation time and other metadata).

[0123] Assume that the initial table structure is as follows:

[0124] File name File size File Format Creation time Picture 1 15MB jpg 2023-11-01 12：00 Picture 2 8MB png 2023-11-05 09：30 Picture 3 12MB jpg 2023-11-10 14：45

[0125] Step 4: After the retrieval acceleration module confirms the existence of the initial table, it uses the TiDB database to execute the user's analysis query request.

[0126] Step 5: The TiDB database performs the optimized search operation according to the query conditions and returns the result data that meets the conditions to the search acceleration module. Assume that the query result is:

[0127] File name File size File Format Creation time Picture 1 15MB jpg 2023-11-01 12：00 Picture 3 12MB jpg 2023-11-10 14：45

[0128] The retrieval acceleration module returns the result data to the object storage access gateway, and finally returns it to the user through an HTTP response.

[0129] In this embodiment, through the pre-created initial table and optimized index structure, the retrieval acceleration module can quickly locate qualified data, reducing the time overhead of full table scan. The second type of retrieval request supports complex queries. Combined with TiDB's OLAP capabilities, it can perform multi-condition filtering, aggregation and other analytical queries to meet users' advanced analysis needs for large-scale object storage data. As a distributed database, TiDB has good scalability and elastic scaling capabilities. Even in high-concurrency request scenarios, the system can maintain stable query performance by dynamically adjusting computing resources.

[0130] In this embodiment, through step S5, it is possible to flexibly select appropriate query optimization strategies according to different types of retrieval requests. For the second type (Analytical) requests, TiDB's distributed query capabilities and pre-configured initial tables are used to achieve fast and efficient data retrieval, greatly improving the user's query experience.

[0131] See also Figure 6 This embodiment provides an object storage retrieval optimization system based on function computing, and the system includes:

[0132] The management console is used to receive the function information selected by the user in the management console and the function code entered by the user, and obtain the function operator according to the function information and the function code; it is also used to obtain the table structure configuration information entered by the user in the management console, and send the table structure configuration information to the configuration center;

[0133] For example, this embodiment provides an object storage retrieval optimization system based on function computing. The management console is the main interface for users to interact with the system and is responsible for receiving and processing user input. The functions of the management console include: receiving function information and function code input by the user, and generating function operators based on this information. Function information includes function runtime environment configuration, such as virtual hardware resources (CPU, memory), runtime environment (Python, Java and other languages), initial concurrency parameters, etc. Function code is user-defined logic used for data processing or analytical calculations. Obtain table structure configuration information input by the user, including data model and retrieval field attributes; data model: define table name, field name, field constraints. Retrieval field attributes: configure indexes for specified fields to optimize retrieval performance. And send the table structure configuration information to the configuration center.

[0134] For example, a user enters a function code in the management console, selects the Python environment in the function information, and configures 2 vCPUs and 4 GB of memory. The user also defines a table structure that contains multiple fields and configures indexes for the specified fields to speed up retrieval.

[0135] The function running information includes: virtual hardware resources of the function running environment, function language running environment, and initial concurrent parameters;

[0136] The table structure configuration information includes: the data model and the attributes of the search field and the index configured for the specified search field; the data model is a table, including the table name, field name, and field constraints.

[0137] The configuration center is used to configure event triggers in the object gateway; and is also used to send the table structure configuration information to the retrieval acceleration module;

[0138] The configuration center in the object storage retrieval optimization system based on function computing provided in this embodiment is a core management module, which is responsible for coordinating and synchronizing the configuration information of each component.

[0139] The configuration center receives the table structure configuration information from the management console and sends it to the retrieval acceleration module to initialize the table structure. It also configures event triggers on the object gateway to achieve automated event monitoring and processing.

[0140] For example, after the user configures the table structure in the management console, the configuration center sends the table structure information to the retrieval acceleration module, and the retrieval acceleration module builds the initial table based on the information. The configuration center also configures event triggers on the object gateway to monitor the operations performed by the object gateway on the storage engine. Once an operation is completed, the event trigger will be triggered and event information will be generated.

[0141] The retrieval acceleration module is used to obtain the initial table through Index Proxy and TiDB according to the table structure configuration information;

[0142] The object storage access gateway is used to obtain a search request input by a user, determine a request type of the search request, and send the search request to the object gateway when the search request is of the first type;

[0143] The retrieval request is an HTTP request; if the first query parameter in the retrieval request is not Analytical, the retrieval request type is determined to be the first type;

[0144] The object storage access gateway in the object storage retrieval optimization system based on function computing provided in this embodiment is the entry module of the system, which is responsible for processing the user's HTTP request and parsing the request type. Its functions include:

[0145] Receive the search request submitted by the user via HTTP and parse the query parameters in the request.

[0146] If the first query parameter in the request is not Analytical, the request is determined to be of the first type, i.e., a non-analytical request. The first type of request will be sent to the Object Gateway for further processing.

[0147] The object gateway is used to execute and complete the operation corresponding to the retrieval request on the storage engine, and after the operation is completed, generate event information corresponding to the retrieval request through an event trigger on the object gateway;

[0148] The object gateway in the object storage retrieval optimization system based on function computing provided in this embodiment is a bridge connecting the storage engine and the function computing module, and is responsible for executing storage operations and generating event information.

[0149] The role of the object gateway is to receive retrieval requests from the object storage access gateway and perform specific operations on the storage engine, such as reading and filtering data. After the storage operation is completed, the configured event trigger is used to generate event information corresponding to the retrieval request. The event information is sent to the function computing module to start the subsequent computing process.

[0150] For example, the object gateway receives a request to write some data to the storage engine and performs the data write operation on the storage engine. After the write operation is completed, the event trigger generates event information and notifies the function computing module.

[0151] A function calculation module, used to run the function operator according to the event information to obtain a calculation result of the function operator;

[0152] The retrieval acceleration module is also used to add the calculation results of the function operator to the initial table to obtain final table data.

[0153] The retrieval acceleration module in the object storage retrieval optimization system based on function computing provided in this embodiment is responsible for using the table structure configuration information and index proxy (Index Proxy) to build the initial table through TiDB and perform accelerated retrieval operations.

[0154] The retrieval acceleration module uses Index Proxy and TiDB to build the initial table and load the necessary indexes based on the table structure information sent by the configuration center. It receives the calculation results from the function calculation module, dynamically adds these results to the initial table, and updates the table data. It supports complex query optimization, such as HTAP (hybrid transaction and analytical processing) operations, to improve query speed.

[0155] For example, after the retrieval acceleration module receives the table structure information from the configuration center, it creates an initial table through TiDB and creates an index for the specified field.

[0156] After the function calculation module completes the analysis and calculation, the function calculation module adds the calculation results (such as new statistical fields) to the table to obtain the final optimized table.

[0157] The function computing module in the object storage retrieval optimization system based on function computing provided in this embodiment is responsible for executing the function operator defined by the user, triggering the calculation by using event information, and returning the calculation result.

[0158] According to the event information provided by the object gateway, the corresponding function operator is called to execute the user-defined calculation logic. It supports flexible operating environment and hardware resource configuration to meet different computing needs. The calculation results are returned to the retrieval acceleration module for updating the final table data.

[0159] Specifically, the object storage access gateway in this embodiment is further configured to send the search request to the search acceleration module when the search request is of the second type;

[0160] If the first query parameter in the search request is Analytical, the search request type is determined to be the second type;

[0161] The retrieval acceleration module is also used to determine whether an initial table exists. If an initial table exists, the operation corresponding to the retrieval request is executed through TiDB to obtain result data corresponding to the retrieval request.

[0162] For example, the object storage access gateway receives a user's HTTP request. Then, the object storage access gateway parses the request parameters and finds that the first query parameter is "Analytical", and determines that this is a second type request. The object storage access gateway sends the request to the retrieval acceleration module for further processing.

[0163] After receiving the search request, the search acceleration module first checks whether the corresponding initial table already exists. Because the user previously configured and issued the table structure in the management console, the search acceleration module confirms that the initial table already exists. Then, the search acceleration module will perform the corresponding query operation through TiDB based on the search request. TiDB executes the query operation and obtains the data results that meet the conditions. The search acceleration module returns the query results to the object storage access gateway. The access gateway then returns the result data to the user, completing the entire process.

[0164] In this embodiment, through the type judgment of the object storage access gateway, analytical requests (second type requests) can be quickly identified and forwarded to the retrieval acceleration module for processing, avoiding confusion between ordinary storage requests and analytical requests and improving system efficiency. Since analytical requests are identified and handed over to the retrieval acceleration module for processing, they will not affect the execution path of ordinary storage requests, thereby improving the overall throughput and concurrent processing capabilities of the system.

[0165] See also Figure 7 The processing process of the object storage retrieval optimization system based on function computing provided in this embodiment during actual use includes:

[0166] The user initiates a retrieval request through the object storage access gateway, which may include read and write operations (retrieval requests including read and write operations are of the first type) or may include analysis requests (analysis requests are of the second type).

[0167] First, determine the type of search request, whether it is the first type or the second type. The determination is based on the first query parameter of the search request: if the first query parameter is not "Analytical", it is considered to be a normal first type request. If the first query parameter is "Analytical", it is considered to be an analytical second type request.

[0168] If the request is of the second type, continue to determine whether there is an initial table;

[0169] If yes, it will enter the accelerated retrieval stage and use the existing index and acceleration module for fast retrieval.

[0170] If so, a native search is performed, that is, the object gateway is directly called to perform regular data operations on the storage engine.

[0171] For the first type of retrieval request, assuming that the retrieval request is a file read request, a corresponding operation is performed to read the corresponding file content from the storage engine, and the read content is returned to the user.

[0172] For the first type of retrieval request, assuming that the retrieval request is a file write request, the corresponding operation is performed to write the data to the storage engine. After the object gateway completes the storage engine operation, event information is generated through the event trigger and sent to the function computing module. The function computing module executes the user-defined function operator based on the event information, and the calculation results are dynamically added to the initial table, further accelerating subsequent retrieval requests.

[0173] This embodiment involves the scenario of object storage retrieval optimization. By designing a comprehensive analysis retrieval system based on function computing, it supports user-defined retrieval fields, thereby significantly improving retrieval efficiency and enhancing retrieval functions. This embodiment provides an object storage retrieval optimization system based on function computing, which uses function operators to provide users with an entry for customizable retrieval conditions and retrieval methods, greatly expanding the online retrieval capabilities of object storage and improving retrieval efficiency. By using the event-driven capabilities of object storage, the reconstruction of retrieval is driven after the normal data flow is completed. In this way, while enriching the retrieval capabilities, it ensures less intrusion into the original read and write processes and maintains the stability of the original process.

[0174] In this embodiment, the object storage access gateway adds the ability to classify and route requests at the API Gateway level. The object storage analysis query class interface is extended, and all object storage requests are classified and routed. This not only reduces the traversal pressure of the object storage gateway, but also provides extremely rich analysis and retrieval capabilities to the outside world. For example, when processing retrieval requests, traditional object storage gateways usually rely on traversal mechanisms. When a user requests to list all files in a directory, the gateway usually traverses the entire directory structure, which will bring huge performance bottlenecks when the number of files is large. High-frequency traversal operations will cause gateway performance to degrade and affect other normal file upload and download operations. In this embodiment, an object storage retrieval optimization system based on function computing receives a first type of retrieval request (file upload, download, delete, etc.) and is still processed by the object storage gateway. Receiving a second type of retrieval request (such as full-text search, aggregate analysis, etc.) is classified and routed to the retrieval acceleration module for processing to avoid affecting the gateway performance. The extended analysis query interface can directly locate files or data blocks through mechanisms such as indexes and pre-built table structures, rather than searching for data by traversing the file list, thereby significantly reducing the performance consumption caused by traversal. In addition, in order to meet more complex retrieval requirements, this embodiment extends new analytical retrieval requests based on the AWS S3 standard API (a set of standardized API interfaces defined by AWS (Amazon Web Services) for its Amazon S3 (Simple Storage Service) object storage service). Through this extension, it is possible to add additional functions such as full-text retrieval, aggregate analysis, and query by metadata filtering. Maintaining compatibility with the standard S3 API ensures that users can still use standard S3 clients to interact with the object storage system without modifying the client configuration.

[0175] It should be noted that, in this article, relational terms such as first and second, etc. are only used to distinguish one entity or operation from another entity or operation, and do not necessarily require or imply any such actual relationship or order between these entities or operations. Moreover, the terms "include", "comprise" or any other variants thereof are intended to cover non-exclusive inclusion, so that a process, method, article or device including a series of elements includes not only those elements, but also other elements not explicitly listed, or also includes elements inherent to such process, method, article or device. In the absence of further restrictions, the elements defined by the sentence "comprise a ..." do not exclude the existence of other identical elements in the process, method, article or device including the elements.

[0176] Each embodiment in this specification is described in a related manner, and the same or similar parts between the embodiments can be referred to each other, and each embodiment focuses on the differences from other embodiments. In particular, for the system embodiment, since it is basically similar to the method embodiment, the description is relatively simple, and the relevant parts can be referred to the partial description of the method embodiment.

[0177] The above description is only a preferred embodiment of the present invention and is not intended to limit the protection scope of the present invention. Any modification, equivalent replacement, improvement, etc. made within the spirit and principle of the present invention are included in the protection scope of the present invention.

Claims

1. A method for optimizing object storage retrieval based on function computing, characterized in that: The method is applied to an object storage retrieval optimization system based on function computing, wherein the system comprises: an object storage access gateway, an object gateway, a function computing module, and a retrieval acceleration module connected in sequence, wherein the object gateway is also connected to a storage engine; the method comprises: The object storage access gateway obtains the search request input by the user and determines the request type of the search request; If the retrieval request is of the first type, the object storage access gateway sends the retrieval request to the object gateway, and the object gateway executes and completes the operation corresponding to the retrieval request in the storage engine, and after the operation is completed, generates event information corresponding to the retrieval request through an event trigger pre-configured on the object gateway; The function calculation module runs a preset function operator according to the event information to obtain a calculation result of the function operator; The retrieval acceleration module adds the calculation result of the function operator to the pre-acquired initial table to obtain the final table data.

2. The object storage retrieval optimization method based on function computing according to claim 1 is characterized in that: The method further comprises: If the retrieval request is of the second type, the object storage access gateway sends the retrieval request to the retrieval acceleration module. The retrieval acceleration module determines whether there is a pre-acquired initial table. If there is a pre-acquired initial table, the operation corresponding to the retrieval request is executed through TiDB to obtain the result data.

3. The object storage retrieval optimization method based on function computing according to claim 2 is characterized in that: The retrieval request is an HTTP request; Accordingly, If the first query parameter in the search request is not Analytical, the search request type is determined to be the first type; If the first query parameter in the search request is Analytical, the search request type is determined to be the second type.

4. The object storage retrieval optimization method based on function computing according to claim 3 is characterized in that: The event information corresponding to the search request includes: operation action, operation object, and operation object attribute; The operation action is a specific action of the operation corresponding to the search request; The operation object is the data object involved in performing the operation action; The operation object attributes include metadata of the operation object.

5. The object storage retrieval optimization method based on function computing according to claim 4 is characterized in that: The function calculation module runs a preset function operator according to the event information to obtain the calculation result of the function operator, which specifically includes: It is determined whether the operation action in the event information is a file writing operation. If so, a preset function operator is run to obtain a calculation result of the function operator.

6. The object storage retrieval optimization method based on function computing according to claim 5 is characterized in that: The function operator is used to determine the search field according to the operation object attribute in the event information and the preset search logic, and use the search field determined by the function operator as the calculation result of the function operator; The search logic is one of the first search logic, the second search logic, and the third search logic; The first search logic is: using the operation object attribute in the event information as a search field; The second search logic is: using the operation object attributes and predefined metadata and / or predefined tags in the event information as search fields; The third search logic is: using the operation object attributes and object attribute data in the event information as search fields; The object attribute data is obtained by processing the operation object in the event information through a designated platform connected to the function calculation module.

7. The object storage retrieval optimization method based on function computing according to claim 6 is characterized in that: The initial table is obtained after Index Proxy in the retrieval acceleration module receives the preset table structure configuration information and initializes the table structure on TiDB; The table structure configuration information includes: the data model and the attributes of the search field and the index configured for the specified search field; The data model is a table, including a table name, field names, and field constraints.

8. An object storage retrieval optimization system based on function computing, characterized in that: The system comprises: The management console is used to receive the function information selected by the user in the management console and the function code entered by the user, and obtain the function operator according to the function information and the function code; it is also used to obtain the table structure configuration information entered by the user in the management console, and send the table structure configuration information to the configuration center; The configuration center is used to configure event triggers in the object gateway; and is also used to send the table structure configuration information to the retrieval acceleration module; The retrieval acceleration module is used to obtain the initial table through Index Proxy and TiDB according to the table structure configuration information; The object storage access gateway is used to obtain a search request input by a user, determine a request type of the search request, and send the search request to the object gateway when the search request is of the first type; The retrieval request is an HTTP request; if the first query parameter in the retrieval request is not Analytical, the retrieval request type is determined to be the first type; The object gateway is used to execute and complete the operation corresponding to the retrieval request on the storage engine, and after the operation is completed, generate event information corresponding to the retrieval request through an event trigger on the object gateway; A function calculation module, used to run the function operator according to the event information to obtain a calculation result of the function operator; The retrieval acceleration module is also used to add the calculation results of the function operator to the initial table to obtain final table data.

9. The object storage retrieval optimization system based on function computing according to claim 8, characterized in that: The object storage access gateway is further configured to send the retrieval request to the retrieval acceleration module when the retrieval request is of the second type; If the first query parameter in the search request is Analytical, the search request type is determined to be the second type; The retrieval acceleration module is also used to determine whether an initial table exists. If an initial table exists, the operation corresponding to the retrieval request is executed through TiDB to obtain result data corresponding to the retrieval request.

10. The object storage retrieval optimization system based on function computing according to claim 9, characterized in that: The function running information includes: virtual hardware resources of the function running environment, function language running environment, and initial concurrent parameters; The table structure configuration information includes: the data model and the attributes of the search field and the index configured for the specified search field; the data model is a table, including the table name, field name, and field constraints.

Images