Distributed file storage micro-service system and control method thereof

Abstract

The invention relates to the technical field of databases, in particular to a distributed file storage micro-service system and a control method thereof. The system comprises a micro-service gateway node, a scheduling center node, a computing node and a storage node, the micro-service gateway node is used for receiving the business service processing request, authenticating the business service processing request and then forwarding the business service processing request to the computing node for processing the corresponding business service according to the reverse proxy rule stored in the dispatching center node; and the computing node is used for analyzing the business service processing request to obtain metadata, determining a storage node for storing a data file corresponding to the business service according to the reverse proxy rule stored in the dispatching center node and the metadata, loading the data file from the storage node, processing the data file and returning the processed data file. According to the method and the system, the system building of a large-scale high-performance heavy data service scene is completed under the condition of no database system.

Description

Technical Field

[0001] The present invention relates to the field of database technology, and in particular to a distributed file storage microservice system and a control method thereof. Background Art

[0002] Database systems are the cornerstone of modern computer information systems. They not only support efficient data storage and management but also provide business operations and decision-making support across a wide range of industries through powerful retrieval, query, security, backup, and analysis capabilities. After decades of development, database systems have evolved into a classic and efficient data storage and query system, as well as a secure and reliable data review system. These advantages have led to the majority of information systems worldwide using one or more database systems for data storage and management. Using a database system has become a best practice for developing and building computer information systems.

[0003] However, as database systems continue to evolve and become the de facto standard for the persistence layer of computer information systems, they have become an essential component for system development. Their importance compels any system development to rely on a database system. When databases are unavailable or inaccessible for unavoidable reasons, system development becomes increasingly difficult. Summary of the Invention

[0004] In view of this, the present invention provides a distributed file storage microservice system and a control method thereof to solve the problem of how to implement system development when the database system is unavailable.

[0005] In a first aspect, the present invention provides a distributed file storage microservice system, which includes: a microservice gateway node, a scheduling center node, a computing node, and a storage node; the microservice gateway node is used to receive a business service processing request, authenticate the business service processing request, and forward the business service processing request to the computing node that processes the corresponding business service according to the reverse proxy rules stored in the scheduling center node; the computing node is used to parse the business service processing request to obtain metadata, determine the storage node that stores the data file corresponding to the business service according to the reverse proxy rules stored in the scheduling center node and the metadata, load the data file from the storage node, and return it after processing the data file.

[0006] The present invention combines functional components such as microservice gateway nodes, scheduling center nodes, computing nodes, and storage nodes to achieve the construction of a system for large-scale, high-performance, and data-intensive business scenarios without a database system. This solves the problem of difficulty in building business systems and storing and retrieving data safely and efficiently when a database system cannot be used due to various factors.

[0007] In an optional embodiment, the system also includes: a storage and computing node; the scheduling center node is used to monitor the business service processing requests received by the microservice gateway node, determine the hot business services received within a preset time that are greater than a threshold, store the hot business services and corresponding data files to the storage and computing node, and update the reverse proxy rules; after the microservice gateway node receives the business service processing request of the hot business service, it forwards the business service processing request to the storage and computing node according to the business service processing request and the reverse proxy rules updated by the scheduling center node; the storage and computing node is used to parse the business service processing request, obtain metadata, load the corresponding data file according to the metadata, and return it after processing the data file.

[0008] In this invention, by setting up integrated storage and computing nodes, hot business services and corresponding data files are transferred to the integrated storage and computing nodes. This allows business logic code and data files to be processed on the same physical node in high-performance scenarios, avoiding data transmission across the network. This reduces network transmission overhead and data handling costs, significantly reduces data loading and synchronization losses, optimizes data access efficiency, and improves the overall performance of high-frequency services.

[0009] In an optional embodiment, the scheduling center node is also used to continuously monitor the business service processing requests of the hot business service. When the business service processing requests received within a preset time are less than a threshold, the corresponding hot business service and the corresponding data file are removed from the storage and computing integrated node, and the reverse proxy rules are updated.

[0010] In the present invention, when the popularity of a hot business service decreases, the processing of the hot business service and the corresponding data files are moved out of the storage and computing integrated node, thereby releasing the resources of the storage and computing integrated node, allowing the storage and computing integrated node to store more other hot business services and improve the utilization rate of the storage and computing integrated node.

[0011] In an optional embodiment, the storage node stores data files corresponding to multiple business services, the storage path of the data files is generated according to the metadata isolation dimension, and the storage format of the data files includes plain text format and multiple engine storage formats; when the computing node loads the data file from the storage node, before processing the data file, it applies for a write lock from the storage node, and the storage node grants the lock to the computing node; when other computing nodes request to process the data file of the storage node, the storage node rejects the request; when the computing node that obtains the write lock completes data processing, it releases the write lock and broadcasts a notification to the computing node that loaded the data file; after receiving the broadcast notification, the other computing nodes reload the data file.

[0012] In the present invention, by adopting a write lock mechanism, data consistency can be guaranteed to avoid file damage or data loss caused by multiple instances writing at the same time. At the same time, the write lock only blocks write operations, and read operations can still be performed, thereby improving system throughput.

[0013] In an optional implementation, when the same business service is deployed on multiple computing nodes, the microservice gateway node uses a load balancer to forward the business service processing request corresponding to the business service.

[0014] In the present invention, by deploying a load balancer in the microservice gateway node, it is possible to determine the request forwarding computing node when the same business service is deployed on multiple computing nodes; at the same time, load balancing can be achieved to avoid excessive load on a single node and improve system availability.

[0015] In an optional embodiment, before the microservice gateway node receives a business service processing request, the computing node and the storage node are further used to send registration information to the scheduling center node, where the registration information includes the address of the business service sent by the computing node and the storage address of the data file sent by the storage node; the scheduling center node is used to generate a reverse proxy rule based on the address and storage address of the business service, and register with the microservice gateway node.

[0016] In the present invention, information is registered with the scheduling center node through the computing node and the storage node, providing a data basis for generating reverse proxy rules in the scheduling center node.

[0017] In an optional implementation, the microservice gateway node is set in an external network firewall, and the scheduling center node, the computing node and the storage node are set in an internal network firewall.

[0018] In the present invention, the microservice gateway node is set in the external network firewall to achieve the reception of requests; the scheduling center node, computing node and storage node are set in the internal network firewall to avoid the risk of being attacked.

[0019] In a second aspect, the present invention provides a control method for a distributed file storage microservice system, applying the distributed file storage microservice system described in the first aspect and any one of the first aspects of the present invention, the method comprising: controlling a microservice gateway node to receive a business service processing request, authenticating the business service processing request, and forwarding the business service processing request to a computing node that processes the corresponding business service according to the reverse proxy rules stored in the scheduling center node; controlling the computing node to parse the business service processing request to obtain metadata, determining a storage node for storing a data file corresponding to the business service according to the reverse proxy rules stored in the scheduling center node and the metadata, loading the data file from the storage node, and returning the data file after processing.

[0020] In a third aspect, the present invention provides a computer device comprising: a memory and a processor, the memory and the processor being communicatively connected to each other, the memory storing computer instructions, and the processor executing the computer instructions to thereby execute the control method of the distributed file storage microservice system of the second aspect mentioned above.

[0021] In a fourth aspect, the present invention provides a computer-readable storage medium having computer instructions stored thereon, the computer instructions being used to enable a computer to execute the control method of the distributed file storage microservice system of the second aspect described above.

[0022] In a fifth aspect, the present invention provides a computer program product, comprising computer instructions, which are used to enable a computer to execute the control method of the distributed file storage microservice system of the second aspect above. BRIEF DESCRIPTION OF THE DRAWINGS

[0023] In order to more clearly illustrate the specific embodiments of the present invention or the technical solutions in the prior art, the following briefly introduces the drawings required for use in the specific embodiments or the description of the prior art. Obviously, the drawings described below are some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying any creative work.

[0024] Figure 1 This is a structural block diagram of a distributed file storage microservice system according to an embodiment of the present invention;

[0025] Figure 2 1 is a flow chart of a control method for a distributed file storage microservice system according to an embodiment of the present invention;

[0026] Figure 3 Schematic diagram of the hardware structure of a computer device according to an embodiment of the present invention. DETAILED DESCRIPTION

[0027] To make the purpose, technical solutions, and advantages of the embodiments of the present invention more clear, the technical solutions in the embodiments of the present invention will be clearly and completely described below in conjunction with the accompanying drawings in the embodiments of the present invention. Obviously, the described embodiments are part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without making creative efforts shall fall within the scope of protection of the present invention.

[0028] According to an embodiment of the present invention, a distributed file storage microservice system is provided, which includes a microservice gateway node, a scheduling center node, a computing node and a storage node; the microservice gateway node is used to receive a business service processing request, authenticate the business service processing request, and forward the business service processing request to the computing node that processes the corresponding business service according to the reverse proxy rules stored in the scheduling center node; the computing node is used to parse the business service processing request to obtain metadata, determine the storage node that stores the data file corresponding to the business service according to the reverse proxy rules stored in the scheduling center node and the metadata, load the data file from the storage node, and return the data file after processing.

[0029] It should be noted that the system can be used in a system architecture without a database, that is, the system provided by this embodiment is used to replace the database system in the relevant technology. Specifically, it is a technology that replaces the role of the traditional database system by combining the distributed file storage provided by this embodiment with the data storage and retrieval engine, and provides a system with data persistent storage and efficient retrieval capabilities in an environment where a database system cannot be used. In this system, the microservice gateway node is set in the external network firewall, and the scheduling center node, the computing node and the storage node are set in the internal network firewall. Among them, if Figure 1 As shown, the microservice gateway node can be deployed in the demilitarized zone (DMZ). The microservice node can receive and distribute inbound traffic, while the dispatch center node, computing nodes, and storage nodes have no direct external network access, reducing the risk of attack. Furthermore, multiple computing nodes and storage nodes can each be included (storage nodes include node4 and node5, and computing nodes include node1, node2, and node3) to enable processing of different business services and storage of different data files.

[0030] Specifically, for the business service processing request received by the microservice gateway node, the business service processing request usually includes the business service to be processed or data information related to the business service. It can be authenticated first, that is, identity authentication is performed. After the authentication is passed, the reverse proxy rules stored in the scheduling center node (the rules include the business services processed by each computing node) are queried to determine which computing node specifically processes the business service to be processed in the request, and then the received business service processing request is forwarded to the corresponding computing node. After receiving the request, the computing node first parses it to obtain the metadata of the business service to be processed, and then initiates a request to the scheduling center based on the metadata, and transmits the metadata to the scheduling center. The scheduling center determines the storage node of the data file of the business service and returns it to the computing node. The computing node then loads the data file from the storage node and reads and processes the data in the data file, such as adding, deleting, checking, and modifying the data.

[0031] The present invention combines functional components such as microservice gateway nodes, scheduling center nodes, computing nodes, and storage nodes to achieve the construction of a system for large-scale, high-performance, and data-intensive business scenarios without a database system. This solves the problem of difficulty in building business systems and storing and retrieving data safely and efficiently when a database system cannot be used due to various factors.

[0032] In an optional embodiment, the system also includes: a storage and computing node; the scheduling center node is used to monitor the business service processing requests received by the microservice gateway node, determine the hot business services received within a preset time that are greater than a threshold, store the hot business services and corresponding data files to the storage and computing node, and update the reverse proxy rules; after the microservice gateway node receives the business service processing request of the hot business service, it forwards the business service processing request to the storage and computing node according to the business service processing request and the reverse proxy rules updated by the scheduling center node; the storage and computing node is used to parse the business service processing request, obtain metadata, load the corresponding data file according to the metadata, process the data file and return it. The scheduling center node is also used to continuously monitor the business service processing requests of the hot business service, and when the business service processing request received within the preset time is less than the threshold, the corresponding hot business service and the corresponding data file are removed from the storage and computing node, and the reverse proxy rules are updated.

[0033] Specifically, during the operation of the system, the dispatch center can monitor the access requests of all business services, and then calculate the access frequency of different business services. When the access frequency of a business service (such as the number of accesses within a preset time) exceeds the preset threshold, the corresponding business service will be marked as a hot business service. The dispatch center then updates its internally stored reverse proxy rules and moves the corresponding hot business services and data files to the storage and computing node. When the microservice gateway node receives an access request for a hot business service again, it forwards the request to the storage and computing node. When the storage and computing node receives a request for a hot business service, it first parses the metadata, then queries and loads the corresponding data files locally, and processes them.

[0034] In addition, during the continuous monitoring of the dispatch center, if it is detected that the access frequency of the hot business service is reduced to below the preset threshold, it is necessary to move the business service and data files from the storage and computing node to the computing node and storage node, release the resources of the storage and computing node, and update the reverse proxy rules. In this way, when the microservice gateway node receives a request for the business service again, it will no longer forward the request to the storage and computing node, but forward it to the corresponding computing node. In addition, it should be noted that for the reverse proxy rules stored in the dispatch center, after each update, they can be registered and updated with the microservice gateway node at the same time, so that the microservice gateway node also contains the latest reverse proxy rules to determine the specific node for forwarding requests (routes).

[0035] In an optional embodiment, the storage node stores data files corresponding to multiple business services, the storage path of the data files is generated according to the metadata isolation dimension, and the storage format of the data files includes plain text format and multiple engine storage formats; wherein, when the computing node loads the data file from the storage node, before processing the data file, it applies for a write lock from the storage node, and the storage node grants the lock to the computing node; when other computing nodes request to process the data file of the storage node, the storage node rejects the request; when the computing node that obtains the write lock completes data processing, it releases the write lock and broadcasts a notification to the computing node that loaded the data file; after receiving the broadcast notification, the other computing nodes reload the data file.

[0036] Specifically, storage nodes store data files for different business services. These files can be distributed across storage nodes based on business isolation factors such as tenant name, business name, service name, and business operation, forming a decentralized business storage matrix that can automatically scale with business scale. Each stored file is encrypted with a key for storage and transmission. As a result, each business service processed by a compute node can quickly locate and retrieve the corresponding data storage file content based on metadata such as the business service's tenant, business name, service name, and request routing.

[0037] Storage nodes can generate self-explanatory storage paths for different data files based on their isolation dimensions. Furthermore, developers can freely choose the storage format of data files stored in storage nodes based on business needs. For example, they can use plain text, JSON, CSV, SQL engine, KV engine storage, and other formats. Compute nodes can also use the engine to add, delete, query, and modify data stored in the data files.

[0038] Furthermore, for data files stored in storage nodes, while a compute node is processing a data file, other compute nodes may also request to process the same data file. Therefore, a write lock mechanism is set up in the storage node. This mechanism allows only one service instance to modify the data file at a time, while other instances can only read it, thus avoiding data conflicts. For example, when service A needs to write to the file entity2.csv, it first requests a write lock from the storage node. If there is currently no lock, the storage node grants the lock, and service A begins writing. Services B and C then request to write to the same file, but the storage node rejects their write requests and allows read requests (to avoid blocking all operations). After service A completes writing, it releases the write lock and broadcasts an update notification to the scheduling center and other services that have loaded the file. After receiving the notification, other services reload the latest file from the storage node, and subsequent write requests can compete for the write lock again.

[0039] In an optional embodiment, when the same business service is deployed on multiple computing nodes, the microservice gateway node uses a load balancer to forward the business service processing request corresponding to the business service. Specifically, when service A can be processed on computing node 1 and computing node 2 respectively, when the microservice gateway node receives the request for the service, the node to which the request is specifically routed can be determined through the load balancer. That is, the load balancer can implement traffic distribution. For example, the load balancer can route the request in a random strategy and randomly route the request to computing node 1 or computing node 2. The load balancer can also adopt a polling strategy, that is, forwarding requests in sequence, such as the first time to computing node 1 and the second time to computing node 2. The load balancer can also adopt a minimum number of connections strategy, that is, preferentially forwarding requests to the computing node with the smallest current load.

[0040] In an optional embodiment, before the microservice gateway node receives a business service processing request, the computing node and the storage node are further used to send registration information to the scheduling center node, where the registration information includes the address of the business service sent by the computing node and the storage address of the data file sent by the storage node; the scheduling center node is used to generate a reverse proxy rule based on the address and storage address of the business service, and register with the microservice gateway node.

[0041] Specifically, different business services can be pre-determined to determine the compute nodes they process. Furthermore, different data files are stored based on data isolation dimensions and corresponding storage paths are established. When compute and storage nodes start up, they register their information with the dispatch center. For compute nodes, this includes the service name, IP address, and port; for storage nodes, the data storage path and file location. Upon receiving the registration information for the compute and storage nodes, the dispatch center generates reverse proxy rules and registers them with the microservice gateway node.

[0042] Based on the above content, the functions of the microservice gateway node, scheduling center node, computing node, storage node, storage-computing integrated node, and other nodes in the system of this embodiment are summarized as follows:

[0043] (1) The microservice gateway provides service governance, request routing, security management, protocol conversion, flow control, load balancing, authentication and authorization, logging and monitoring functions.

[0044] (2) The dispatch center is the core of the efficient operation of the entire system. It is closely integrated with the microservice gateway and is responsible for storing and distributing metadata such as the tenant, business name, service name, and request routing of each business-related storage. Business instances support active query and passive monitoring to perceive the location of business-related storage files. The dispatch center can also dynamically adjust the location of business system code and storage files by analyzing hot businesses and hot data, and dispatch them to the storage and computing nodes to improve the overall operation efficiency of the system.

[0045] (3) A computing node is a computing computer node used to run the specific business implementation code functions of each service. It is stateless and can be started or shut down at any time according to system scheduling, expansion, and contraction. It is responsible for deploying business service code and running it. The implementation logic of each business service quickly locates and obtains the corresponding data storage file content based on metadata such as the business tenant, business name, service name, and request routing, and implements data addition, deletion, query, and modification through the data engine used by the business. After the service is deployed, it actively registers its own path and address with the microservice gateway registration center. After receiving it, the microservice gateway will add the reverse proxy rules corresponding to the path. Multiple identical business service instances will distribute traffic through the load balancer.

[0046] (4) A storage node is a storage computer node used to store persistent business data. It is stateful in nature, and business data is encrypted and stored in separate, isolated file directories in plain text and binary formats. Specifically, the storage node is responsible for storing the data storage files corresponding to each business. The file storage path will be generated based on the business metadata isolation dimension. The generated path will be self-explanatory. Based on the metadata, the path can also be used to quickly locate and obtain the file content. The file storage format is freely selected by the developer based on the business's own needs, and supports a variety of formats such as plain text, JSON, CSV, SQL engine, and KV engine storage.

[0047] (5) The storage-computing node is a stateful computer node that balances computing performance and storage capacity. It is used to schedule and run special business services with high performance, high concurrency, and high-frequency data access. Specifically, the storage-computing node is a node that supports both the deployment of business service code and the storage of business data. The storage-computing design is the core of the system's high-performance operation. By dynamically analyzing hot businesses and hot data, they are dynamically scheduled to run on the same storage-computing node. Code execution and data access are both performed locally, achieving efficient system operation.

[0048] According to an embodiment of the present invention, an embodiment of a distributed file storage microservice system is provided. It should be noted that the steps shown in the flowcharts of the accompanying drawings can be executed in a computer system such as a set of computer-executable instructions, and although a logical order is shown in the flowcharts, in some cases, the steps shown or described can be executed in an order different from that shown here.

[0049] In this embodiment, a control method for a distributed file storage microservice system is provided, which can be used in electronic devices such as computers, mobile phones, tablet computers, etc. Figure 2 is a flow chart of a control method for a distributed file storage microservice system according to an embodiment of the present invention. Figure 2 As shown, the process includes the following steps:

[0050] Step S101: Control the microservice gateway node to receive a business service processing request, authenticate the business service processing request, and forward the business service processing request to a computing node that processes the corresponding business service according to the reverse proxy rules stored in the scheduling center node.

[0051] Step S102, the control computing node is used to parse the business service processing request, obtain metadata, determine the storage node that stores the data file corresponding to the business service based on the reverse proxy rules stored in the scheduling center node and the metadata, load the data file from the storage node, process the data file, and return it.

[0052] As a specific application example of the embodiment of the present invention, the control method is specifically implemented using the following process:

[0053] Step 1, prepare the host nodes: microservice gateway node gateway; scheduling center node master; computing nodes node1, node2, node3; storage nodes node4, node5; storage and computing node node6. The microservice gateway node gateway is behind the external network firewall device, located in the DMZ, receiving and distributing the ingress traffic. The scheduling center node, computing node, storage node and storage and computing node are behind the intranet gateway device, without direct external network access, and are interconnected within the intranet. The architecture topology is as follows: Figure 1 .

[0054] Step 2: Deploy the microservice gateway, configure dynamic reverse proxy rules, and complete traffic distribution. The address of this embodiment is https: / / api.gateway.com / .

[0055] Step 3: Deploy the dispatch center and load the service governance module for automatic service registration and discovery. The dispatch center stores the entire system's configuration and runtime metadata, including the package locations and startup commands for all services, the operating status of all services, and the storage file locations of all domains. It also communicates with the microservice gateway to manage services and complete service registration and discovery. Domain: In domain-driven design, it refers to the collection of knowledge, rules, logic, and behaviors for a specific business or application. For example, an order management domain handles operations such as order creation, modification, cancellation, and payment.

[0056] Step 4: Deploy three business services: svr1, svr2, and svr3. The svr1 service handles one domain, contains 10 interfaces, runs on node1, and stores data on node4. The svr2 service handles two domains, contains 20 interfaces, runs on node2 and node3, and stores data on node5. svr2 runs two instances, and traffic is randomly loaded to node 1 or 2. The following reverse proxy rules are registered with the microservice gateway through the scheduling center:

[0057] / path1=>node1:3000, port 3000 of node1 runs svr1;

[0058] / path2=>node2:8080, port 8080 of node2 runs svr2;

[0059] / path2=>node3:8080, where port 8080 of node3 runs svr2.

[0060] Step 5. After the request GET https: / / api.gateway.com / path1 / entity1 reaches the microservice gateway, the gateway first resolves the tenant ID as tanent1 based on the authentication information, and then routes it to the svr1 service on the node1 node. The DataFactory.loadFile() method, the basic tool of the svr1 service business code, will load and cache data files to the dispatch center based on the current service's tenant ID, service name, entity ID, read-write type and other meta-information. Finally, the data storage file is loaded from node4 to complete the data reading of entity entity1. In this example, entity1 will be stored in the form of a JSON file. After srv1 parses entity1.json, it reads the data list corresponding to the conditions and returns it to the client. The loadFile pseudo code is as follows:

[0061]

[0062]

[0063] Step 6: After the POST request https: / / api.gateway.com / path2 / {entity2|entity3} reaches the microservice gateway, the gateway first resolves the tenant ID as tanent2 based on the authentication information and then routes it to the svr2 service on node2 or node3. In this embodiment, entity2 is stored in a CSV file, which is a series of structured system configuration information; entity3 is stored in the RocksDB engine, a high-performance KV format storage engine that supports high-performance KV storage of extremely large data volumes and complex data types. Business code will use the correct engine to operate the file based on the storage type to complete business data access operations. The first service to access the data file in write mode will acquire a write lock. Subsequent services accessing the file will only be allowed to access the file in read-only mode. After the service holding the write lock completes the write operation, it will broadcast the write operation command and data to other services that have loaded the file and then release the write lock. Other services will update the loaded file, and services that need the write lock will compete for the write lock.

[0064] In step 7, after a large number of POSTs to https: / / api.gateway.com / path2 / entity2 arrive at the microservice gateway, the hotspot service data will be updated to the scheduling center. After reaching the threshold, the scheduling center will schedule the svr2 service and the current data file to the storage and computing integrated node node6, and forward all traffic of the microservice gateway svr2 service to node6 to improve the concurrency performance of the svr2 service and eliminate the performance loss caused by storage file scheduling and synchronization.

[0065] In step 8, after the dispatch center detects that the srv2 service traffic peak has subsided and the time has reached the threshold, it will re-switched the traffic to the ordinary j computing node node2 and stop the srv2 service on the storage and computing node node6, releasing resources to wait for other hot services to be scheduled.

[0066] In the present invention, a database-free system architecture design is adopted, and the database components in the system architecture are removed and replaced by distributed ordinary file storage. Multiple files are scattered in various computer nodes of the system along with the business matrix. The system can quickly locate and obtain the corresponding data storage file content according to the metadata such as the tenant to which the business belongs, the service name, the request route, and the like, and realize the addition, deletion, query and modification of data through the data engine used by the business. It completely replaces the traditional database system. At the same time, a data storage method for business-related data files is adopted, and the storage location of the data files is located through the business metadata. Combined with the type of business operation on the data, the data files can be quickly loaded, accessed and synchronized. The data is no longer concentrated in a centralized database system, but is distributed in ordinary storage nodes of arbitrarily expanded scale. In addition, a hot business detection and optimization scheduling algorithm is also adopted. According to the analysis of the execution frequency of the business code and the frequency of data file scheduling, the hot business and data are scheduled to the same storage and computing node, which greatly reduces the loss of data loading and synchronization, optimizes data access efficiency, and improves the overall performance of high-frequency services.

[0067] The embodiment of the present invention also provides a computer device having the above Figure 1 The distributed file storage microservice system shown.

[0068] See also Figure 3 , Figure 3 is a structural diagram of a computer device provided by an optional embodiment of the present invention, such as Figure 3 As shown, the computer device includes: one or more processors 10, memory 20, and interfaces for connecting various components, including high-speed interfaces and low-speed interfaces. Various components utilize different buses to communicate with each other and can be installed on a common mainboard or installed in other ways as needed. The processor can process the instructions executed in the computer device, including instructions stored in the memory or on the memory to display the graphical information of the GUI on an external input / output device (such as, a display device coupled to the interface). In some optional embodiments, if necessary, multiple processors and / or multiple buses can be used together with multiple memories and multiple memories. Equally, multiple computer devices can be connected, and each device provides part of the necessary operations (for example, as a server array, a group of blade servers, or a multi-processor system). Figure 3 A processor 10 is taken as an example.

[0069] The processor 10 may be a central processing unit, a network processor, or a combination thereof. The processor 10 may further include a hardware chip. The hardware chip may be an application-specific integrated circuit, a programmable logic device, or a combination thereof. The programmable logic device may be a complex programmable logic device, a field programmable gate array, a general purpose array logic, or any combination thereof.

[0070] The memory 20 stores instructions that can be executed by at least one processor 10, so as to enable at least one processor 10 to implement the system shown in the above embodiment.

[0071] The memory 20 may include a program storage area and a data storage area, wherein the program storage area may store an operating system, an application required for at least one function; the data storage area may store data created based on the use of a computer device for displaying a small program landing page, etc. In addition, the memory 20 may include a high-speed random access memory, and may also include a non-transient memory, such as at least one disk storage device, a flash memory device, or other non-transient solid-state storage device. In some optional embodiments, the memory 20 may optionally include a memory remotely located relative to the processor 10, and these remote memories may be connected to the computer device via a network. Examples of the above-mentioned network include, but are not limited to, the Internet, an intranet, a local area network, a mobile communication network, and a combination thereof.

[0072] The memory 20 may include a volatile memory, such as a random access memory; the memory may also include a non-volatile memory, such as a flash memory, a hard disk or a solid-state drive; the memory 20 may also include a combination of the above types of memory.

[0073] The computer device further includes a communication interface 30 for the computer device to communicate with other devices or a communication network.

[0074] The embodiment of the present invention also provides a computer-readable storage medium. The system according to the embodiment of the present invention can be implemented in hardware, firmware, or as a computer code that can be recorded in a storage medium, or downloaded through a network and originally stored in a remote storage medium or a non-temporary machine-readable storage medium and will be stored in a local storage medium, so that the system described herein can be stored in such software processing on a storage medium using a general-purpose computer, a dedicated processor, or programmable or dedicated hardware. Among them, the storage medium can be a magnetic disk, an optical disk, a read-only storage memory, a random access memory, a flash memory, a hard disk or a solid-state drive, etc.; further, the storage medium can also include a combination of the above types of memory. It can be understood that a computer, a processor, a microprocessor controller or programmable hardware includes a storage component that can store or receive software or computer code. When the software or computer code is accessed and executed by a computer, a processor or hardware, the system shown in the above embodiment is implemented.

[0075] A portion of the present invention may be applied as a computer program product, such as a computer program instruction, which, when executed by a computer, can call or provide the system and / or technical solution according to the present invention through the operation of the computer. Those skilled in the art should understand that the form in which the computer program instruction exists in a computer-readable medium includes, but is not limited to, a source file, an executable file, an installation package file, etc. Accordingly, the way in which the computer program instruction is executed by the computer includes, but is not limited to: the computer directly executes the instruction, or the computer compiles the instruction and then executes the corresponding compiled program, or the computer reads and executes the instruction, or the computer reads and installs the instruction and then executes the corresponding installed program. Here, the computer-readable medium may be any available computer-readable storage medium or communication medium that can be accessed by the computer.

[0076] Although the embodiments of the present invention have been described with reference to the accompanying drawings, those skilled in the art may make various modifications and variations without departing from the spirit and scope of the present invention. Such modifications and variations are all within the scope defined by the appended claims.

Claims

1. A distributed file storage microservice system, characterized in that: The system includes: a microservice gateway node, a scheduling center node, a computing node, and a storage node; The microservice gateway node is used to receive business service processing requests, authenticate the business service processing requests, and forward the business service processing requests to the computing node that processes the corresponding business service according to the reverse proxy rules stored in the scheduling center node; The computing node is used to parse the business service processing request to obtain metadata, determine the storage node that stores the data file corresponding to the business service based on the reverse proxy rules stored in the scheduling center node and the metadata, load the data file from the storage node, and return it after processing the data file.

2. The system according to claim 1, wherein: The system further includes: a storage and computing integrated node; The dispatch center node is used to monitor the business service processing requests received by the microservice gateway node, determine the hot business services received within a preset time that are greater than a threshold, store the hot business services and corresponding data files in the storage and computing integrated node, and update the reverse proxy rules; After receiving the business service processing request of the hotspot business service, the microservice gateway node forwards the business service processing request to the storage and computing integrated node according to the business service processing request and the reverse proxy rules updated by the scheduling center node; The storage and computing integrated node is used to parse the business service processing request, obtain metadata, load the corresponding data file according to the metadata, process the data file and return it.

3. The system according to claim 2, characterized in that: The scheduling center node is also used to continuously monitor the business service processing requests of hot business services. When the business service processing requests received within a preset time are less than a threshold, the corresponding hot business service and the corresponding data file are removed from the storage and computing integrated node, and the reverse proxy rules are updated.

4. The system according to claim 1, wherein: The storage node stores data files corresponding to multiple business services. The storage path of the data files is generated according to the metadata isolation dimension. The storage format of the data files includes plain text format and multiple engine storage formats. When a computing node loads a data file from the storage node and applies for a write lock from the storage node before processing the data file, the storage node grants the lock to the computing node; When other computing nodes request to process the data file of the storage node, the storage node rejects the request; When the computing node that acquired the write lock completes data processing, it releases the write lock and broadcasts a notification to the computing nodes that loaded the data file; After receiving the broadcast notification, other computing nodes reload the data file.

5. The system according to claim 1, wherein: When the same business service is deployed on multiple computing nodes, the microservice gateway node uses a load balancer to forward the business service processing request corresponding to the business service.

6. The system according to claim 1, wherein: Before the microservice gateway node receives the business service processing request, the computing node and the storage node are further configured to send registration information to the scheduling center node, where the registration information includes the address of the business service sent by the computing node and the storage address of the data file sent by the storage node; The scheduling center node is used to generate a reverse proxy rule according to the address and storage address of the business service, and register with the microservice gateway node.

7. The system according to claim 1, wherein: The microservice gateway node is set in the external network firewall, and the scheduling center node, the computing node and the storage node are set in the internal network firewall.

8. A control method for a distributed file storage microservice system, characterized in that: Applied to the distributed file storage microservice system according to any one of claims 1 to 7, the method comprising: Control the microservice gateway node to receive the business service processing request, authenticate the business service processing request, and forward the business service processing request to the computing node that processes the corresponding business service according to the reverse proxy rules stored in the scheduling center node; The control computing node is used to parse the business service processing request to obtain metadata, determine the storage node that stores the data file corresponding to the business service based on the reverse proxy rules stored in the scheduling center node and the metadata, load the data file from the storage node, and return it after processing the data file.

9. A computer device, characterized in that: include: A memory and a processor, wherein the memory and the processor are communicatively connected to each other, the memory stores computer instructions, and the processor executes the control method of the distributed file storage microservice system according to claim 8 by executing the computer instructions.

10. A computer-readable storage medium, characterized in that The computer-readable storage medium stores computer instructions, which are used to enable a computer to execute the control method of the distributed file storage microservice system according to claim 8.

Images