User information synchronization method and system based on cloud storage system

By using distributed message queue Nats as communication middleware in cloud storage system, the coupling between storage services and user management services is disabled, and the user information is effectively synchronized, and the stability and performance problems caused by direct communication in cloud storage system are solved. , improve the overall service quality and security of the system.

CN115665191BActive Publication Date: 2024-11-01SHANDONG LANGCHAO YUNTOU INFORMATION TECH CO LTD
View PDF 6 Cites 0 Cited by

Patent Information

Application Number
CN202211224655.8
Authority / Receiving Office
CN · China
Patent Type
Patents(China)
Current Assignee / Owner
Filing Date
2022-10-09
Publication Date
2024-11-01
Estimated Expiration
2042-10-09

AI Technical Summary

Technical Problem

Direct communication between storage services and user management services in cloud storage systems leads to high coupling, affecting system stability and performance, and is seriously wasted resources.

Method used

Use distributed message queue Nats as a communication middleware between user management services and data storage services. User information synchronization is achieved through topic publishing and subscription mechanisms, and timed tasks are used in data storage services to detect message queues to ensure the real-time and Security.

Benefits of technology

Decoupling of service, improve system stability and throughput, reduce resource waste, enhance communication security, reduce invalid request processing, and improve overall service quality.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure CN115665191B_ABST
    Figure CN115665191B_ABST
Patent Text Reader

Abstract

The present invention discloses a user information synchronization method and system based on a cloud storage system, and belongs to the field of cloud storage technology. The technical problem to be solved by the present invention is how to improve the stability, security and service quality of the cloud storage system, and reduce the service pressure of the cloud storage system and reduce resource waste. The technical scheme adopted is as follows: the method is specifically as follows: a distributed message queue Nats is used as a communication middleware between a user management service and a data storage service; the user management service publishes topics related to user information in the distributed message queue Nats, and the data storage service subscribes to related topics; when user information changes in the cloud storage system, the user management service publishes a message to a designated topic of the distributed message queue Nats; after the data storage service detects the message sent by the user management service to the distributed message queue Nats, the detected message sent by the user management service to the distributed message queue Nats is consumed.
Need to check novelty before this filing date? Find Prior Art

Description

Technical Field

[0001] The present invention relates to the field of cloud storage technology, and in particular to a user information synchronization method and system based on a cloud storage system. Background Art

[0002] With the development of Internet technology, the way of software development is also changing. Although the traditional centralized development method has many advantages such as simple development, avoiding duplicate development, simple management, and no scheduling consumption, its disadvantages are also obvious, such as low efficiency, inflexibility, difficult maintenance, poor stability, and insufficient scalability.

[0003] Unlike centralized development architecture, microservice architecture decouples solutions by decomposing functions into discrete services and provides more flexible service support. Applications are created around business domain components, which can be developed, managed, and iterated independently. Using cloud architecture and platform-based deployment, management, and service functions in decentralized components makes product delivery easier. Due to the advantages of microservice architecture such as flexible deployment, high fault tolerance, and easy scalability, more and more software development, including cloud services, choose to adopt microservice architecture.

[0004] The cloud storage system is composed of many independent services. Each service is managed and deployed in a distributed manner. In order to complete a business, it is often necessary to cooperate with multiple services. For example, in the object storage service, user management needs to be completed by the storage service and the user management service. The user information in the two services needs to be synchronized for the business to proceed normally. Therefore, how to effectively synchronize the user information between services greatly affects the service quality of the entire storage service.

[0005] At present, in cloud storage systems, storage services and user management services often use direct communication to synchronize user data. Although this method is easy to develop and maintain, it makes the two services highly coupled. If one service has a problem, it will directly affect the normal operation of the other service. In addition, the storage service needs to determine whether each request is synchronized and parse the communication content, which will greatly reduce the performance of the storage service.

[0006] Therefore, how to improve the stability, security and service quality of cloud storage systems, while reducing the service pressure of cloud storage systems and reducing resource waste is a technical problem that needs to be solved urgently. Summary of the invention

[0007] The technical task of the present invention is to provide a user information synchronization method and system based on a cloud storage system to solve the problem of how to improve the stability, security and service quality of the cloud storage system, while reducing the service pressure of the cloud storage system and reducing resource waste.

[0008] The technical task of the present invention is achieved in the following manner: a user information synchronization method based on a cloud storage system, the method is specifically as follows:

[0009] Use distributed message queue Nats as the communication middleware between user management service and data storage service;

[0010] The user management service publishes topics related to user information in the distributed message queue Nats, and the data storage service subscribes to related topics. The data storage service uses a scheduled task to detect in real time whether there are subscription messages in the distributed message queue Nats.

[0011] When user information changes in the cloud storage system (e.g., user deletion, key addition, etc.), the user management service publishes a message to the specified topic of the distributed message queue Nats;

[0012] After the data storage service detects the message sent by the user management service to the distributed message queue Nats, it consumes the message sent by the detected user management service to the distributed message queue Nats;

[0013] The data storage service updates the user information based on the message content sent by the user management service to the distributed message queue Nats and persists it to the data persistence layer, thereby realizing the synchronization of user information in the user management service.

[0014] Preferably, the data storage service, as a consumer of user information, starts a separate thread to detect relevant topic information in the distributed message queue Nats in real time to ensure the real-time nature of the messages.

[0015] As a preferred embodiment, the distributed message queue Nats is used as the message bus for communication between the user management service and the data storage service;

[0016] When the communication between the user management service and the data storage service is interrupted, the distributed message queue Nats will save the unconsumed messages in the queue and wait for the communication to be reconnected. The data storage service can then continue to consume the unconsumed historical messages, thereby ensuring the synchronization of user information.

[0017] More preferably, a distributed lock is added at the location where the data storage service performs the specific operation of synchronizing user information to ensure that only one storage service process is synchronizing user information at the same time, thereby ensuring the security of the synchronized information.

[0018] A user information synchronization system based on a cloud storage system, the system comprising an access layer, an application layer and a storage layer;

[0019] The access layer is used to receive requests from clients and forward them to the application layer, communicating with other services through the communication message bus;

[0020] The application layer is used to process business logic at the application level and provide business services. It also receives and processes requests from the access layer. The application layer is also used to interact with the storage layer and write or read data from the storage layer.

[0021] The storage layer is used for data persistence and is completely transparent to the business side. That is, it provides real data storage services and stores data that needs to be persisted in storage devices.

[0022] Preferably, the access layer communicates with other services via a distributed message queue Nats;

[0023] The access layer adopts civetweb server to improve the high-concurrency request processing capability.

[0024] As a preferred option, the data storage service, as a consumer of user information, starts a separate thread to detect relevant topic information in the distributed message queue Nats in real time to ensure the real-time nature of the message;

[0025] Add a distributed lock at the location where the data storage service performs the specific operation of synchronizing user information to ensure that only one storage service process is synchronizing user information at the same time, thereby ensuring the security of the synchronized information.

[0026] Preferably, the working process of the system is as follows:

[0027] (1) The user changes the user information on the front end;

[0028] (2) The user management service receives information about changes to user data;

[0029] (3) The user management service sends messages about user data changes in the form of topics to the distributed message queue Nats;

[0030] (4) The data storage service subscribes to the topic of user-related information from the distributed message queue Nats, and communicates with the distributed message queue Nats in real time to detect user-related information;

[0031] (5) When the data storage service detects changes in user data in the distributed message queue Nats, the data storage service will persist the user information received from the distributed message queue Nats to the underlying storage device to complete the synchronization.

[0032] An electronic device comprising: a memory and at least one processor;

[0033] Wherein, the memory stores a computer program;

[0034] The at least one processor executes the computer program stored in the memory, so that the at least one processor executes the user information synchronization method based on the cloud storage system as described above.

[0035] A computer-readable storage medium stores a computer program, and the computer program can be executed by a processor to implement the user information synchronization method based on the cloud storage system as described above.

[0036] The user information synchronization method and system based on the cloud storage system of the present invention have the following advantages:

[0037] (1) The present invention solves the problem caused by the storage service and user management service in the cloud storage system using direct communication to synchronize user data. Using Nats as the message bus for communication between the storage service and the user management service can remove the strong coupling relationship between the two and improve the stability of the system.

[0038] (ii) The present invention uses Nats as the message bus to fully utilize the high performance characteristics of Nats, improve the system throughput, reduce latency, and improve the overall service quality of the cloud storage system;

[0039] (III) The present invention uses message middleware for communication between services, which can reduce system service pressure and reduce resource waste;

[0040] (iv) The present invention uses the unique communication mechanism of NATS to ensure the confidentiality of communication, protect user privacy, and improve the security of system services;

[0041] (V) The entire system of the present invention is driven by message events. The user synchronization message driving system in Nats processes the synchronization data and no longer processes a large number of invalid requests, which greatly reduces the burden on the system and realizes the synchronization of user information. BRIEF DESCRIPTION OF THE DRAWINGS

[0042] The present invention is further described below in conjunction with the accompanying drawings.

[0043] Attached Figure 1 A flowchart of a method for synchronizing user information based on a cloud storage system;

[0044] Attached Figure 2 A diagram showing the communication between the data storage service and the user management service via NATS.

[0045] Attached Figure 3 It is a structural block diagram of the user information synchronization system based on the cloud storage system;

[0046] Attached Figure 4 The flowchart of the working process of the user information synchronization system based on the cloud storage system. DETAILED DESCRIPTION

[0047] The user information synchronization method and system based on the cloud storage system of the present invention are described in detail below with reference to the accompanying drawings and specific embodiments of the specification.

[0048] Embodiment 1:

[0049] As attached Figure 1 As shown, this embodiment provides a user information synchronization method based on a cloud storage system, and the method is specifically as follows:

[0050] S1. Use the distributed message queue Nats as the communication middleware between the user management service and the data storage service, as shown in the attached Figure 2 As shown;

[0051] S2. The user management service publishes topics related to user information in the distributed message queue Nats, and the data storage service subscribes to related topics. The data storage service uses a scheduled task to detect in real time whether there is a subscription message in the distributed message queue Nats.

[0052] S3. When user information changes in the cloud storage system (e.g., user deletion, key addition, etc.), the user management service publishes a message to the specified topic of the distributed message queue Nats;

[0053] S4. After the data storage service detects the message sent by the user management service to the distributed message queue Nats, it consumes the detected message sent by the user management service to the distributed message queue Nats;

[0054] S5. The data storage service updates the user information according to the message content sent by the user management service to the distributed message queue Nats and persists it to the data persistence layer, thereby realizing the synchronization of user information in the user management service.

[0055] In this embodiment, Nats is selected as the message bus for communication between the data storage service and the user management service, which improves the overall throughput and ensures the transmission efficiency of the message.

[0056] Distributed message queue Nats is an open source high-performance message middleware that can achieve a throughput of millions per second while ensuring end-to-end message transmission latency within milliseconds. Therefore, the extremely high performance of distributed message queue Nats is one of the reasons for choosing it as the message bus for communication between storage services and user management services. On the other hand, all publishers and subscribers communicate through distributed message queue Nats, and services are not directly connected to each other, which provides good support for the scalability of the system.

[0057] This embodiment uses the distributed message queue Nats as the message middleware for communication, rather than direct communication between services. This communication method can avoid triggering the user information synchronization mechanism in the storage service every time a request arrives at the storage service, which can reduce the ineffective consumption of resources and improve the service performance of the storage service.

[0058] This embodiment uses the distributed message queue Nats as the message middleware for communication, and can use the unique communication mechanism of Nats to ensure the security of message transmission. User information is undoubtedly a very private information. The ability of cloud storage service providers to protect user privacy greatly affects the user's choice of service products, and the secure communication mechanism of Nats can effectively ensure the confidentiality of communication.

[0059] The data storage service in this embodiment acts as a consumer of user information and starts a separate thread to detect relevant topic information in the distributed message queue Nats in real time to ensure the real-time nature of the message.

[0060] In this embodiment, the user management service and the data storage service use the distributed message queue Nats as the message bus for communication;

[0061] Using the distributed message queue Nats as the communication message bus can ensure that messages are not missed even if the communication is interrupted. When the communication between the user management service and the data storage service is interrupted, the distributed message queue Nats will save the unconsumed messages in the queue and wait for the communication to be reconnected. The data storage service can then continue to consume the unconsumed historical messages, thereby ensuring the synchronization of user information.

[0062] In order to ensure efficient service performance, cloud storage systems generally use multiple storage service processes to provide services. If multiple processes synchronize user information at the same time, it will cause great insecurity. Therefore, distributed locks can be added to the location where the data storage service performs specific operations to synchronize user information to ensure that only one storage service process is synchronizing user information at the same time, thereby ensuring the security of the synchronized information.

[0063] Embodiment 2:

[0064] As attached Figure 3 As shown, this embodiment provides a user information synchronization system based on a cloud storage system, which includes an access layer, an application layer and a storage layer;

[0065] The access layer is used to receive requests from clients and forward them to the application layer, communicating with other services through the communication message bus;

[0066] The application layer is used to process business logic at the application level and provide business services. It also receives and processes requests from the access layer. The application layer is also used to interact with the storage layer and write or read data from the storage layer.

[0067] The storage layer is used for data persistence and is completely transparent to the business side. That is, it provides real data storage services and stores data that needs to be persisted in storage devices.

[0068] The access layer in this embodiment communicates with other services through the distributed message queue Nats;

[0069] The access layer in this embodiment can be some high-performance servers, such as civetweb, etc., to improve the high-concurrency request processing capability.

[0070] The data storage service in this embodiment acts as a consumer of user information and starts a separate thread to detect relevant topic information in the distributed message queue Nats in real time to ensure the real-time nature of the message;

[0071] Add a distributed lock at the location where the data storage service performs the specific operation of synchronizing user information to ensure that only one storage service process is synchronizing user information at the same time, thereby ensuring the security of the synchronized information.

[0072] As attached Figure 4 As shown in the figure, the working process of the system is as follows:

[0073] (1) The user changes the user information on the front end;

[0074] (2) The user management service receives information about changes to user data;

[0075] (3) The user management service sends messages about user data changes in the form of topics to the distributed message queue Nats;

[0076] (4) The data storage service subscribes to the topic of user-related information from the distributed message queue Nats, and communicates with the distributed message queue Nats in real time to detect user-related information;

[0077] (5) When the data storage service detects changes in user data in the distributed message queue Nats, the data storage service will persist the user information received from the distributed message queue Nats to the underlying storage device to complete the synchronization.

[0078] Embodiment 3:

[0079] This embodiment also provides an electronic device, including: a memory and a processor;

[0080] Wherein, the memory stores computer-executable instructions;

[0081] The processor executes the computer-executable instructions stored in the memory, so that the processor executes the user information synchronization method based on the cloud storage system in any embodiment of the present invention.

[0082] The processor may be a central processing unit (CPU), or other general-purpose processors, digital signal processors (DSP), application-specific integrated circuits (ASIC), field-programmable gate arrays (FPGA) or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc. The processor may be a microprocessor or any conventional processor, etc.

[0083] The memory can be used to store computer programs and / or modules. The processor realizes various functions of the electronic device by running or executing the computer programs and / or modules stored in the memory, and calling the data stored in the memory. The memory can mainly include a program storage area and a data storage area, wherein the program storage area can store an operating system, at least one application required for a function, etc.; the data storage area can store data created according to the use of the terminal, etc. In addition, the memory can also include a high-speed random access memory, and can also include a non-volatile memory, such as a hard disk, a memory, a plug-in hard disk, a smart memory card (SMC), a secure digital (SD) card, a flash memory card, at least one disk storage period, a flash memory device, or other volatile solid-state storage devices.

[0084] Embodiment 4:

[0085] This embodiment also provides a computer-readable storage medium, in which a plurality of instructions are stored, and the instructions are loaded by a processor, so that the processor executes the user information synchronization method based on a cloud storage system in any embodiment of the present invention. Specifically, a system or device equipped with a storage medium can be provided, on which a software program code that implements the functions of any of the above embodiments is stored, and a computer (or CPU or MPU) of the system or device reads and executes the program code stored in the storage medium.

[0086] In this case, the program code itself read from the storage medium can realize the function of any one of the above-mentioned embodiments, and thus the program code and the storage medium storing the program code constitute a part of the present invention.

[0087] The storage medium embodiments for providing the program code include a floppy disk, a hard disk, a magneto-optical disk, an optical disk (such as CD-ROM, CD-R, CD-RW, DVD-ROM, DVD-RYM, DVD-RW, DVD+RW), a magnetic tape, a non-volatile memory card, and a ROM. Alternatively, the program code can be downloaded from a server computer via a communication network.

[0088] In addition, it should be clear that the functions of any of the above embodiments can be implemented not only by executing the program code read by the computer, but also by enabling an operating system operating on the computer to complete part or all of the actual operations based on instructions from the program code.

[0089] In addition, it can be understood that the program code read from the storage medium is written to a memory provided in an expansion board inserted into the computer or written to a memory provided in an expansion unit connected to the computer, and then based on the instructions of the program code, a CPU installed on the expansion board or the expansion unit is enabled to perform part or all of the actual operations, thereby realizing the functions of any of the above-mentioned embodiments.

[0090] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention, rather than to limit it. Although the present invention has been described in detail with reference to the aforementioned embodiments, those skilled in the art should understand that they can still modify the technical solutions described in the aforementioned embodiments, or replace some or all of the technical features therein with equivalents. However, these modifications or replacements do not cause the essence of the corresponding technical solutions to deviate from the scope of the technical solutions of the embodiments of the present invention.

Claims

1. A user information synchronization method based on a cloud storage system, characterized in that: The method is as follows: Use distributed message queue Nats as the communication middleware between user management service and data storage service; The user management service publishes topics related to user information in the distributed message queue Nats, and the data storage service subscribes to related topics. The data storage service uses a scheduled task to detect in real time whether there are subscription messages in the distributed message queue Nats. When user information changes in the cloud storage system, the user management service publishes a message to the designated topic of the distributed message queue Nats; After the data storage service detects the message sent by the user management service to the distributed message queue Nats, it consumes the message sent by the detected user management service to the distributed message queue Nats; The data storage service updates the user information based on the message content sent by the user management service to the distributed message queue Nats and persists it to the data persistence layer, thus realizing the synchronization of user information in the user management service. Among them, the data storage service, as a consumer of user information, starts a separate thread to detect relevant topic information in the distributed message queue Nats in real time; The distributed message queue Nats is used as the message bus for communication between the user management service and the data storage service; When the communication between the user management service and the data storage service is interrupted, the distributed message queue Nats saves the unconsumed messages in the queue and waits for the communication to be reconnected. The data storage service can then continue to consume the unconsumed historical messages. Add a distributed lock at the location where the data storage service performs the specific operation of synchronizing user information to ensure that only one storage service process is synchronizing user information at the same time.

2. A user information synchronization system based on a cloud storage system, characterized in that: The system includes an access layer, an application layer, and a storage layer; The access layer is used to receive requests from clients and forward them to the application layer, communicating with other services through the communication message bus; The application layer is used to process business logic at the application level and provide business services. It also receives and processes requests from the access layer. The application layer is also used to interact with the storage layer and write or read data from the storage layer. The storage layer is used for data persistence and is completely transparent to the business side. It provides real data storage services and stores data that needs to be persisted in storage devices. Among them, the access layer communicates with other services through the distributed message queue Nats; The access layer uses civetweb server to improve the high-concurrency request processing capability; As a consumer of user information, the data storage service starts a separate thread to detect relevant topic information in the distributed message queue Nats in real time; Add a distributed lock to the location where the data storage service performs the specific operation of synchronizing user information to ensure that only one storage service process is synchronizing user information at the same time; The working process of the system is as follows: (1) The user changes the user information on the front end; (2) The user management service receives information about changes to user data; (3) The user management service sends messages about user data changes to the distributed message queue Nats in the form of topics; (4) The data storage service subscribes to the topic of user-related information from the distributed message queue Nats, and communicates with the distributed message queue Nats in real time to detect user-related information; (5) When the data storage service detects changes in user data in the distributed message queue Nats, the data storage service will persist the user information received from the distributed message queue Nats to the underlying storage device to complete the synchronization.

3. An electronic device, characterized in that: include: memory and at least one processor; Wherein, the memory stores a computer program; The at least one processor executes the computer program stored in the memory, so that the at least one processor executes the user information synchronization method based on the cloud storage system as claimed in claim 1.

4. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores a computer program, which can be executed by a processor to implement the user information synchronization method based on a cloud storage system as claimed in claim 1.

Citation Information

Patent Citations

  • Storage system of distributed real-time database

    CN107229639A

  • Home automation system including cloud and home message queue synchronization and related methods

    CN108141395A

  • Security-connected framework

    CN109871348A

  • Method and system for transmitting data through one-way communication between servers

    CN112839038A

  • Memory computing service framework and implementation system

    CN113486099A