Concurrent content providing system

Abstract

The invention provides a concurrent content providing system. The concurrent content providing system comprises a plurality of distributed servers, a message forwarding platform, a first database, a second database and a monitoring module, wherein the message forwarding platform, the first database and the second database are connected with the servers; the first database comprises document data, and core data and associated data corresponding to the document data; a second database; obtaining document data from the first database, and storing the document data in a second database; the server is provided with a memory module, and the server obtains associated data from the first database and stores the associated data in the memory module; and the monitoring module is used for monitoring a data change event in the first database, and broadcasting the data change event to all the servers through the message forwarding platform, so that the servers synchronize the associated data in the memory module in real time. By utilizing the data characteristics in the first database, when the server responds to the high-concurrency access request of the client, the server can only access the core data in the first database and obtain the document data and the associated data from the second database and the memory module; therefore, the response duration of the high-concurrency access request is reduced to a great extent; furthermore, a monitoring module is arranged to monitor the completion condition of the data change event of each server, so that the consistency of the associated data in each memory module in the whole distributed server is ensured.

Description

Technical Field

[0001] The present application relates to the field of computer technology, and in particular to a concurrent content providing system. Background Art

[0002] In the context of existing distributed system technology, data synchronization delay, system performance and scalability have become three major technical challenges, especially for data-intensive and user-interaction-intensive application scenarios such as e-commerce platforms.

[0003] Taking e-commerce platforms as an example, product information, user reviews, price changes and other data need to be synchronized in real time between multiple servers to ensure that users can obtain the latest data no matter which node they access. However, traditional data synchronization methods often rely on periodic database polling, which not only introduces significant data update delays, but also may not reflect data changes in a timely manner during peak hours, affecting user experience and transaction decisions.

[0004] At the same time, as the number of e-commerce platform users and transaction volume increase, the number of requests that the system needs to handle also increases. In addition, the rapid development of e-commerce platforms also requires that their backend systems can be flexibly expanded to support growing business needs.

[0005] In this case, how to effectively reduce the access frequency of the database and reduce read and write delays while ensuring data consistency has become the key to improving system performance.

[0006] However, traditional data synchronization solutions often have limitations in scalability because they may rely on centralized data management or synchronization mechanisms, which may cause bottlenecks or single points of failure in distributed environments, further limiting the system's scalability and stability.

[0007] A common solution in the prior art is to introduce a cache layer to reduce database pressure and maintain data consistency through the synchronization mechanism of a centralized database. However, the synchronization mechanism that relies on a centralized database places high demands on system performance when processing large-scale, high-concurrency read and write requests. When system performance is limited, it is prone to high response delays and data asynchrony. Especially during promotional activities or special events on e-commerce platforms, the surge in data access and update requests will put tremendous pressure on the central database. Data updates at the cache layer usually rely on expiration mechanisms or manually triggered updates, making it difficult to meet the real-time requirements for data in high-concurrency scenarios.

[0008] Therefore, finding a new technical solution that can overcome the above limitations and meet the requirements of efficient, real-time and scalable data synchronization has become an urgent need in the industry. Summary of the invention

[0009] The purpose of the present application is to solve the problems of long response time, poor data synchronization and real-time performance, and weak scalability of content providing systems in the prior art when facing high-concurrency access requests. Therefore, the present application provides a concurrent content providing system, which utilizes the data characteristics in the first database to synchronize related data to the memory modules of each distributed server, so that when the server responds to the high-concurrency access requests of the client, the server only accesses the core data in the first database, and obtains document data and related data from the second database and the memory module, thereby greatly reducing the response time for high-concurrency access requests; further, by setting a listening module to monitor the completion of data change events of each server, the consistency of related data in each memory module in the overall distributed server is ensured.

[0010] The embodiment of the present application provides a concurrent content providing system, including:

[0011] A plurality of distributed servers, a message forwarding platform connected to the servers, a first database and a second database, and a monitoring module;

[0012] The first database includes document data, and core data and associated data corresponding to the document data;

[0013] the second database; obtaining document data from the first database and storing it in the second database;

[0014] The server is provided with a memory module, and the server obtains the associated data from the first database and stores the data in the memory module;

[0015] The multiple distributed servers are all registered in the monitoring module, which is used to monitor data change events in the first database and broadcast the data change events to all the servers through the message forwarding platform so that the servers can synchronize the related data in the memory module in real time.

[0016] By adopting the above technical solution, the associated data is stored in the memory module of each server, so that the server can directly obtain the associated data from the local memory module. On the one hand, the associated data acquisition time is greatly reduced, and on the other hand, the amount of data acquired by the server from the first database and the time to access the first database are reduced, the performance pressure of the first database is reduced, and the time for the server to acquire core data and associated data is reduced as a whole. At the same time, a monitoring module is set to ensure that when the data in the first database is frequently updated, each server can obtain data change events through the message forwarding platform, and update the associated data of the memory module from the first database in real time.

[0017] In some embodiments, the content providing process includes:

[0018] Get the client's access request;

[0019] Based on the access request, using a search engine module to obtain document data matching the access request from the second database;

[0020] Based on the document data obtained by the search engine module, the server obtains the corresponding core data by accessing the first database; at the same time, the server calls the memory module to obtain the corresponding associated data;

[0021] The server feeds back to the client the document data acquired by the search engine module, as well as the core data and associated data acquired by the server.

[0022] By adopting the above technical solution, the search engine module can quickly parse the data expected to be obtained by the access request and determine the corresponding document data in the second database, so that the server can better match the corresponding core data and related data according to the corresponding document data; at the same time, the document data in the first database is synchronized to the second database, further reducing the access pressure on the first database.

[0023] In some embodiments, the process of real-time synchronization of the monitoring module includes:

[0024] Monitor data change events in the first database;

[0025] Determining whether the changed data in the data change event is the associated data;

[0026] If so, the data change event is packaged as an update message structure and placed in the message queue of the message forwarding platform. Then, the message forwarding platform broadcasts the data change event to each of the servers; the server obtains the corresponding change data from the first database according to the data change event and synchronizes it to the memory module.

[0027] By adopting the above technical solution, data change events can be quickly synchronized to each server through the high concurrency capability of the message forwarding platform, so that each server can perform data update operations synchronously, thereby greatly reducing the data synchronization time and ensuring the real-time and consistency of data in the memory modules of each server.

[0028] In some embodiments, the associated data is classified, and a corresponding message structure template is constructed according to the category of the associated data;

[0029] The process of real-time synchronization of the monitoring module also includes:

[0030] When it is determined that the changed data is associated data, determining the category of the associated data to which the changed data belongs, and using a corresponding message structure template to package the changed data into an update message structure;

[0031] The message structure template includes:

[0032] type CacheMsg struct {

[0033] Channel string`json:"channel"

[0034] Type string `json:"type"`

[0035] Key string `json:"key"`

[0036] Extra string `json:"extra"`

[0037] }

[0038] Among them, type indicates the category of the associated data, key indicates the further classification id of "type", and extra indicates the operations of adding, deleting, and modifying the database.

[0039] By adopting the above technical solution, by pre-building a message structure template, the data change event can be quickly converted into a corresponding update message structure, thereby improving the message forwarding speed of the message forwarding platform. Furthermore, whether the data change event is related data can be determined by whether it conforms to the message structure template.

[0040] In some embodiments, the real-time synchronization process of the monitoring module further includes:

[0041] After synchronizing the changed data to the memory module, the server notifies the monitoring module;

[0042] The monitoring module marks that the corresponding server completes the data change event;

[0043] Furthermore, the monitoring module checks at a first time interval whether all servers have completed the data change event;

[0044] If not, wait for a second period of time and then check whether all servers have completed the data change event; if so, end;

[0045] If after waiting for the second period of time, there is a server that has not completed the data change event, an alarm mechanism is triggered;

[0046] The first duration and the second duration are 0.1 to 30 seconds.

[0047] In some embodiments, the message forwarding platform adopts a publish-subscribe mechanism of Redis (Remote Dictionary Server);

[0048] The server is provided with a unique lightweight execution thread Goroutine, and the data change event broadcasted by the message forwarding platform is monitored through the lightweight execution thread Goroutine.

[0049] In some embodiments, the content providing system is used to provide commodity data services;

[0050] Wherein, the document data is set as a commodity index table;

[0051] The core data is set as the inventory data and price data corresponding to the goods in the goods index table

[0052] The associated data includes multiple categories, including classification tree, brand, hot keywords, and others;

[0053] The classification tree includes a hierarchical relationship tree of the classifications corresponding to all commodity sets;

[0054] The brand mentioned here refers to the brand to which each product belongs;

[0055] The hot keywords refer to high-frequency keywords in access requests;

[0056] Based on the hot keywords, a hot data area is constructed, and the core data related to the hot keywords in the first database is placed in the hot data area.

[0057] In some embodiments, the search engine module is provided with a natural speech processing module;

[0058] The search engine module performs word segmentation on the access request through the natural speech processing module, searches for at least one document data associated in the second database based on the word segmentation result, and feeds back the searched at least one document data result to the server;

[0059] The server accesses the first database according to the at least one document data fed back by the search engine module, and sequentially obtains the core data corresponding to the at least one document data;

[0060] At the same time, the server calls the memory module to directly obtain all associated data corresponding to the at least one document data according to the at least one document data fed back by the search engine module.

[0061] Other features and corresponding beneficial effects of the present invention are described in the latter part of the specification, and it should be understood that at least part of the beneficial effects become obvious from the description in the specification of the present invention. DETAILED DESCRIPTION

[0062] The following specific embodiments illustrate the implementation of the present invention, and those skilled in the art can easily understand other advantages and effects of the present invention from the contents disclosed in this specification. Although the description of the present invention will be introduced in conjunction with the preferred embodiment, this does not mean that the features of this invention are limited to this implementation. On the contrary, the purpose of introducing the invention in conjunction with the implementation is to cover other options or modifications that may extend based on the claims of the present invention. In order to provide a deep understanding of the present invention, the following description will include many specific details. The present invention can also be implemented without using these details. In addition, in order to avoid confusion or blurring the focus of the present invention, some specific details will be omitted in the description. It should be noted that the embodiments of the present invention and the features in the embodiments can be combined with each other without conflict.

[0063] The technical solution of the present invention will be described clearly and completely below. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of them. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0064] In the description of the present invention, it should be noted that the terms "first", "second", and "third" are used for descriptive purposes only and should not be understood as indicating or implying relative importance or implicitly indicating the number of the indicated technical features. Therefore, a feature defined as "first", "second", etc. may explicitly or implicitly include one or more of the features. In the description of the invention of this application, unless otherwise specified, "plurality" means two or more.

[0065] In the description of the present invention, it should be noted that, unless otherwise clearly specified and limited, the terms "installed", "connected", and "connected" should be understood in a broad sense, for example, it can be a fixed connection, a detachable connection, or an integral connection; it can be a mechanical connection or an electrical connection; it can be a direct connection, or it can be indirectly connected through an intermediate medium, or it can be the internal communication of two components. For ordinary technicians in this field, the specific meanings of the above terms in the present invention can be understood according to specific circumstances.

[0066] In order to make the objectives, technical solutions and advantages of the present invention more clear, the embodiments of the present invention will be described in further detail below.

[0067] The embodiment of the present application provides a concurrent content providing system, including:

[0068] A plurality of distributed servers, a message forwarding platform connected to the servers, a first database and a second database, and a monitoring module;

[0069] The first database includes document data, and core data and associated data corresponding to the document data;

[0070] Second database: obtaining document data from the first database and storing it in the second database;

[0071] The server is provided with a memory module, and the server obtains the associated data from the first database and stores the data in the memory module;

[0072] Multiple distributed servers are registered in the monitoring module, which is used to monitor data change events in the first database and broadcast the data change events to all servers through the message forwarding platform so that the servers can synchronize the associated data in the memory module in real time.

[0073] The above method stores the associated data in the memory module of each server, so that the server can directly obtain the associated data from the local memory module. On the one hand, it greatly reduces the time for obtaining the associated data. On the other hand, it reduces the amount of data that the server obtains from the first database and the time it takes to access the first database, reduces the performance pressure of the first database, and also reduces the time for the server to obtain core data and associated data as a whole. At the same time, a monitoring module is set to ensure that when the data in the first database is frequently updated, each server can obtain the data change event through the message forwarding platform, and update the associated data of the memory module from the first database in real time.

[0074] It should be noted that the first database has high security performance and good reliability; therefore, the first database contains complete and comprehensive data. The data of the second database and the server memory module need to be calibrated according to the first database.

[0075] Furthermore, core data is usually data that seriously affects the operation of the system, for example, data related to property settlement, which directly affects user decisions. Therefore, the core data needs to be stored in the first database and directly obtained from the first database to ensure the accuracy and security of the core data. At the same time, most of the core data (for example, property settlement data) is text data, which makes the server obtain the amount of data from the first database and the time to access the first database extremely short (compared to the server obtaining large amounts of data such as pictures and videos from the first database).

[0076] In one embodiment, the content providing system is used to provide commodity data services;

[0077] Among them, the document data is set as a commodity index table;

[0078] The core data is set to the inventory data and price data corresponding to the product in the product index table

[0079] Related data includes multiple categories, including classification trees, brands, popular keywords, and others;

[0080] The classification tree includes the hierarchical relationship tree of the classifications corresponding to all product sets;

[0081] Brand refers to the brand to which each product belongs; brands with similar names need to be quickly identified and normalized.

[0082] Hot keywords refer to high-frequency keywords in access requests;

[0083] Based on the hot keywords, a hot data area is constructed, and the core data related to the hot keywords in the first database is placed in the hot data area, so as to quickly return the user query results.

[0084] In other alternative implementations, the content providing system may also be applied to other data-intensive application scenarios with frequent user interactions, such as social media platforms, online education platforms, real-time video conferencing, etc.

[0085] In one embodiment, the content provision process includes:

[0086] Get the client's access request;

[0087] Based on the access request, using a search engine module to obtain document data matching the access request from a second database;

[0088] Based on the document data obtained by the search engine module, the server obtains the corresponding core data by accessing the first database; at the same time, the server calls the memory module to obtain the corresponding associated data;

[0089] The server feeds back to the client the document data acquired by the search engine module, as well as the core data and associated data acquired by the server.

[0090] The search engine module can quickly parse the data expected to be obtained by the access request and determine the corresponding document data in the second database, so that the server can better match the corresponding core data and related data according to the corresponding document data; at the same time, the document data in the first database is synchronized to the second database, further reducing the access pressure on the first database.

[0091] It should be noted that the memory module has a large cache bandwidth, so it has a strong anti-concurrency capability, and there is no distance limit for data acquisition in the memory module, so the response speed is extremely fast. The QPS of the memory is about 10 times that of the Redis server with the same configuration.

[0092] In one example, the client is set to an e-commerce platform. When a user searches for the keyword "milk" on the e-commerce platform, the client sends an access request to the server. The server receives the access request, which takes about 50 milliseconds. The search engine returns the corresponding document data according to the access request, which takes about 100 milliseconds. The server accesses the first database to query the core data, which takes about 80 milliseconds. The server obtains the associated data through the memory module, which takes about 1 millisecond. The server returns the data to the client, which takes about 50 milliseconds. At this time, the total time for the user to obtain the page on the e-commerce platform is about 300ms. In the prior art, when the client sends requests with high concurrency, it takes about 1 to 2 seconds to obtain the page.

[0093] In one embodiment, the process of real-time synchronization of the monitoring module includes:

[0094] Monitor data change events in the first database;

[0095] Determine whether the changed data in the data change event is related data;

[0096] If so, the data change event is packaged as an update message structure and placed in the message queue of the message forwarding platform. Then, the message forwarding platform broadcasts the data change event to each server; the server obtains the corresponding change data from the first database according to the data change event and synchronizes it to the memory module.

[0097] By adopting the above technical solution, data change events can be quickly synchronized to each server through the high concurrency capability of the message forwarding platform, so that each server can perform data update operations synchronously, thereby greatly reducing the data synchronization time and ensuring the real-time and consistency of data in the memory modules of each server.

[0098] In one embodiment, the associated data is classified, and a corresponding message structure template is constructed according to the category of the associated data;

[0099] The real-time synchronization process of the monitoring module also includes:

[0100] When it is determined that the changed data is associated data, the category of the associated data to which the changed data belongs is determined, and the changed data is packaged into an update message structure using a corresponding message structure template;

[0101] The message structure template includes:

[0102] type CacheMsg struct {

[0103] Channel string`json:"channel"

[0104] Type string `json:"type"`

[0105] Key string `json:"key"`

[0106] Extra string `json:"extra"`

[0107] }

[0108] Among them, type indicates the category of the associated data, key indicates the further classification id of "type", and extra indicates the operations of adding, deleting, and modifying the database.

[0109] In one embodiment, by pre-building a message structure template, the data change event can be quickly converted into a corresponding update message structure, thereby improving the message forwarding speed of the message forwarding platform. Further, whether the data change event is related data can be determined by whether it conforms to the message structure template.

[0110] In one embodiment, the real-time synchronization process of the monitoring module further includes:

[0111] After the server synchronizes the changed data to the memory module, it notifies the monitoring module;

[0112] The monitoring module marks the corresponding server as completing the data change event;

[0113] Furthermore, the monitoring module checks at a first time interval whether all servers have completed the data change event;

[0114] If not, wait for the second time and then check whether all servers have completed the data change event; if so, end;

[0115] If after waiting for the second period of time, there are servers with uncompleted data change events, the alarm mechanism will be triggered;

[0116] The first duration and the second duration are 0.1 to 30 seconds.

[0117] It should be noted that each server takes several seconds to obtain and process messages, which is not very timely compared to millimeter-level responses. However, the associated data in the memory module is non-core data, such as product classification, brand, keyword and other data, so the second-level lag of the associated data will not affect the settlement of user property and cause related financial losses. At the same time, users do not have very high requirements for the error of associated data, so the update frequency of seconds in exchange for the millisecond-level data acquisition experience on the user side can greatly improve the user experience.

[0118] In one embodiment, the message forwarding platform adopts a publish-subscribe mechanism of Redis (Remote Dictionary Server);

[0119] An independent lightweight execution thread Goroutine is set up in the server, and the lightweight execution thread Goroutine monitors the data change events broadcast by the message forwarding platform.

[0120] The above method uses the publish-subscribe mechanism of Redis to achieve efficient message delivery. By creating channels in Redis, various parts of the system can publish messages or subscribe to channels. When data is updated, data change events will be published to a specific channel, and all service nodes (i.e. servers) subscribed to the channel can receive messages of these data change events in real time and update the local cache accordingly. This achieves high-concurrency message delivery, reduces network latency, and improves the real-time performance of data synchronization.

[0121] In one embodiment, the search engine module is provided with a natural speech processing module;

[0122] The search engine module segments the access request through the natural speech processing module, searches for at least one document data associated in the second database based on the segmentation result, and feeds back the searched at least one document data result to the server;

[0123] The server accesses the first database according to the at least one document data fed back by the search engine module, and sequentially obtains the core data corresponding to the at least one document data;

[0124] At the same time, the server calls the memory module to directly obtain all associated data corresponding to the at least one document data according to the at least one document data fed back by the search engine module.

[0125] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention, rather than to limit it. Although the present invention has been described in detail with reference to the aforementioned embodiments, those skilled in the art should understand that they can still modify the technical solutions described in the aforementioned embodiments, or replace some or all of the technical features therein with equivalents. However, these modifications or replacements do not cause the essence of the corresponding technical solutions to deviate from the scope of the technical solutions of the embodiments of the present invention.

Claims

1. A concurrent content providing system, characterized in that: include: A plurality of distributed servers, a message forwarding platform connected to the servers, a first database and a second database, and a monitoring module; The first database includes document data, and core data and associated data corresponding to the document data; the second database; obtaining document data from the first database and storing it in the second database; The server is provided with a memory module, and the server obtains the associated data from the first database and stores the data in the memory module; The multiple distributed servers are all registered in the monitoring module, which is used to monitor data change events in the first database and broadcast the data change events to all the servers through the message forwarding platform so that the servers can synchronize the related data in the memory module in real time.

2. A concurrent content providing system as claimed in claim 1, characterized in that: The content providing process includes: Get the client's access request; Based on the access request, using a search engine module to obtain document data matching the access request from the second database; Based on the document data obtained by the search engine module, the server obtains the corresponding core data by accessing the first database; at the same time, the server calls the memory module to obtain the corresponding associated data; The server feeds back to the client the document data acquired by the search engine module, as well as the core data and associated data acquired by the server.

3. A concurrent content providing system as claimed in claim 1, characterized in that: The process of real-time synchronization of the monitoring module includes: Monitor data change events in the first database; Determining whether the changed data in the data change event is the associated data; If so, the data change event is packaged as an update message structure and placed in the message queue of the message forwarding platform. Then, the message forwarding platform broadcasts the data change event to each of the servers; the server obtains the corresponding change data from the first database according to the data change event and synchronizes it to the memory module.

4. A concurrent content providing system as claimed in claim 3, characterized in that: Classifying the associated data, and constructing a corresponding message structure template according to the category of the associated data; The process of real-time synchronization of the monitoring module also includes: When it is determined that the changed data is associated data, the category of the associated data to which the changed data belongs is determined, and the changed data is packaged into an update message structure using a corresponding message structure template; The message structure template includes: type CacheMsg struct { Channel string`json:"channel" Type string `json:"type"` Key string `json:"key"` Extra string `json:"extra"` } Among them, type indicates the category of the associated data, key indicates the further classification id of "type", and extra indicates the operations of adding, deleting, and modifying the database.

5. A concurrent content providing system as claimed in claim 3, characterized in that: The process of real-time synchronization of the monitoring module also includes: After synchronizing the changed data to the memory module, the server notifies the monitoring module; The monitoring module marks that the corresponding server completes the data change event; Furthermore, the monitoring module checks at a first time interval whether all servers have completed the data change event; If not, wait for a second period of time and then check whether all servers have completed the data change event; if so, end; If after waiting for the second period of time, there is a server that has not completed the data change event, an alarm mechanism is triggered; The first duration and the second duration are 0.1 to 30 seconds.

6. A concurrent content providing system as claimed in claim 3, characterized in that: The message forwarding platform adopts the publish-subscribe mechanism of Redis (Remote Dictionary Server); The server is provided with a unique lightweight execution thread Goroutine, and the data change event broadcasted by the message forwarding platform is monitored through the lightweight execution thread Goroutine.

7. A concurrent content providing system as claimed in claim 1, characterized in that: The content providing system is used to provide commodity data services; Wherein, the document data is set as a commodity index table; The core data is set as the inventory data and price data corresponding to the commodities in the commodity index table. The associated data includes multiple categories, specifically including classification tree, brand, hot keywords, and others; The classification tree includes a hierarchical relationship tree of the classifications corresponding to all commodity sets; The brand mentioned here refers to the brand to which each product belongs; The hot keywords refer to high-frequency keywords in access requests; Based on the hot keywords, a hot data area is constructed, and the core data related to the hot keywords in the first database is placed in the hot data area.

8. A concurrent content providing system as claimed in claim 2, characterized in that: The search engine module is provided with a natural speech processing module; The search engine module performs word segmentation on the access request through the natural speech processing module, searches for at least one document data associated in the second database based on the word segmentation result, and feeds back the searched at least one document data result to the server; The server accesses the first database according to the at least one document data fed back by the search engine module, and sequentially obtains the core data corresponding to the at least one document data; At the same time, the server calls the memory module to directly obtain all associated data corresponding to the at least one document data according to the at least one document data fed back by the search engine module.