Large model processing method and device based on distributed cache, equipment and medium

Abstract

The invention provides a large model processing method and device based on distributed cache, equipment and a medium, and relates to the technical field of large language models and data processing in artificial intelligence. According to the specific implementation scheme, a center scheduling service receives response information sent by a large model service, the center scheduling service writes the response information into a kafka service based on network load information obtained in advance, and for each Redis cache in at least one Redis cache, the response information in the kafka service is consumed and locally cached. By means of the method, the concurrent processing capacity, the load balancing and expansibility and the data reliability and durability of the system are improved, the user experience is optimized, and the number of users supported by the service can be remarkably increased.

Description

Technical Field

[0001] The present disclosure relates to the fields of large language models and data processing technology in artificial intelligence, and in particular to a large model processing method, device, equipment and medium based on distributed cache. Background Art

[0002] A large model is a large language model that is trained based on a deep learning algorithm and large amounts of training data. It has excellent generalization and expression capabilities and can be widely used in image recognition, natural language processing, speech recognition, and other fields. However, as large models are increasingly used in various fields, their stability becomes critical.

[0003] In the prior art, in order to ensure the availability and stability of large model services, the user's access speed is limited.

[0004] However, the above method of limiting user access speed to ensure model stability will make large models unable to support massive user services, which in turn leads to poor user experience. Summary of the invention

[0005] The present disclosure provides a method, apparatus, device and medium for large model processing based on distributed cache.

[0006] According to a first aspect of the present disclosure, a large model processing method based on distributed cache is provided, comprising:

[0007] The central scheduling service receives the response information sent by the large model service, wherein the response information is feedback information corresponding to the user's access demand;

[0008] Based on the pre-acquired network load information, the central scheduling service writes the response information into the Kafka service;

[0009] For each Redis cache in at least one Redis cache, the response information in the Kafka service is consumed and cached locally.

[0010] According to a second aspect of the present disclosure, a large model processing device based on distributed cache is provided, comprising:

[0011] A receiving unit, used for the central scheduling service to receive the response information sent by the large model service, wherein the response information is feedback information corresponding to the user's access demand;

[0012] A writing unit, configured for the central scheduling service to write the response information into the Kafka service based on the pre-acquired network load information;

[0013] The consumption unit is used to consume the response information in the kafka service and perform local caching for each Redis cache in at least one Redis cache.

[0014] According to a third aspect of the present disclosure, there is provided an electronic device, including:

[0015] at least one processor; and

[0016] a memory communicatively connected to the at least one processor; wherein,

[0017] The memory stores instructions executable by the at least one processor, and the instructions are executed by the at least one processor to enable the at least one processor to perform the method described in any one aspect.

[0018] According to a fourth aspect of the present disclosure, a non-transitory computer-readable storage medium storing computer instructions is provided, wherein the computer instructions are used to cause the computer to execute the method described in any one of the aspects.

[0019] According to a fifth aspect of the present disclosure, a computer program product is provided, comprising: a computer program, wherein the computer program is stored in a readable storage medium, at least one processor of an electronic device can read the computer program from the readable storage medium, and the at least one processor executes the computer program so that the electronic device executes the method described in any one aspect.

[0020] The technology disclosed in the present invention solves the problem of how to enable a large model to support massive user services, thereby improving user experience.

[0021] It should be understood that the content described in this section is not intended to identify the key or important features of the embodiments of the present disclosure, nor is it intended to limit the scope of the present disclosure. Other features of the present disclosure will become easily understood through the following description. BRIEF DESCRIPTION OF THE DRAWINGS

[0022] The accompanying drawings are used to better understand the present solution and do not constitute a limitation of the present disclosure.

[0023] Figure 1 It is a system architecture diagram that can implement the large model processing method based on distributed cache of the embodiment of the present disclosure;

[0024] Figure 2 A schematic diagram of a large model processing method based on distributed cache provided in the first embodiment of the present disclosure;

[0025] Figure 3 A schematic diagram of a large model processing method based on distributed cache provided in the second embodiment of the present disclosure;

[0026] Figure 4 A schematic diagram of a large model processing method based on distributed cache provided in the third embodiment of the present disclosure;

[0027] Figure 5 A schematic diagram of a large model processing method based on distributed cache provided in the fourth embodiment of the present disclosure;

[0028] Figure 6 A schematic diagram of a large model processing method based on distributed cache provided in the fifth embodiment of the present disclosure;

[0029] Figure 7 A flowchart of a large model processing method based on distributed cache provided in the sixth embodiment of the present disclosure;

[0030] Figure 8 A flowchart of a large model processing method based on distributed cache provided in the seventh embodiment of the present disclosure;

[0031] Fig. 9 A schematic diagram of a large model processing method based on distributed cache provided in the eighth embodiment of the present disclosure;

[0032] Fig.10 A schematic diagram of a large model processing method based on distributed cache provided in the ninth embodiment of the present disclosure;

[0033] Fig.11 A structural block diagram of a large model processing device based on distributed cache provided in an embodiment of the present disclosure;

[0034] Fig.12 A schematic block diagram of an electronic device for implementing an embodiment of the present disclosure. DETAILED DESCRIPTION

[0035] The following is a description of exemplary embodiments of the present disclosure in conjunction with the accompanying drawings, including various details of the embodiments of the present disclosure to facilitate understanding, which should be considered as merely exemplary. Therefore, it should be recognized by those of ordinary skill in the art that various changes and modifications may be made to the embodiments described herein without departing from the scope and spirit of the present disclosure. Similarly, for the sake of clarity and conciseness, descriptions of well-known functions and structures are omitted in the following description.

[0036] In the related art, a big model is a large language model that is obtained through training with large-scale training data based on a deep learning algorithm. It has excellent generalization and expression capabilities and can be widely used in image recognition, natural language processing, speech recognition and other fields. However, with the increasing application of big models in various fields, its stability has become the key. In the prior art, in order to ensure the availability and stability of big model services, the user's access speed is limited. However, the above-mentioned method of limiting the user's access speed to ensure the stability of the model will make the big model unable to support massive user services, which will lead to poor user experience.

[0037] In response to this existing problem, the technical conception process of the present invention is as follows: In the process of studying the big model, the inventor found that in order to ensure the availability and stability of the service, the big model will limit the user's access speed, and thus cannot support a large number of traffic requests, and will also lead to slow response speed on the user side and poor user experience. Therefore, the inventor considered whether it is possible to configure multiple Redis-based caches, and maximize the number of users that the service can support by horizontally expanding the cache. And coordinate each cache based on the central scheduling service and ensure data consistency. In order to ensure that the response information is processed in a timely and efficient manner, thereby improving the responsiveness of the entire system. The response information sent by the large model service can be written to the kafka service based on the central scheduling service, and then multiple Redis-based caches can be locally cached by consuming data in the kafka service.

[0038] Figure 1 is a system architecture diagram that can implement the large model processing method based on distributed cache in the embodiment of the present disclosure, such as Figure 1 As shown, the system architecture includes front-end service, central scheduling service, kafka service, multiple Redis-based caches and large model service. Among them, the front-end service is connected to the central scheduling service, the central scheduling service is connected to multiple Redis-based caches through the raft protocol, the kafka service is connected to multiple Redis-based caches, and the large model service is connected to the central scheduling service.

[0039] After the large model service obtains the response information through reasoning, it sends the response information to the central scheduling service. The central scheduling service writes the response information to the Kafka service based on the network load information and the status of the consumer instance in the Kafka service. The central scheduling service updates the metadata of each Redis-based cache based on the status information and network load information of multiple Redis-based caches, and synchronizes the metadata to each Redis-based cache through the Raft protocol. The Redis-based cache can consume the data stored in the Kafka service and cache it locally based on the metadata synchronized by the central scheduling service.

[0040] When the front-end service receives a large number of access requests, it will query each Redis-based cache to see if there is corresponding response information. If so, the local cached response information will be directly fed back to the user. If not, the access request will be sent to the large model service for reasoning based on the central scheduling service, so as to obtain the response information and feed it back to the user. At the same time, the central scheduling service will cache the response information based on the above method.

[0041] Optionally, the above components can be deployed in the same device. In a small system or development and testing environment, in order to simplify deployment and configuration, all services can be deployed on the same physical or virtual device. It can also be deployed in different devices, each service can be deployed on an independent physical or virtual server, or deployed and managed through containerization technology (such as Docker) and orchestration tools (such as Kubernetes). This embodiment is not specifically limited.

[0042] In one possible implementation, the communication between services can be done in a variety of ways, such as:

[0043] HTTP / HTTPS: Applicable to the communication between front-end services and central scheduling services and large model services.

[0044] TCP / IP: Almost all underlying communications between services rely on the TCP / IP protocol. TCP is a connection-oriented, reliable, byte stream-based transport layer communication protocol that ensures the correct transmission of data packets.

[0045] Raft protocol: Specific to the communication between the central scheduling service and multiple Redis-based caches, used to achieve distributed consistency. Raft is a consistency algorithm for managing replication logs, which handles all write operations by electing a leader and synchronizing data through log replication.

[0046] Kafka message queue: Kafka services can be used to asynchronously transmit messages between multiple Redis-based caches and central scheduling services. Kafka is a distributed stream processing platform that can handle high-throughput data.

[0047] Redis protocol: Suitable for communication between multiple Redis-based caches. Although the central scheduling service manages the state of the Redis cache through the Raft protocol, Redis itself also supports key-value storage and retrieval through its proprietary protocol.

[0048] Other protocols (such as gRPC, AMQP, etc.) or tools (such as message queues RabbitMQ, ZeroMQ, etc.) can also be used to achieve communication between services.

[0049] The specific communication method is not specifically limited in this embodiment.

[0050] The present invention discloses a large model processing method, device, equipment and storage medium based on distributed cache, which are applied to the large language model and data processing technology fields in artificial intelligence. Based on the central scheduling service, multiple cache instances are coordinated to consume the historically processed data of the large model stored in the Kafka service. Then, when the user requests, the response information existing in any cache can be directly fed back to the user. By horizontally expanding the cache, the number of users that the service can support is maximized, and the response speed and user experience on the user side are improved.

[0051] The following specific embodiments are used to describe in detail the technical solution of the present invention and how the technical solution of the present invention solves the above-mentioned technical problems. The following specific embodiments can be combined with each other, and the same or similar concepts or processes may not be repeated in some embodiments. The embodiments of the present invention will be described below in conjunction with the accompanying drawings.

[0052] In the technical solution of the present disclosure, the collection, storage, use, processing, transmission, provision and disclosure of user personal information involved are in compliance with the provisions of relevant laws and regulations and do not violate public order and good morals.

[0053] The following is an introduction to the specific implementation of the large model processing method based on distributed cache provided by the present invention.

[0054] Figure 2 A schematic diagram of a large model processing method based on distributed caching provided in the first embodiment of the present disclosure is shown in FIG. Figure 2 The large model processing method based on distributed cache shown in the figure can be configured as a large model processing device based on distributed cache, and the device can be located in an electronic device. The large model processing method based on distributed cache can include the following steps:

[0055] S201: The central scheduling service receives the response information sent by the large model service.

[0056] S202: Based on the pre-acquired network load information, the central scheduling service writes the response information into the Kafka service.

[0057] In order to support high-concurrency processing, after receiving the response information, the large model service sends the response information to the central scheduling service, and the central scheduling service writes the response information to the Kafka service. Among them, the Kafka service is used to process high-throughput data streams and can support the publishing and subscription of millions of messages per second. This enables the service to handle requests and responses from a large number of users at the same time.

[0058] Exemplarily, the central scheduling service first collects network load information in real time or regularly through network monitoring tools or systems (such as network traffic monitoring software). It may include bandwidth usage, latency, packet loss rate, etc. The collected network load information is then further analyzed to evaluate the health and capacity of the current network. In order to optimize the data transmission path and efficiency, the central scheduling service determines the specific partition where the response information is written to the Kafka service based on the analyzed network conditions and the load conditions of the Kafka service itself.

[0059] In one possible implementation, the central scheduling service determines one or more optimal partitions to write the response information, for example, selecting the partition with the lowest current load to balance the overall load of the Kafka service. If the response information contains a partition key, ensure that data with the same key is written to the same partition. Consider the historical performance data of the partitions and select those partitions that performed better in the past. Consider the replicas of the partitions and select partitions that have enough replicas and are distributed on different nodes to improve the fault tolerance of the system. After determining the best partition, the central scheduling service configures the Kafka producer to write the response information to the partition.

[0060] Optionally, after the write operation is completed, the central scheduling service will monitor the subsequent performance of the partition, including indicators such as write success rate, latency, and throughput. If any problems or performance degradation are found, the central scheduling service can also adjust the partition strategy. At the same time, the central scheduling service will also feedback the monitoring results to the system administrator for further optimization and adjustment.

[0061] For example, suppose there is a Kafka cluster with three nodes and six partitions (two partitions per node). The central scheduling service finds that the two partitions of node A (partition 1 and partition 2) are currently under low load, while the partitions of node B and node C are under high load. At this time, if there is a response message containing a partition key that needs to be written to Kafka, the central scheduling service will check whether the partition corresponding to the key has been determined. If it has not been determined or can be reallocated, the partition 1 of node A (assuming it is the best partition selected according to some strategy) may be selected to write the response message. If the response message does not contain a partition key, the central scheduling service can select a partition for writing based on polling or other partitioning strategies.

[0062] S203: For each Redis cache in at least one Redis cache, consume the response information in the Kafka service and cache it locally.

[0063] In this step, in order to achieve efficient local caching, multiple Redis caches are configured, and each Redis cache consumes the response information in the Kafka service for local caching.

[0064] Exemplarily, each Redis cache needs to be configured with one or more Kafka consumers. These consumers will subscribe to specific topics in Kafka, which contain response information that needs to be cached. Among them, the consumer configuration may include: the address of the Kafka cluster, authentication information, subscribed topics, consumer group ID to ensure that multiple consumers can consume messages in a load-balanced manner, message deserializers, and other consumer configuration items, such as offset, commit strategy, batch size, etc.

[0065] After the consumer is started, it will continuously pull messages from Kafka. Each message pulled is a batch of records, which contains multiple response information. For each message pulled from Kafka, the consumer can also process it. The processing process may include: deserializing the message body, converting it from the format stored in Kafka (such as byte array) to a format that the application can understand (such as JSON, string, etc.), parsing the message content, extracting the response information that needs to be cached, and performing some additional processing or verification on the response information according to business needs. After processing the message, the consumer needs to cache the response information in the local Redis cache: use Redis client libraries (such as Jedis, Lettuce, etc.) to interact with the Redis cache, select the appropriate Redis data structure to store the response information (such as string, hash table, list, set, etc.), and execute Redis commands to write the response information to the cache.

[0066] Optionally, you can also set an expiration time (TTL) as needed, set an expiration time for the cached response information, and automatically delete it from the cache after the expiration. Set cache invalidation to ensure that old data in the cache is updated or deleted in a timely manner when the source data changes. Set cache elimination to eliminate old cache items according to certain strategies (such as LRU, LFU, etc.) when the cache space is insufficient.

[0067] In one possible implementation, the central scheduling service can monitor the consumption and caching process in real time. Specifically, it can monitor the performance and status of Kafka consumers, record the logs of message consumption, including successful and failed messages, and monitor the performance and status of Redis cache.

[0068] The large model processing method based on distributed cache provided in this embodiment is that the central scheduling service receives the response information sent by the large model service, and based on the pre-acquired network load information, the central scheduling service writes the response information into the kafka service, and for each Redis cache in at least one Redis cache, consumes the response information in the kafka service and caches it locally. The above method improves the system's concurrent processing capability, load balancing and scalability, data reliability and persistence, and optimizes the user experience, which can significantly increase the number of users supported by the service.

[0069] Figure 3 This is a flow chart of a large model processing method based on distributed cache provided in the second embodiment of the present disclosure, refer to Figure 3 The large model processing method based on distributed cache shown is different from the aforementioned embodiment in that, in this embodiment, consuming the response information in the Kafka service and caching it locally can be refined as: based on the metadata synchronized by the central scheduling service, consuming the response information in the Kafka service and caching it locally.

[0070] like Figure 3 As shown, the method comprises the following steps:

[0071] S301: The central scheduling service receives the response information sent by the large model service.

[0072] S302: Based on the pre-acquired network load information, the central scheduling service writes the response information into the Kafka service.

[0073] The specific implementation of step S301 and step S302 is the same as that in the above-mentioned embodiment, and will not be repeated here.

[0074] S303: For each Redis cache in at least one Redis cache, based on the metadata synchronized by the central scheduling service, consume the response information in the Kafka service and perform local caching.

[0075] In this step, in order to coordinate various cache instances and ensure data consistency, the central scheduling service can synchronize metadata to each Redis cache. The metadata includes the consumption partition and offset of the Redis cache consumption data, and then consume the response information based on the metadata. The consumption partition refers to which partition of the Kafka service the Redis cache consumes the response information, and the offset refers to the progress of the Redis cache consuming the response information in the consumption partition of the Kafka service.

[0076] The specific consumption and caching process is the same as the specific implementation method in the aforementioned embodiment, and will not be repeated here.

[0077] The large model processing method based on distributed cache provided in this embodiment is that the central scheduling service receives the response information sent by the large model service. Based on the pre-acquired network load information, the central scheduling service writes the response information to the Kafka service. For each Redis cache in at least one Redis cache, based on the metadata synchronized by the central scheduling service, the response information in the Kafka service is consumed and cached locally. The central scheduling service intelligently writes the response information sent by the large model service to the Kafka service based on the network load information, ensuring that the data can enter the processing flow efficiently, reducing the delay of the data during the transmission process, and ensuring that the data can be processed and transmitted in a timely and orderly manner even in high concurrency. The response speed of the overall service and the efficiency of data access are improved.

[0078] Figure 4 This is a flow chart of a large model processing method based on distributed cache provided in the third embodiment of the present disclosure, refer to Figure 4 The large model processing method based on distributed cache shown is different from the aforementioned embodiment in that, in this embodiment, for each Redis cache, the central scheduling service updates the metadata corresponding to the Redis cache based on the mapping table, and the central scheduling service synchronizes the metadata corresponding to the Redis cache to the Redis cache through the Raft protocol.

[0079] like Figure 4 As shown, the method comprises the following steps:

[0080] S401: The central scheduling service receives the response information sent by the large model service.

[0081] S402: Based on the pre-acquired network load information, the central scheduling service writes the response information into the Kafka service.

[0082] The specific implementation of step S401 and step S402 is the same as that in the above-mentioned embodiment, and will not be repeated here.

[0083] S403: For each Redis cache, the central scheduling service updates the metadata corresponding to the Redis cache based on the mapping table.

[0084] In this step, in order to fully and reasonably coordinate the Redis cache, the metadata corresponding to the Redis cache can be updated based on a preset mapping table, wherein the mapping table includes a mapping relationship between each Redis cache and a consumer instance in the kafka service.

[0085] For example, assuming that the mapping relationships included in the mapping table are cache 1-partition 1, cache 2-partition 3, cache 3-partition 3, based on the mapping relationships, the consumption partition of the metadata of cache 1 is updated to partition 1, the consumption partition of the metadata of cache 2 is updated to partition 3, and the consumption partition of the metadata of cache 3 is updated to partition 3.

[0086] It should be noted that the offset in the metadata is fed back by the Kafka service to the central scheduling service. The offset in the metadata can be directly updated according to the offset fed back by the Kafka service.

[0087] Optionally, the mapping table can also include: Redis cache ID, which uniquely identifies each Redis cache instance. Kafka consumer group ID, which identifies the Kafka consumer group associated with the Redis cache. Kafka topic, which specifies the Kafka topic that the consumer group needs to subscribe to. It can also include the Redis address, port, password, key-value mapping rules, data expiration policy, etc.

[0088] S404: The central scheduling service synchronizes the metadata corresponding to the Redis cache to the Redis cache through the Raft protocol.

[0089] In this step, in order to ensure data consistency and avoid duplicate consumption, the central scheduling service will periodically synchronize metadata to the Redis cache based on the Raft protocol.

[0090] For example, in the Raft protocol, nodes are usually divided into three roles: Leader, Follower, and Candidate. Leader is responsible for processing all requests, Follower receives log entries from Leader and updates local status, and Candidate is a temporary role in the election process, used to run for Leader.

[0091] The central scheduling service needs to configure multiple Raft nodes, which can be physically independent servers or virtual machines. Each node needs to run an implementation of the Raft protocol and be able to communicate with other nodes.

[0092] When the system starts, all nodes are in the Follower state. If there is no Leader for a period of time (for example, due to network partition or Leader failure), a Leader election will occur. During the election, the Candidate node will send voting requests to other nodes and try to get more than half of the votes. Once a Candidate node successfully becomes a Leader, it will start processing requests.

[0093] When the central scheduling service needs to update the metadata of the Redis cache, it sends an update request to the Leader node of the Raft cluster. After receiving the request, the Leader node appends it to the local log as a new log entry and sends an AppendEntries request to other Follower nodes to copy the log entry.

[0094] After receiving the AppendEntries request, the Follower node writes the log entry to the local log and sends a response to the Leader. After receiving enough responses (usually more than half of the Follower nodes), the Leader marks the log entry as committed. At this point, the metadata update is considered successful and can be applied to the Redis cache.

[0095] Once the log entry is committed, the leader node notifies all follower nodes and the central scheduling service that the update is complete. The central scheduling service can then apply the update to the Redis cache or notify other system components to perform corresponding operations.

[0096] S405: For each Redis cache in at least one Redis cache, based on the metadata synchronized by the central scheduling service, consume the response information in the Kafka service and perform local caching.

[0097] The specific implementation of step S405 is the same as that in the above-mentioned embodiment, and will not be repeated here.

[0098] The large model processing method based on distributed cache provided in this embodiment is that the central scheduling service receives the response information sent by the large model service. Based on the pre-acquired network load information, the central scheduling service writes the response information to the kafka service. For each Redis cache, the central scheduling service updates the metadata corresponding to the Redis cache based on the mapping table. The central scheduling service synchronizes the metadata corresponding to the Redis cache to the Redis cache through the Raft protocol. For each Redis cache in at least one Redis cache, based on the metadata synchronized by the central scheduling service, the response information in the kafka service is consumed and cached locally. The real-time scheduling strategy based on network conditions optimizes the data transmission path, reduces network congestion, and improves data transmission efficiency. Data load balancing is achieved, and overloading of a single node is avoided, ensuring the overall stability and reliability of the system. At the same time, data access speed is improved and system latency is reduced.

[0099] Figure 5 This is a flow chart of a large model processing method based on distributed cache provided in the fourth embodiment of the present disclosure, refer to Figure 5 The large model processing method based on distributed cache shown is different from the aforementioned embodiment in that, in this embodiment, the central scheduling service can also update the mapping table based on the status information of each Redis cache, the network load information and the load information of the Kafka service.

[0100] like Figure 5 As shown, the method comprises the following steps:

[0101] S501: The central scheduling service receives the response information sent by the large model service.

[0102] S502: Based on the pre-acquired network load information, the central scheduling service writes the response information into the Kafka service.

[0103] The specific implementation of step S501 and step S502 is the same as that in the above-mentioned embodiment, and will not be repeated here.

[0104] S503: The central scheduling service updates the mapping table based on the status information of each Redis cache, the network load information, and the load information of the Kafka service.

[0105] In this step, in order to balance the load and ensure the stability and efficiency of access, the central scheduling service can monitor the Redis cache and Kafka service in real time, and then update the mapping table based on the status information of the Redis cache, the network load information and the load information of the Kafka service.

[0106] Optionally, the central scheduling service can monitor key indicators such as the hit rate, memory usage, number of connections, and command execution time of the Redis cache. Monitor the latency, bandwidth usage, and packet loss rate of different network paths to understand the impact of the network on data transmission. Unbalanced network load may cause data transmission delays or failures, which in turn affects system performance. Monitor the partition load, message accumulation, and consumer lag of the Kafka cluster.

[0107] The central dispatch service analyzes the collected monitoring data in real time to identify hot spots, bottlenecks, and potential overload risks in the system. Based on the analysis results, the current load of the Redis cache and Kafka services is evaluated to determine whether the mapping table needs to be adjusted to balance the load.

[0108] Develop specific adjustment strategies based on the load assessment results. For example, if a Redis cache instance is overloaded, consider migrating some data to other instances with lower loads; if a partition consumer of Kafka is lagging too much, consider adding consumer instances or reallocating consumers to balance the load.

[0109] S504: For each Redis cache, the central scheduling service updates the metadata corresponding to the Redis cache based on the updated mapping table.

[0110] S505: The central scheduling service synchronizes the metadata corresponding to the Redis cache to the Redis cache through the Raft protocol.

[0111] S506: For each Redis cache in at least one Redis cache, based on the metadata synchronized by the central scheduling service, consume the response information in the Kafka service and perform local caching.

[0112] The specific implementation of step S504 to step S506 is the same as that in the above-mentioned embodiment, and will not be repeated here.

[0113] The large model processing method based on distributed cache provided in this embodiment is that the central scheduling service receives the response information sent by the large model service. Based on the pre-acquired network load information, the central scheduling service writes the response information into the kafka service. The central scheduling service updates the mapping table based on the status information of each Redis cache, the network load information and the load information of the kafka service. For each Redis cache, the central scheduling service updates the metadata corresponding to the Redis cache based on the updated mapping table. The central scheduling service synchronizes the metadata corresponding to the Redis cache to the Redis cache through the Raft protocol. For each Redis cache in at least one Redis cache, based on the metadata synchronized by the central scheduling service, the response information in the kafka service is consumed and cached locally. The scheduling strategy based on network load information realizes intelligent distribution of data, avoids network congestion, and improves the efficiency of data transmission. And based on the local cache strategy, the number of visits to the large model service is reduced, the data reading speed is improved, and the load of the large model service is reduced.

[0114] Figure 6 A flow chart of a large model processing method based on distributed cache provided in the fifth embodiment of the present disclosure, referring to Figure 6 The large model processing method based on distributed cache shown is different from the aforementioned embodiment in that, in this embodiment, based on the metadata synchronized by the central scheduling service, the response information in the Kafka service is consumed, which can be refined as follows: for each Redis cache, the Redis cache determines whether the partition of the Redis cache consumption data has changed based on the current consumption partition and the consumption partition synchronized by the central scheduling service. If it is determined that the partition of the Redis cache consumption data has changed, the response information in the consumption partition synchronized by the central scheduling service is consumed based on the offset synchronized by the central scheduling service.

[0115] like Figure 6 As shown, the method comprises the following steps:

[0116] S601: The central scheduling service receives the response information sent by the large model service.

[0117] S602: Based on the pre-acquired network load information, the central scheduling service writes the response information into the Kafka service.

[0118] The specific implementation of step S601 and step S602 is the same as that in the above-mentioned embodiment, and will not be repeated here.

[0119] S603: For each Redis cache, the Redis cache determines whether the partition of the Redis cache consumption data has changed according to the current consumption partition and the consumption partition synchronized with the central scheduling service.

[0120] S604: If it is determined that the partition of the Redis cache consumption data has changed, based on the offset synchronized by the central scheduling service, the response information in the consumption partition synchronized by the central scheduling service is consumed and cached locally.

[0121] For each Redis cache, after receiving the metadata synchronized by the central scheduling service, the Redis cache compares the consumption partition in the metadata with the current consumption partition. If they are the same partition, the response information will continue to be consumed based on the offset in the metadata and cached locally.

[0122] The large model processing method based on distributed cache provided in this embodiment, the central scheduling service receives the response information sent by the large model service, and based on the pre-acquired network load information, the central scheduling service writes the response information to the kafka service. For each Redis cache, the Redis cache determines whether the partition of the Redis cache consumption data has changed based on the current consumption partition and the consumption partition synchronized by the central scheduling service. If it is determined that the partition of the Redis cache consumption data has changed, based on the offset synchronized by the central scheduling service, the response information in the consumption partition synchronized by the central scheduling service is consumed and cached locally. The above method realizes efficient data consumption and caching, enhances system scalability, and improves system stability and reliability through dynamic partition management. At the same time, it significantly improves the system's ability and efficiency in processing large model service response information. A service that supports massive users is realized.

[0123] Figure 7 This is a flow chart of a large model processing method based on distributed cache provided in the sixth embodiment of the present disclosure, refer to Figure 7The large model processing method based on distributed cache shown is different from the aforementioned embodiment in that, in this embodiment, based on the metadata synchronized by the central scheduling service, the response information in the Kafka service is consumed, and it can also be refined as follows: for each Redis cache, the Redis cache determines whether the partition of the Redis cache consumption data has changed based on the current consumption partition and the consumption partition synchronized by the central scheduling service. If it is determined that the partition of the Redis cache consumption data has not changed, then based on the offset synchronized by the central scheduling service, continue to consume the response information in the current consumption partition.

[0124] like Figure 7 As shown, the method comprises the following steps:

[0125] S701: The central scheduling service receives the response information sent by the large model service.

[0126] S702: Based on the pre-acquired network load information, the central scheduling service writes the response information into the Kafka service.

[0127] S703: For each Redis cache, the Redis cache determines whether the partition of the Redis cache consumption data has changed according to the current consumption partition and the consumption partition synchronized with the central scheduling service.

[0128] The specific implementation of step S701 to step S703 is the same as that in the above-mentioned embodiment, and will not be repeated here.

[0129] S704: If it is determined that the partition of the Redis cache consumption data has not changed, based on the offset synchronized by the central scheduling service, continue to consume the response information in the current consumption partition and perform local caching.

[0130] For each Redis cache, after receiving the metadata synchronized by the central scheduling service, the Redis cache compares the consumption partition in the metadata with the current consumption partition. If they are not the same partition, the consumption of the response information in the current consumption partition is stopped, and the response information in the consumption partition in the metadata is consumed again and cached locally.

[0131] The large model processing method based on distributed cache provided in this embodiment, the central scheduling service receives the response information sent by the large model service, and based on the pre-acquired network load information, the central scheduling service writes the response information to the kafka service. For each Redis cache, the Redis cache determines whether the partition of the Redis cache consumption data has changed based on the current consumption partition and the consumption partition synchronized by the central scheduling service. If it is determined that the partition of the Redis cache consumption data has not changed, then based on the offset synchronized by the central scheduling service, continue to consume the response information in the current consumption partition and cache it locally. Through the above method, the data consumption process is optimized, data consistency is enhanced, and the stability and reliability of the system are improved. At the same time, the response information is cached in Redis, which can significantly reduce the number of visits to kafka or large model services, reduce system latency, and improve user response speed.

[0132] Figure 8 This is a flow chart of a large model processing method based on distributed cache provided in the seventh embodiment of the present disclosure, refer to Figure 8 The large model processing method based on distributed cache shown is different from the aforementioned embodiment in that, in this embodiment, based on the metadata synchronized by the central scheduling service, the response information in the Kafka service is consumed, and it can also be further refined as follows: for each Redis cache, based on the consumption progress of the Redis cache itself, if the consumption progress is less than the offset synchronized by the central scheduling service, consumption continues; if the consumption progress is greater than or equal to the offset synchronized by the central scheduling service, consumption is stopped.

[0133] like Figure 8 As shown, the method comprises the following steps:

[0134] S801: The central scheduling service receives the response information sent by the large model service.

[0135] S802: Based on the pre-acquired network load information, the central scheduling service writes the response information into the Kafka service.

[0136] The specific implementation of step S801 and step S802 is the same as that in the above-mentioned embodiment, and will not be repeated here.

[0137] S803: For each Redis cache, based on the consumption progress of the Redis cache itself, if the consumption progress is less than the offset synchronized by the central scheduling service, consumption continues.

[0138] S804: If the consumption progress is greater than or equal to the offset synchronized by the central scheduling service, consumption is stopped.

[0139] The central scheduling service monitors the offset of the Kafka service, especially the latest offset consumed by each partition. And synchronize this offset information to the consumers of the Redis cache, that is, each Redis cache. Each Redis cache maintains a local or remote offset record, which indicates the offset of the currently consumed Kafka message. And periodically check the offset synchronized by the central scheduling service. Decide whether to continue or stop consumption based on the comparison result. If the local offset is less than the synchronized offset, it means that there are messages that have not been consumed, so continue to consume. If the local offset is greater than or equal to the synchronized offset, it means that the latest position has been consumed, and you can pause consumption or perform other operations (such as waiting for new messages).

[0140] The large model processing method based on distributed cache provided in this embodiment, the central scheduling service receives the response information sent by the large model service, and based on the pre-acquired network load information, the central scheduling service writes the response information to the kafka service, and for each Redis cache, based on the consumption progress of the Redis cache itself, if the consumption progress is less than the offset synchronized by the central scheduling service, then continue to consume, if the consumption progress is greater than or equal to the offset synchronized by the central scheduling service, then stop consuming. Through the above method, repeated consumption of data is avoided, resource utilization and data processing efficiency are improved, and high-concurrency data processing and caching are also achieved.

[0141] Fig. 9 This is a flow chart of a large model processing method based on distributed cache provided in the eighth embodiment of the present disclosure, with reference to Fig. 9 The large model processing method based on distributed cache shown is different from the aforementioned embodiment in that, in this embodiment, the front-end service can also respond to at least one access demand and query whether there is response information corresponding to at least one access demand in each Redis cache. For each access demand, if there is response information corresponding to the access demand in any Redis cache, the response information corresponding to the access demand will be fed back to the user corresponding to the access demand through the front-end service.

[0142] like Fig. 9 As shown, the method comprises the following steps:

[0143] S901: In response to at least one access request, the front-end service queries each Redis cache to see whether there is response information corresponding to at least one access request.

[0144] S902: For each access requirement, if there is response information corresponding to the access requirement in any Redis cache, the response information corresponding to the access requirement is fed back to the user corresponding to the access requirement through the front-end service.

[0145] In order to improve user response speed and user experience, after receiving an access request, the front-end service queries the Redis cache for response information corresponding to the access request. If the corresponding response information exists, there is no need to perform large model inference, and the response information in the cache is directly fed back to the user corresponding to the access request through the front-end service.

[0146] In the large model processing method based on distributed cache provided in this embodiment, the front-end service responds to at least one access demand and queries each Redis cache whether there is response information corresponding to at least one access demand. For each access demand, if there is response information corresponding to the access demand in any Redis cache, the response information corresponding to the access demand is fed back to the user corresponding to the access demand through the front-end service. Through the interaction between the front-end service and the Redis cache, efficient response and fast feedback to the access demand are achieved, and the processing efficiency of high concurrent access is improved.

[0147] Fig.10 This is a flow chart of a large model processing method based on distributed cache provided in the ninth embodiment of the present disclosure, refer to Fig.10 The big model processing method based on distributed cache shown is different from the aforementioned embodiment in that, in this embodiment, for each access demand, if there is no response information corresponding to the access demand in each Redis cache, the access demand is sent to the big model service through the central scheduling service. The big model service infers the access demand, obtains the response information corresponding to the access demand, and feeds back to the user corresponding to the access demand through the front-end service.

[0148] like Fig.10 As shown, the method comprises the following steps:

[0149] S1001: In response to at least one access request, the front-end service queries each Redis cache to see whether there is response information corresponding to at least one access request.

[0150] The specific implementation of step S1001 is the same as that in the aforementioned embodiment, and will not be repeated here.

[0151] S1002: For each access request, if there is no response information corresponding to the access request in each Redis cache, the access request is sent to the large model service through the central scheduling service.

[0152] S1003: The large model service infers the access demand, obtains response information corresponding to the access demand, and feeds back the response information to the user corresponding to the access demand through the front-end service.

[0153] If there is no response information corresponding to the access request in any Redis cache, the public service needs to infer the response information and feed it back to the user corresponding to the access request through the front-end service.

[0154] It should be noted that the response information obtained after the big model service completes the reasoning can be cached locally through the big model processing method in distributed cache of the aforementioned embodiments.

[0155] In the large model processing method based on distributed cache provided in this embodiment, the front-end service responds to at least one access demand and queries each Redis cache whether there is response information corresponding to at least one access demand. For each access demand, if there is no response information corresponding to the access demand in each Redis cache, the access demand is sent to the large model service through the central scheduling service. The large model service infers the access demand, obtains the response information corresponding to the access demand, and feeds back to the user corresponding to the access demand through the front-end service. By reasonably utilizing the Redis cache and the large model service, efficient processing and accurate response to access demands are achieved, and the overall performance and user experience of the system are improved.

[0156] Fig.11 A block diagram of a large model processing device based on distributed cache provided in an embodiment of the present disclosure, referring to Fig.11 The large model processing device 1100 based on distributed cache shown in the figure includes the following units:

[0157] Receiving unit 1101, used for the central scheduling service to receive the response information sent by the large model service, where the response information is feedback information corresponding to the user's access demand;

[0158] The writing unit 1102 is used for the central scheduling service to write the response information into the Kafka service based on the pre-acquired network load information;

[0159] The consumption unit 1103 is used to consume the response information in the kafka service and perform local caching for each Redis cache in at least one Redis cache.

[0160] In one possible design, the consumer unit 1103 includes:

[0161] The consumption module is used to consume the response information in the Kafka service and cache it locally based on the metadata synchronized by the central scheduling service. The metadata includes the consumption partition and offset of the Redis cache consumption data.

[0162] Optionally, the large model processing device 1100 based on distributed cache further includes:

[0163] A first updating unit 1104 is configured to update, for each Redis cache, the metadata corresponding to the Redis cache based on a mapping table by the central scheduling service, where the mapping table includes a mapping relationship between each Redis cache and a consumer instance in the kafka service;

[0164] The synchronization unit 1105 is used for the central scheduling service to synchronize the metadata corresponding to the Redis cache to the Redis cache through the Raft protocol.

[0165] In a possible implementation, the large model processing device 1100 based on distributed cache further includes:

[0166] The second updating unit 1106 is used for the central scheduling service to update the mapping table based on the status information of each Redis cache, the network load information and the load information of the Kafka service.

[0167] In one possible implementation, the consumption module includes:

[0168] The determination submodule is used for each Redis cache to determine whether the partition of the Redis cache consumption data has changed based on the current consumption partition and the consumption partition synchronized with the central scheduling service.

[0169] The first consumption submodule is used to consume the response information in the consumption partition synchronized by the central scheduling service based on the offset synchronized by the central scheduling service if it is determined that the partition of the Redis cache consumption data has changed.

[0170] In a possible implementation, the consumption module further includes:

[0171] The second consumption submodule is used to continue consuming the response information in the current consumption partition based on the offset synchronized by the central scheduling service if it is determined that the partition of the Redis cache consumption data has not changed.

[0172] Optional, consumption module, also includes:

[0173] The third consumption submodule is used for each Redis cache. Based on the consumption progress of the Redis cache itself, if the consumption progress is less than the offset synchronized by the central scheduling service, consumption will continue;

[0174] The fourth consumption submodule is used to stop consumption if the consumption progress is greater than or equal to the offset synchronized by the central scheduling service.

[0175] Optionally, the large model processing device 1100 based on distributed cache further includes:

[0176] A query unit 1107 is used for the front-end service to query each Redis cache whether there is response information corresponding to at least one access requirement in response to at least one access requirement;

[0177] The first feedback unit 1108 is used to feed back the response information corresponding to each access requirement to the user corresponding to the access requirement through the front-end service if there is response information corresponding to the access requirement in any Redis cache.

[0178] Optionally, the large model processing device 1100 based on distributed cache further includes:

[0179] The sending unit 1109 is used for sending the access request to the large model service through the central scheduling service for each access request if there is no response information corresponding to the access request in each Redis cache;

[0180] The second feedback unit 1110 is used for the large model service to infer the access demand, obtain the response information corresponding to the access demand, and feed back the response information to the user corresponding to the access demand through the front-end service.

[0181] According to an embodiment of the present disclosure, the present disclosure also provides an electronic device, a readable storage medium and a computer program product.

[0182] According to an embodiment of the present disclosure, the present disclosure also provides a computer program product, which includes: a computer program, the computer program is stored in a readable storage medium, at least one processor of an electronic device can read the computer program from the readable storage medium, and at least one processor executes the computer program so that the electronic device executes the solution provided by any of the above embodiments.

[0183] Fig.12 A schematic block diagram of an electronic device implementing an embodiment of the present disclosure is shown in FIG. Fig.12 As shown, Fig.12 A schematic block diagram of an example electronic device 1200 that can be used to implement an embodiment of the present disclosure is shown. The electronic device is intended to represent various forms of digital computers, such as laptop computers, desktop computers, workstations, personal digital assistants, servers, blade servers, mainframe computers, and other suitable computers. The electronic device can also represent various forms of mobile devices, such as personal digital processing, cellular phones, smart phones, wearable devices, and other similar computing devices. The components shown herein, their connections and relationships, and their functions are merely examples and are not intended to limit the implementation of the present disclosure described and / or required herein.

[0184] like Fig.12As shown, the electronic device 1200 includes a computing unit 1201, which can perform various appropriate actions and processes according to a computer program stored in a read-only memory (ROM) 1202 or a computer program loaded from a storage unit 1208 into a random access memory (RAM) 1203. In the RAM 1203, various programs and data required for the operation of the electronic device 1200 can also be stored. The computing unit 1201, the ROM 1202, and the RAM 1203 are connected to each other via a bus 1204. An input / output (I / O) interface 1205 is also connected to the bus 1204.

[0185] Multiple components in the electronic device 1200 are connected to the I / O interface 1205, including: an input unit 1206, such as a keyboard, a mouse, etc.; an output unit 1207, such as various types of displays, speakers, etc.; a storage unit 1208, such as a disk, an optical disk, etc.; and a communication unit 1209, such as a network card, a modem, a wireless communication transceiver, etc. The communication unit 1209 allows the device 1200 to exchange information / data with other devices through a computer network such as the Internet and / or various telecommunication networks.

[0186] The computing unit 1201 may be a variety of general and / or special processing components with processing and computing capabilities. Some examples of the computing unit 1201 include, but are not limited to, a central processing unit (CPU), a graphics processing unit (GPU), various dedicated artificial intelligence (AI) computing chips, various computing units running machine learning model algorithms, digital signal processors (DSPs), and any appropriate processors, controllers, microcontrollers, etc. The computing unit 1201 performs the various methods and processes described above, such as a large model processing method based on a distributed cache. For example, in some embodiments, the large model processing method based on a distributed cache may be implemented as a computer software program, which is tangibly included in a machine-readable medium, such as a storage unit 1208. In some embodiments, part or all of the computer program may be loaded and / or installed on the electronic device 1200 via the ROM 1202 and / or the communication unit 1209. When the computer program is loaded into the RAM 1203 and executed by the computing unit 1201, one or more steps of the large model processing method based on a distributed cache described above may be executed. Alternatively, in other embodiments, the computing unit 1201 may be configured to execute the large model processing method based on distributed cache in any other appropriate manner (for example, by means of firmware).

[0187] Various implementations of the systems and techniques described above herein can be implemented in digital electronic circuit systems, integrated circuit systems, field programmable gate arrays (FPGAs), application specific integrated circuits (ASICs), application specific standard products (ASSPs), systems on chips (SOCs), complex programmable logic devices (CPLDs), computer hardware, firmware, software, and / or combinations thereof. These various implementations can include: being implemented in one or more computer programs that can be executed and / or interpreted on a programmable system including at least one programmable processor, which can be a special purpose or general purpose programmable processor that can receive data and instructions from a storage system, at least one input device, and at least one output device, and transmit data and instructions to the storage system, the at least one input device, and the at least one output device.

[0188] The program code for implementing the method of the present disclosure may be written in any combination of one or more programming languages. These program codes may be provided to a processor or controller of a general-purpose computer, a special-purpose computer, or other programmable data processing device, so that the program code, when executed by the processor or controller, enables the functions / operations specified in the flow chart and / or block diagram to be implemented. The program code may be executed entirely on the machine, partially on the machine, partially on the machine and partially on a remote machine as a stand-alone software package, or entirely on a remote machine or server.

[0189] In the context of the present disclosure, a machine-readable medium may be a tangible medium that may contain or store a program for use by or in conjunction with an instruction execution system, device, or equipment. A machine-readable medium may be a machine-readable signal medium or a machine-readable storage medium. A machine-readable medium may include, but is not limited to, an electronic, magnetic, optical, electromagnetic, infrared, or semiconductor system, device, or equipment, or any suitable combination of the foregoing. A more specific example of a machine-readable storage medium may include an electrical connection based on one or more lines, a portable computer disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the foregoing.

[0190] To provide interaction with a user, the systems and techniques described herein can be implemented on a computer having: a display device (e.g., a CRT (cathode ray tube) or LCD (liquid crystal display) monitor) for displaying information to the user; and a keyboard and pointing device (e.g., a mouse or trackball) through which the user can provide input to the computer. Other types of devices can also be used to provide interaction with the user; for example, the feedback provided to the user can be any form of sensory feedback (e.g., visual feedback, auditory feedback, or tactile feedback); and input from the user can be received in any form (including acoustic input, voice input, or tactile input).

[0191] The systems and techniques described herein may be implemented in a computing system that includes back-end components (e.g., as a data server), or a computing system that includes middleware components (e.g., an application server), or a computing system that includes front-end components (e.g., a user computer with a graphical user interface or a web browser through which a user can interact with implementations of the systems and techniques described herein), or a computing system that includes any combination of such back-end components, middleware components, or front-end components. The components of the system may be interconnected by any form or medium of digital data communication (e.g., a communication network). Examples of communication networks include: a local area network (LAN), a wide area network (WAN), and the Internet.

[0192] A computer system may include a client and a server. The client and the server are generally remote from each other and usually interact through a communication network. The relationship between the client and the server is generated by computer programs running on the corresponding computers and having a client-server relationship with each other. The server may be a cloud server, also known as a cloud computing server or a cloud host, which is a host product in the cloud computing service system to solve the defects of difficult management and weak business scalability in traditional physical hosts and VPS services ("ViRtual PRivate SeRveR", or "VPS" for short). The server may also be a server of a distributed system, or a server combined with a blockchain.

[0193] It should be understood that the various forms of processes shown above can be used to reorder, add or delete steps. For example, the steps recorded in this disclosure can be executed in parallel, sequentially or in different orders, as long as the desired results of the technical solutions disclosed in this disclosure can be achieved, and this document does not limit this.

[0194] The above specific implementations do not constitute a limitation on the protection scope of the present disclosure. It should be understood by those skilled in the art that various modifications, combinations, sub-combinations and substitutions can be made according to design requirements and other factors. Any modification, equivalent substitution and improvement made within the spirit and principle of the present disclosure shall be included in the protection scope of the present disclosure.

Claims

1. A large model processing method based on distributed cache, comprising: The central scheduling service receives the response information sent by the large model service, wherein the response information is feedback information corresponding to the user's access demand; Based on the pre-acquired network load information, the central scheduling service writes the response information into the Kafka service; For each Redis cache in at least one Redis cache, the response information in the Kafka service is consumed and cached locally.

2. The method according to claim 1, wherein: The consuming the response information in the kafka service and caching it locally includes: Based on the metadata synchronized by the central scheduling service, the response information in the kafka service is consumed and cached locally, wherein the metadata includes the consumption partition and offset of the Redis cache consumption data.

3. The method according to claim 2, further comprising: For each Redis cache, the central scheduling service updates the metadata corresponding to the Redis cache based on a mapping table, wherein the mapping table includes a mapping relationship between each Redis cache and a consumer instance in the kafka service; The central scheduling service synchronizes the metadata corresponding to the Redis cache to the Redis cache through the Raft protocol.

4. The method according to claim 3, further comprising: The central scheduling service updates the mapping table based on the status information of each Redis cache, the network load information and the load information of the Kafka service.

5. The method according to claim 2, wherein: The consuming the response information in the Kafka service based on the metadata synchronized by the central scheduling service includes: For each Redis cache, the Redis cache determines whether the partition of the Redis cache consumption data has changed according to the current consumption partition and the consumption partition synchronized with the central scheduling service; If it is determined that the partition of the Redis cache consumption data has changed, based on the offset synchronized by the central scheduling service, the response information in the consumption partition synchronized by the central scheduling service is consumed.

6. The method according to claim 5, further comprising: If it is determined that the partition of the Redis cache consumption data has not changed, then based on the offset synchronized by the central scheduling service, continue to consume the response information in the currently consumed consumption partition.

7. The method according to claim 5 or 6, further comprising: For each Redis cache, based on the consumption progress of the Redis cache itself, if the consumption progress is less than the offset synchronized by the central scheduling service, consumption continues; If the consumption progress is greater than or equal to the offset of the central scheduling service synchronization, consumption is stopped.

8. The method according to claim 1, further comprising: The front-end service responds to at least one access request by querying each Redis cache whether there is response information corresponding to the at least one access request; For each access requirement, if there is response information corresponding to the access requirement in any Redis cache, the response information corresponding to the access requirement is fed back to the user corresponding to the access requirement through the front-end service.

9. The method according to claim 8, further comprising: For each access requirement, if there is no response information corresponding to the access requirement in each Redis cache, the access requirement is sent to the large model service through the central scheduling service; The large model service infers the access demand, obtains response information corresponding to the access demand, and feeds back the response information to the user corresponding to the access demand through the front-end service.

10. A large model processing device based on distributed cache, comprising: A receiving unit, used for the central scheduling service to receive the response information sent by the large model service, wherein the response information is feedback information corresponding to the user's access demand; A writing unit, configured for the central scheduling service to write the response information into the Kafka service based on the pre-acquired network load information; The consumption unit is used to consume the response information in the kafka service and perform local caching for each Redis cache in at least one Redis cache.

11. The device according to claim 10, wherein: The consumer unit comprises: The consumption module is used to consume the response information in the Kafka service and cache it locally based on the metadata synchronized by the central scheduling service. The metadata includes the consumption partition and offset of the Redis cache consumption data.

12. The apparatus according to claim 11, further comprising: A first updating unit, configured to update, for each Redis cache, metadata corresponding to the Redis cache based on a mapping table by the central scheduling service, wherein the mapping table includes a mapping relationship between each Redis cache and a consumer instance in the kafka service; The synchronization unit is used for the central scheduling service to synchronize the metadata corresponding to the Redis cache to the Redis cache through the Raft protocol.

13. The apparatus according to claim 12, further comprising: The second updating unit is used for the central scheduling service to update the mapping table based on the status information of each Redis cache, the network load information and the load information of the Kafka service.

14. The device according to claim 11, wherein: The consumption module comprises: A determination submodule is used to determine, for each Redis cache, whether the partition of the Redis cache consumption data has changed according to the consumption partition currently consumed and the consumption partition synchronized with the central scheduling service; The first consumption submodule is used to consume the response information in the consumption partition synchronized by the central scheduling service based on the offset synchronized by the central scheduling service if it is determined that the partition of the Redis cache consumption data has changed.

15. The device according to claim 14, wherein: The consumption module further includes: The second consumption submodule is used to continue consuming the response information in the current consumption partition based on the offset synchronized by the central scheduling service if it is determined that the partition of the Redis cache consumption data has not changed.

16. The device according to claim 14 or 15, wherein: The consumption module further includes: The third consumption submodule is used for each Redis cache, based on the consumption progress of the Redis cache itself, if the consumption progress is less than the offset synchronized by the central scheduling service, then continue consumption; The fourth consumption submodule is used to stop consumption if the consumption progress is greater than or equal to the offset of the central scheduling service synchronization.

17. The apparatus according to claim 10, further comprising: A query unit, configured for the front-end service to query, in response to at least one access demand, whether there is response information corresponding to the at least one access demand in each Redis cache; The first feedback unit is used to feed back the response information corresponding to each access requirement to the user corresponding to the access requirement through the front-end service if the response information corresponding to the access requirement exists in any Redis cache.

18. The apparatus according to claim 17, further comprising: A sending unit, for each access demand, if there is no response information corresponding to the access demand in each Redis cache, sending the access demand to the large model service through the central scheduling service; The second feedback unit is used for the large model service to infer the access demand, obtain response information corresponding to the access demand, and feed back the response information to the user corresponding to the access demand through the front-end service.

19. An electronic device comprising: at least one processor; as well as a memory communicatively connected to the at least one processor; wherein, The memory stores instructions that can be executed by the at least one processor, and the instructions are executed by the at least one processor to enable the at least one processor to perform the method according to any one of claims 1 to 9.

20. A non-transitory computer-readable storage medium storing computer instructions, wherein: The computer instructions are used to cause the computer to execute the method according to any one of claims 1-9.

21. A computer program product, comprising a computer program, which, when executed by a processor, implements the steps of the method according to any one of claims 1 to 9.

Images