Message processing method and device, equipment, storage medium and program product

Abstract

The invention provides a message processing method and device, equipment, a storage medium and a program product. The method comprises the steps of obtaining a target message from a message cache space in response to a message processing request sent by a first client; wherein the message cache space comprises at least one message storage space, and each message storage space comprises an environment identifier of a message operation environment; the message processing request carries a first environment identifier of a target message operation environment; the target message is stored in a target message partition of a target message storage space corresponding to the first environment identifier; and sending the target message to the first client. According to the message processing method and device, the target message can be effectively processed on the basis of the message storage space under the condition that the external resource cost does not need to be increased.

Description

Technical Field

[0001] The present application relates to the field of data processing technology, and in particular to a message processing method, device, equipment, storage medium and program product. Background Art

[0002] With the development of computer technology, various application systems will continuously produce various data like data factories. In order to ensure the orderly transmission of massive data, related technologies are based on the high throughput and low latency of Kafka to process large amounts of data in real time or offline to meet various demand scenarios. In the Kafka message processing system, producers can send messages in units of partitions; consumers can consume messages in units of partitions. For multi-environment deployment scenarios, different Kafka clusters or different message topics will be created according to different message operating environments to achieve message consumption isolation. In addition, if there is cross-consumption of messages between multiple message operating environments, the application processing will be restarted after the code is updated or the configuration is updated. Summary of the invention

[0003] The embodiments of the present application provide a message processing method, apparatus, device, storage medium and program product, which can effectively process the target message based on the message storage space without increasing the external resource cost.

[0004] The technical solution of the embodiment of the present application is implemented as follows:

[0005] An embodiment of the present application provides a message processing method, which includes: in response to a message processing request sent by a first client, obtaining a target message from a message cache space; wherein the message cache space includes at least one message storage space, each message storage space includes an environment identifier of a message running environment; the message processing request carries a first environment identifier of the target message running environment; the target message is stored in a target message partition of the target message storage space corresponding to the first environment identifier; and the target message is sent to the first client.

[0006] An embodiment of the present application provides a message processing device, including: a target message acquisition module, used to obtain a target message from a message cache space in response to a message processing request sent by a first client; wherein the message cache space includes at least one message storage space, each message storage space includes an environment identifier of a message running environment; the message processing request carries a first environment identifier of the target message running environment; the target message is stored in a target message partition of the target message storage space corresponding to the first environment identifier; and a message sending module, used to send the target message to the first client.

[0007] An embodiment of the present application provides an electronic device, comprising: a memory for storing computer-executable instructions; and a processor for implementing the message processing method provided in the embodiment of the present application when executing the computer-executable instructions stored in the memory.

[0008] An embodiment of the present application provides a computer-readable storage medium storing computer-executable instructions for implementing the message processing method provided in the embodiment of the present application when executed by a processor.

[0009] An embodiment of the present application provides a computer program product, which includes executable instructions, and the executable instructions are stored in a computer-readable storage medium; wherein, when a processor of an electronic device reads the executable instructions from the computer-readable storage medium and executes the executable instructions, the message processing method provided in the embodiment of the present application is implemented.

[0010] The embodiments of the present application have the following beneficial effects:

[0011] The message cache space includes at least one message storage space, each message storage space includes an environment identifier of a message running environment; the target message is stored in the target message partition of the target message storage space corresponding to the first environment identifier. In this way, on the one hand, the message storage space in the message cache space corresponds to the message running environment one by one. In the scenario of multi-environment deployment, messages in different message running environments are respectively stored in message partitions of different message storage spaces. In other words, the message partitions are divided and isolated by the message storage space, and each message storage space corresponds to a message running environment, thereby avoiding the intersection of messages in multiple message running environments. On the other hand, based on the first environment identifier of the target message running environment carried in the message processing request, the target message storage space corresponding to the target message in the cache module can be determined, and the target message can be directly obtained from the target message partition of the target message storage space, so that there is no need to traverse all message partitions, which can greatly improve the reading efficiency of the target message. BRIEF DESCRIPTION OF THE DRAWINGS

[0012] Figure 1 It is a structural diagram of a message processing system provided in an embodiment of the present application;

[0013] Figure 2 This is an optional flowchart of the message processing method provided in the embodiment of the present application;

[0014] Figure 3 This is another optional flowchart of the message processing method provided in the embodiment of the present application;

[0015] Figure 4 It is a schematic diagram of the system architecture of the message processing system provided in an embodiment of the present application;

[0016] Figure 5 This is a schematic diagram of the implementation of the message storage process in the Kafka cluster provided in the embodiment of the present application;

[0017] Figure 6 It is a schematic diagram of implementing attribute information of adding message storage space provided by an embodiment of the present application;

[0018] Figure 7 It is a structural diagram of a message processing device provided in an embodiment of the present application;

[0019] Figure 8 It is a schematic diagram of the composition structure of the electronic device provided in the embodiment of the present application. DETAILED DESCRIPTION

[0020] In order to make the purpose, technical solutions and advantages of the present application clearer, the present application will be further described in detail below in conjunction with the accompanying drawings. The described embodiments should not be regarded as limiting the present application. All other embodiments obtained by ordinary technicians in the field without making creative work are within the scope of protection of this application.

[0021] In the following description, reference is made to “some embodiments”, which describe a subset of all possible embodiments, but it will be understood that “some embodiments” may be the same subset or different subsets of all possible embodiments and may be combined with each other without conflict.

[0022] Unless otherwise defined, all technical and scientific terms used in the embodiments of the present application have the same meanings as those commonly understood by those skilled in the art. The terms used in the embodiments of the present application are only for the purpose of describing the embodiments of the present application and are not intended to limit the present application.

[0023] The relevant data collection and processing in the embodiments of this application should be strictly in accordance with the requirements of relevant national laws and regulations when applied in examples, obtain the informed consent or separate consent of the personal information subject, and carry out subsequent data use and processing within the scope of authorization of laws and regulations and the personal information subject.

[0024] Before further describing the embodiments of the present application in detail, the nouns and terms involved in the embodiments of the present application are explained. The nouns and terms involved in the embodiments of the present application are subject to the following interpretations.

[0025] 1) Kafka: It is a high-throughput distributed publish-subscribe messaging system. Kafka can process all action stream data of consumers on the website. Action stream data includes but is not limited to real-time data streams such as user access logs, search queries, and social interactions.

[0026] 2) Topic: A logical classification of messages. A topic is an organizational unit of messages in Kafka. It is a collection of messages with the same data structure. A topic can contain one or more partitions and can be subscribed to by consumers in one or more consumer groups to consume messages. Messages in a topic are broadcast, that is, messages published to a topic are sent to all subscribers.

[0027] 3) Message partition: It is a physical concept and a logically separated part of a message topic. Each message partition is an ordered sequence of messages in a message topic. Each message partition can store data on different Kafka servers (Brokers) in the Kafka cluster.

[0028] 4) Unitization: The deployment unit of the unitized application service product layer refers to a self-contained set that can complete all business operations. This set can independently complete a specific business operation and is an independent business unit. For example, one or more message partitions are unitized to obtain a message storage space, through which business operations in a specific message operating environment can be independently completed, thereby completing the consumption of messages.

[0029] In the related art, for message processing in multiple message operating environments, different kafka clusters or different message topics are created according to different message operating environments to achieve message consumption isolation (i.e. effective processing). For example, in some business scenarios, kafka is used to decouple multiple businesses, thereby introducing kafka for business processing, but the business traffic of such businesses themselves is relatively small, and they also have multiple different message operating environments. In this case, the related art may deploy multiple sets of kafka clusters or multiple groups of different topics for such businesses, but whether adding kafka clusters or adding topics, it will cause resource loss. Moreover, in real business scenarios, the business traffic of some message operating environments is not particularly large, which is easy to cause an imbalance between resource costs and actual benefits. In addition, if there is cross-consumption of messages between multiple message operating environments, the related art will consider developing and updating the code or restarting the application processing after updating the configuration. At this time, it is unacceptable in terms of business continuity or application development and transformation costs.

[0030] Based on at least one problem existing in the related art, the embodiment of the present application divides the message partitions of the message topic in the Kafka cluster into units, and divides the multiple message partitions under a message topic into different message storage spaces according to the actual business needs (i.e., the message operating environment), and each message operating environment corresponds to a message storage space. The flexibility of using Kafka is increased, so that when the client sends a message, the message storage space corresponding to the message can be determined according to the message operating environment of the message, and the message can be sent and stored. Without increasing the cost of external resources, the purpose of consumer isolation can be achieved by obtaining the message from the message partition of the message storage space corresponding to the message and processing the message, which greatly reduces the cost of resource use. In addition, the difficulty of using Kafka in a multi-environment deployment architecture is simplified, and the controllability of the client over message isolation transmission is enhanced.

[0031] See also Figure 1 , Figure 1 It is a structural diagram of the message processing system 100 architecture provided in an embodiment of the present application. To support a message processing application, a terminal 400 is connected to a server 200 via a network 300. The network 300 may be a wide area network or a local area network, or a combination of the two.

[0032] The terminal 400 is used to send a message processing request to the server 200 through the first client, and the server 200 is used to respond to the message processing request sent by the first client and obtain the target message from the message cache space; wherein the message cache space includes at least one message storage space, and each message storage space includes an environment identifier of a message running environment; the message processing request carries the first environment identifier of the target message running environment; the target message is stored in the target message partition of the target message storage space corresponding to the first environment identifier; the target message is sent to the first client to enable the first client to process the target message in the target message running environment and obtain the message processing result of the target message.

[0033] In some embodiments, the server 200 may be an independent physical server, or a server cluster or distributed system composed of multiple physical servers, or a cloud server that provides basic cloud computing services such as cloud services, cloud databases, cloud computing, cloud functions, cloud storage, network services, cloud communications, middleware services, domain name services, security services, content delivery networks (CDN), and big data and artificial intelligence platforms. The terminal 400 may be a smart phone, a tablet computer, a laptop computer, a desktop computer, a smart speaker, a smart watch, a car terminal, etc., but is not limited thereto. The terminal 400 and the server 200 may be directly or indirectly connected via wired or wireless communication, which is not limited in the embodiments of the present application.

[0034] The message processing method provided in each embodiment of the present application can be executed by an electronic device, wherein the electronic device can be a server or a terminal, that is, the message processing method of each embodiment of the present application can be executed by a server, can be executed by a terminal, or can be executed through interaction between a server and a terminal.

[0035] See also Figure 2 , Figure 2 is an optional flow chart of the message processing method provided in the embodiment of the present application, which will be combined with Figure 2 The steps shown are described in detail, and the execution subject of the message processing method is taken as an example to describe the method as a server. The method includes the following steps S101 to S102:

[0036] Step S101, in response to a message processing request sent by a first client, obtaining a target message from a message cache space.

[0037] In an embodiment of the present application, the first client refers to a client capable of message processing, for example, the first client may be a message consumer in a kafka message queue system. The message cache space is a message storage mechanism responsible for receiving and saving messages from a message producer (for example, a message sender in a kafka message queue system) until the message can be read and processed by the consumer (i.e., the first client). For example, the message cache space may be a cache module in a kafka server in a kafka message queue system. The message cache space includes at least one message storage space, each message storage space including an environment identifier of a message runtime environment, that is, the message storage space is a basic data unit in the message cache space, and each message storage space corresponds to a message runtime environment. The message runtime environment refers to the environment on which the message depends when the message is delivered and processed in the system, for example, the message runtime environment may be a production environment, a test environment, a gray environment, etc. The name of each message storage space may be an environment identifier of a message runtime environment, for example, the environment identifier of the production environment is unit1, the environment identifier of the test environment is unit2, and the environment identifier of the gray environment message is unit3.

[0038] The message processing request carries the first environment identifier of the target message running environment. The target message running environment refers to the message running environment corresponding to the target message in the message running environment. The first environment identifier refers to the environment identifier of the target message running environment. The target message refers to the message that needs to be transmitted and processed at the current moment. The target message is stored in the target message partition of the target message storage space corresponding to the first environment identifier. For example, when the message running environment of the target message is the production environment, the target message is stored in the target message partition of the target message storage space corresponding to unit1.

[0039] When a message processing request is received from the first client, the target message storage space corresponding to the first environment identifier of the target message running environment carried in the message processing request can be determined, and the target message can be directly obtained from the target message partition of the target message storage space in the message cache space.

[0040] Here, when storing messages in the message cache space, the messages are stored in different message storage spaces according to the different message running environments to which the messages belong, and the name of each message storage space corresponds to the environment identifier of the message running environment, so that messages in different message running environments are isolated from each other, avoiding the intersection of messages in multiple message running environments in the multi-environment deployment scenario. In order to achieve effective processing of messages in different message running environments when processing the messages later.

[0041] Step S102: Send the target message to the first client.

[0042] In the embodiment of the present application, the first client is used to process the target message in the target message operating environment to obtain the message processing result of the target message. After obtaining the target message from the message cache space, the target message will be sent to the first client, and the target message will be processed by the first client in the target message operating environment. The message processing process may involve steps such as parsing the message content of the target message, executing corresponding business logic, and generating message processing results.

[0043] For example, suppose that whenever a new order is generated on an e-commerce platform, the system will use the order information as the target message and determine the message running environment of the order information; use the message storage space in the message cache space corresponding to the environment identifier of the message running environment as the target message storage space, and send the order information to the target message partition of the target message storage space for message storage. When the first client (for example, an order processing program) needs to process the order information (for example, generate logistics information, update inventory, etc.), the server receives the message processing request (i.e., order processing request) generated by the first client, obtains the order information from the message cache space, sends the order information to the first client, and the first client parses the order information to obtain the message processing result.

[0044] Here, by sending the target message to the first client, the target message received by the first client is a message under the target message operating environment, and there is no intersection of messages under multiple message operating environments, thereby enabling the first client to effectively process the target message under the target message operating environment.

[0045] The message processing method provided by the embodiment of the present application first obtains the target message from the message cache space in response to the message processing request sent by the first client; then, the target message is sent to the first client. Among them, the first client is used to process the target message in the target message running environment to obtain the message processing result of the target message. The message processing request carries the first environment identifier of the target message running environment; the message cache space includes at least one message storage space, and each message storage space includes an environment identifier of a message running environment; the target message is stored in the target message partition of the target message storage space corresponding to the first environment identifier. In this way, the message storage space in the message cache space corresponds to the message running environment one by one. In the scenario of multi-environment deployment, messages in different message running environments are respectively stored in message partitions of different message storage spaces, avoiding the intersection of messages in multiple message running environments. Based on the first environment identifier of the target message running environment carried in the message processing request, the target message storage space corresponding to the target message in the cache module can be determined, and the target message can be directly obtained from the target message partition of the target message storage space. In addition, the target message is sent to a first client for processing the target message in the target message operating environment, that is, the target message received by the first client is a message in the target message operating environment, and the target message can be effectively processed by the first client in the target message operating environment.

[0046] The message processing method in the embodiment of the present application will be described below in conjunction with the interaction process between the server and the terminal (i.e., the above-mentioned first client) in the message processing system. It should be noted that the message processing method here is a message processing method implemented by interaction between the server and the terminal, which is substantially the same as the message processing method executed by the server in the above-mentioned embodiment. The only difference is that the embodiment of the present application also describes the actions performed by the terminal during the execution of the message processing method, and some steps can be executed by both the terminal and the server. Therefore, for the steps in this embodiment that are the same in content as those in the above-mentioned embodiment but have different execution subjects, this embodiment is only an exemplary description. In the process of implementation, it can be executed by any execution subject, and the embodiment of the present application does not limit this.

[0047] See also Figure 3 , Figure 3 is another optional flow chart of the message processing method provided in the embodiment of the present application, such as Figure 3 As shown, the method includes the following steps S201 to S208:

[0048] Step S201: The terminal receives a message processing operation input by a user.

[0049] In the embodiment of the present application, the user can input the message processing operation for the target message in the client (i.e., the first client) of the message processing system. In the message processing system, a message processing function can be provided, and the user can input the message processing operation for the target message on the message processing function page. In some embodiments, when inputting the message processing operation for the target message, the target message category can be input at the same time, and the target message category and the first environment identifier of the target message running environment can also be input at the same time.

[0050] Step S202: The terminal generates a message processing request in response to the message processing operation.

[0051] In an embodiment of the present application, the data input by the user can be encapsulated into the message processing request. For example, the target message category input by the user can be encapsulated into the message processing request, and the target message category input by the user and the first environment identifier of the target message running environment can also be encapsulated into the message processing request.

[0052] Step S203: The terminal sends a message processing request to the server.

[0053] Step S204: the server determines a message sending strategy for the target message in response to the message processing request.

[0054] In an embodiment of the present application, before responding to a message processing request sent by the first client, it is first necessary to determine the cache structure of the message cache space; then, based on a preset message running environment, the message partitions in the cache structure are structurally adjusted to obtain an updated cache structure.

[0055] The cache structure of the message cache space includes N message partitions, where N is an integer greater than 1. For the cache structure of the message cache space, there are two cases in the embodiment of the present application: in one case, if the cache structure of the message cache space has not implemented the adjustment of the message partitions in the cache structure based on the message running environment before the current moment, the cache structure of the message cache space includes N message partitions located under the message topic. In another case, if the cache structure of the message cache space has implemented the adjustment of the message partitions in the cache structure based on the message running environment at the historical moment before the current moment, the cache structure of the message cache space includes a message storage space located under the message topic and corresponding to the message running environment at the historical moment, and a message partition located in the message storage space. Message topics and message partitions belong to two different levels of the message queue system. The message topic is a logical classification in the message queue system, which is used to classify messages. The message producer sends messages to the message topic according to the message category, and the message consumer (i.e., the first client) subscribes to messages from the message topic according to the message category. Messages of different message categories belong to different message topics. For example, order creation, payment, order delivery, order cancellation and other order-related messages are classified in the order management topic, and application error logs, access logs, performance logs and other log messages related to application operation are classified in the log management topic. Message partitions are physical divisions of message topics. Each message topic includes at least one message partition, and each message partition includes an ordered and immutable message sequence.

[0056] The preset message running environment refers to the environment that the preset message relies on when it is delivering and processing messages in the system. The type of message running environment can be set according to actual needs. For example, the preset message running environment can be a production environment, a test environment, a grayscale environment, etc. The updated cache structure includes a message storage space located under the message subject and corresponding to the preset message running environment, and a message partition located in the message storage space. Adjustment refers to changing the message partitions in the cache structure of the message cache space, unitizing the message partitions in the cache structure, and updating the cache structure of the message cache space. According to the above two situations, the adjustment process of the message partitions in the cache structure based on the preset message running environment can also be divided into three situations: Situation 1. When corresponding to the first situation above, the adjustment process of the message partitions in the cache structure is to add a layer of cache structure of the message storage space corresponding to the preset message running environment on the basis of the message subject and the message partitions under the message subject. For example, assuming that before the current moment, the message running environment of the message is not considered, then there is no message storage space corresponding to the message running environment in the cache structure of the message cache space. When at the current moment, the message running environment of the message is taken into consideration, and the preset message running environment includes environment A and environment B, the message partitions 1 to 6 in the cache structure are adjusted so that message partitions 1 to 3 are divided into the message storage space A corresponding to environment A, and message partitions 4 to 6 are divided into the message storage space B corresponding to environment B, that is, the message storage space A and the message storage space B corresponding to environment A and environment B are added.

[0057] Case 2: When it corresponds to the other case mentioned above, and there is no new message running environment compared to the message running environment at the historical moment, the message partitions in the cache structure are not adjusted, and the cache structure of the message cache space is not updated. For example, assuming that the message running environment at the historical moment includes environment A, environment B, and environment C, which correspond to message storage space A, message storage space B, and message storage space C, respectively, and the preset message running environment also includes environment A, environment B, and environment C, that is, there is no new message running environment, then there is no need to adjust the message partitions in the cache structure, that is, there is no need to update the cache structure of the message cache space.

[0058] Case 3: When it corresponds to another case mentioned above, and there is a new message running environment compared to the message running environment at the historical moment, the adjustment process of the message partitions in the cache structure is to re-unitize the message partitions in the cache structure on the basis of the message subject, the message storage space under the message subject and corresponding to the message running environment at the historical moment, and the message partitions in the message storage space, and increase the message storage space corresponding to the new message running environment. For example, assuming that the message running environment at the historical moment includes environment A and environment B, which correspond to message storage space A and message storage space B respectively, and the preset message running environment includes environment A, environment B and environment C, then environment C is the new running environment. At this time, it is necessary to re-unitize message partitions 1 to 6 in the cache structure, that is, to re-divide the above 6 message partitions, so that on the basis of message storage space A and message storage space B, message storage space C corresponding to environment C is added.

[0059] The updated cache structure at the current moment can be used as the cache structure of the message cache space at the next moment. After determining the cache structure of the message cache space, the message partitions in the cache structure are unitized by adjustment, that is, the message partitions in any message topic are divided into multiple message storage spaces according to the preset message running environment, so that messages in different message running environments can be obtained in any message topic, which is conducive to the isolated transmission and processing of messages.

[0060] In some embodiments, based on a preset message running environment, the message partitions in the cache structure are adjusted to obtain an updated cache structure, which can be achieved in the following way: first, a second environment identifier of the preset message running environment is obtained; then, a message storage space corresponding to the second environment identifier is created under the message topic; a preset number of message partitions are filtered out from N message partitions; finally, the filtered message partitions are divided into the message storage space corresponding to the second environment identifier to obtain an updated cache structure.

[0061] Each preset message running environment corresponds to a second environment identifier. For example, the second environment identifier of the production environment can be P, the second environment identifier of the test environment can be T, and the second environment identifier of the gray environment can be G. Creating a message storage space corresponding to the second environment identifier under the message subject can be based on the number of preset message running environments, creating the same number of message storage spaces under the message subject, and the second environment identifier can also be used as the name of the message storage space. Before the division, the number of message partitions in each message storage space can be preset, and which message partitions among the N message partitions are to be divided into a certain message storage space corresponding to the second environment identifier can be randomly selected. For example, the N message partitions can be evenly divided into the message storage space in order, and the N message partitions can also be divided into the message storage space as needed according to the amount of messages in different message running environments. There is no specific limitation on the division method.

[0062] Here, by creating a message storage space corresponding to the second environment identifier under the message topic, the messages stored in the message cache space are classified based on the message category, and the messages belonging to the same message category are divided again based on the message running environment, so that when the target message is subsequently read from the message cache space, the storage location of the target message can be quickly determined by determining the target message category and the environment identifier of the target message running environment. In addition, messages in different message running environments are stored in message partitions of different message storage spaces, which can solve the problem of cross-consumption of messages in multi-environment deployment scenarios and realize effective processing of target messages on the first client.

[0063] In some embodiments, creating a message storage space corresponding to a second environment identifier under a message subject can be achieved in the following manner: determining a storage directory structure of a message cache space; creating a second-level directory under a first-level directory corresponding to the message subject in the storage directory structure, and adding attribute information of the message storage space to the subject information of the message subject under the first-level directory; and creating a message storage space corresponding to the second environment identifier based on the second-level directory and the attribute information.

[0064] The storage directory structure includes multiple hierarchical directories, for example, a first-level directory corresponding to a message topic and a hierarchical directory corresponding to a message partition. The message topic corresponds to the first-level directory in the storage directory structure, which is used to store the metadata of the message topic. For example, the first-level directory may be / brokers / topics / ${topic}. The second-level directory is the hierarchical directory corresponding to the message storage space to be created in the storage directory structure, and the second-level directory is located under the first-level directory. Create a second-level directory corresponding to the message storage space under the first-level directory. For example, the second-level directory may be / brokers / topics / ${topic} / unit. In this way, the message partition corresponds to the third-level directory in the storage directory structure. For example, the third-level directory may be / brokers / topics / ${topic} / unit / partitions / ${partition}.

[0065] The topic information of the message topic refers to data related to the message topic. For example, the topic information may be the topic identifier of the message topic, the partition identifier and the number of partitions of the message partitions contained in the message topic, etc. On the basis of creating the second-level directory, the attribute information of the message storage space is added to the topic information of the message topic. The attribute information refers to data related to the message storage space. The attribute information includes the second environment identifier of the preset message running environment. For example, the attribute information may be the unit name (i.e., the second environment identifier), the number of partitions contained in each message storage space, and the partition identifier, etc. Here, by creating a second-level directory corresponding to the message storage space and adding the attribute information of the message storage space, it is possible to accurately create a message storage space corresponding to the environment identifier in the message topic, so that the message storage space can be isolated and delivered in a multi-environment deployment scenario based on the message storage space in the future, and the cross-consumption problem of messages can be effectively solved.

[0066] In the embodiment of the present application, the message sending strategy is used to characterize the sending method when the second client sends the target message to the server, and the second client is the message production end of the target message. The message sending strategy can be a unit routing strategy used to characterize that the target message is sent to the target message partition in the target message storage space, or it can be a random routing strategy used to characterize that the target message is randomly sent to the first message partition.

[0067] After receiving the message processing request sent by the first client, it is first necessary to determine the message sending policy of the second client for sending the target message. This can be done by storing the target message in the message cache space while storing the policy identifier of the message sending policy of the target message. When the first client needs to obtain the target message from the message cache space, the corresponding policy identifier is first obtained to determine the message sending policy of the second client for sending the target message. It can also be done that when the second client sends the target message, the message sending policy corresponding to the target message is saved in the configuration management center for managing message sending policies and message processing policies. When the first client needs to obtain the target message from the message cache space, the message sending policy corresponding to the target message is determined from the configuration management center.

[0068] Step S205: The server determines, based on the message sending policy, a message processing policy for instructing to process the message stored in the target storage location.

[0069] In an embodiment of the present application, a message processing strategy is used to indicate message processing for messages stored in a target storage location. Based on the message processing strategy, the method in which the first client obtains the target message from the server can be determined. The first client is the message consumer of the target message. To ensure that the first client can accurately obtain the target message and process it correctly, the message processing strategy needs to be consistent with the message sending strategy. On the basis of determining the message sending strategy, the message processing strategy can be determined. The message processing strategy includes a unit processing strategy and a random processing strategy. The unit processing strategy corresponds to the unit routing strategy of the message sending strategy, and the random processing strategy corresponds to the random routing strategy of the message sending strategy.

[0070] In some embodiments, based on the message sending strategy, determining a message processing strategy for indicating message processing to be performed on messages stored in a target storage location can be achieved in the following manner: when the message sending strategy is a unit routing strategy, determining the message processing strategy for indicating message processing to be performed on messages stored in the target storage location to be a unit processing strategy; when the message sending strategy is a random routing strategy, determining the message processing strategy for indicating message processing to be performed on messages stored in the target storage location to be a random processing strategy.

[0071] Among them, the unit routing strategy is used to characterize that the target message is sent to the target message partition of the target message storage space, and the unit processing strategy is used to indicate that the message stored in the target message partition of the target message storage space is processed; the random routing strategy is used to characterize that the target message is randomly sent to the first message partition; the random processing strategy is used to indicate that the message stored in the first message partition is processed. The first message partition refers to the message partition where the target message is randomly sent by the second client without considering the message running environment. In other words, the message sending strategy corresponds to the message processing strategy one by one. When the second client sends the target message to the target message partition of the target message storage space according to the unit routing strategy, the server needs to extract the target message from the target message partition of the target message storage space of the message cache space according to the unit processing strategy; when the second client randomly sends the target message to the first message partition according to the random routing strategy and ignores the dimension of the message storage space, the server needs to extract the target message from the first message partition of the message cache space according to the unit processing strategy.

[0072] Here, a message processing strategy for executing message processing is quickly determined based on the message sending strategy, so that the storage location of the target message in the message cache space can be accurately determined based on the message processing strategy, thereby achieving effective processing of the target message.

[0073] Step S206: The server obtains the target message from the message cache space based on the message processing strategy.

[0074] In an embodiment of the present application, obtaining a target message from a message cache space based on a message processing strategy can be achieved in the following manner: when the message processing strategy is used to indicate message processing for a message stored in a target message partition (i.e., the message processing strategy is a unit processing strategy), first, obtaining a target message category and a first environment identifier of a target message operating environment carried in a message processing request; determining a target message subject corresponding to the target message category from the message cache space; then, determining a target message storage space corresponding to the first environment identifier from the target message subject; and finally, obtaining the target message from a target message partition of the target message storage space.

[0075] The unit processing strategy is used to indicate that the message processing is performed on the message stored in the target message partition of the target message storage space. When the message processing strategy is the unit processing strategy, the message processing request is parsed and processed to obtain the target message category corresponding to the target message and the first environment identifier of the target message running environment. The cache structure of the message cache space is an updated cache structure, so the message cache space includes a message subject, a message storage space located under the message subject and corresponding to the preset message running environment, and a message partition located in the message storage space. The target message category refers to the message category to which the target message belongs in all message categories. The message category corresponds to the message subject, and different message subjects store messages of different message categories. Therefore, when the target message category of the target message is determined, it can be determined that the target message is stored in the target message subject of the message cache space. The target message running environment is the message running environment in which the target message is located in the preset message running environment. Each message running environment has a unique environment identifier, and each environment identifier corresponds to a message storage space in the message subject. On the basis of determining the target message subject and the first environment identifier of the target message running environment, it can be determined that the target message is stored in the target message storage space corresponding to the first environment identifier in the target message subject. The final storage location of the target message in the message cache space is in the target message partition of the target message storage space.

[0076] Here, the storage location of the target message in the message cache space can be quickly determined through the target message category and the first environment identifier of the target message operating environment carried in the message processing request, so that the first client can accurately obtain the target message in the target message operating environment, thereby realizing effective processing of the target message.

[0077] In other embodiments, obtaining the target message from the message cache space based on the message processing strategy can also be achieved in the following ways: when the message processing strategy is a random processing strategy, obtaining the target message category carried in the message processing request; determining the target message topic corresponding to the target message category from the message cache space; and obtaining the target message from the first message partition of the target message topic.

[0078] That is to say, without considering the message running environment of the message, the second client randomly sends the target message to the first message partition of the target message topic corresponding to the target message category based on the random sending strategy. The message processing strategy is determined to be a random processing strategy, and the message storage space in the message cache space can be directly ignored. The message processing request is parsed to obtain the target message category of the target message. Based on the target message category, the target message topic storing the target message can be determined, and the target message can be directly obtained from the first message partition of the target message topic.

[0079] Here, for multi-environment deployment scenarios and single-environment deployment scenarios, two different message sending strategies and message processing strategies can be used respectively to achieve effective processing of target messages without increasing external resource costs.

[0080] Step S207: the server sends a target message to the terminal.

[0081] Step S208: the terminal processes the target message in the target message running environment to obtain a message processing result of the target message.

[0082] In the embodiment of the present application, the terminal is the first client. After the first client receives the target message, the first client processes the target message according to the preset business logic in the target message operation environment to obtain the message processing result of the target message. Since the target message is a message in the target message operation environment, when the first client processes the target message, there is no cross-consumption problem of the target message, and the target message can be effectively processed.

[0083] In the embodiment of the present application, by adjusting the cache structure of the message cache space, the message cache space can add the division of messages under different message running environments on the basis of distinguishing the message categories when storing messages. There is no need to deploy different message cache spaces or increase the message topics in the message cache space for different message running environments, which greatly reduces resource loss and improves resource utilization.

[0084] In addition, by configuring the message sending strategy and the message processing strategy, the message processing method provided in the embodiment of the present application is applicable to both multi-environment deployment scenarios and single-environment deployment scenarios, and can realize effective processing of target messages in two different scenarios. In particular, in the multi-environment deployment scenario, after determining the target message category of the target message and the first environment identifier of the target message operating environment, the target message can be quickly obtained from the message cache space, and in the target message operating environment, the target message can be effectively processed through the first client.

[0085] The following is an explanation of an exemplary application of the embodiments of the present application in a practical application scenario.

[0086] The embodiment of the present application provides a message processing system, which includes a message sending end 401 (i.e., the second client mentioned above), a Kafka cluster 402, a message consuming end 403 (i.e., the first client mentioned above), and a configuration management center 404. The system architecture diagram of the message processing system is shown in FIG. Figure 4As shown. For multi-environment deployment scenarios, the Kafka server in the Kafka cluster 402 is modified, and the message unit parameter is added to the message topic to divide the message partition into units to obtain multiple message units (i.e., the above-mentioned message storage space). The unit name of the message unit is the environment identifier of the message running environment of the message. Messages in different message running environments can be stored separately in the message partitions of the message unit corresponding to the environment identifier. Based on the message unit, the cross-consumption problem of messages in multi-environment deployment scenarios can be effectively avoided, and the target message can be effectively processed.

[0087] The configuration management center 404 is used to configure the message sending routing strategy 405 (i.e., the above-mentioned message sending strategy) of the message sending end 401 and the message processing strategy 406 of the message consuming end; wherein, the message sending routing strategy 405 is used to control the sending behavior of the message sending end 401 for the target message, and the message processing strategy 406 is used to control the consumption behavior of the message consuming end 403 for the target message. The message sending routing strategy 405 includes a unit routing strategy and a random routing strategy. The message processing strategy 406 includes a limited unit consumption strategy (i.e., the above-mentioned unit processing strategy) and a full consumption strategy (i.e., the above-mentioned random processing strategy).

[0088] The message sending end 401 is used to respond to the data sending request, obtain the message sending routing strategy 405 from the configuration management center 404, and confirm the sending unit when the message sending routing strategy 405 is a unit routing strategy, that is, it can be determined whether the message unit to which the target message is sent is the message unit 1 or the message unit 2 of the message topic A in the kafka cluster 402. It should be noted that the kafka cluster 402 includes one or more kafka servers, which can be selected according to actual business needs. The message topic corresponds to the message category of the message. Before confirming the sending unit, the message category of the target message must also be determined. The message unit corresponds to the message running environment of the message, and the sending unit can be confirmed based on the actual message running environment of the target message. The message unit includes at least one message partition, for example Figure 4 In the example, message unit 1 includes message partition 1, message partition 2 and message partition 3, and message unit 2 includes message partition 4, message partition 5 and message partition 6. After confirmation, the target message is sent to the message partition of the message unit corresponding to the message sending routing strategy 405 of the kafka cluster 402. When the message sending routing strategy 405 is a random routing strategy, the target message can be randomly sent to the message partition of the message topic A without distinguishing the message unit.

[0089] The kafka cluster 402 is used to receive the target message sent by the message sending end 401, allocate the target message to the message partition of the message unit corresponding to the message sending routing strategy 405; and send a data consumption request to the message consumer end 403 that has pre-subscribed to the message topic to which the message unit belongs.

[0090] The message consumer 403 is used to respond to the data consumption request sent by the kafka cluster 402 and obtain the message processing strategy 406 from the configuration management center 404. When the message processing strategy 406 is a limited unit consumption strategy, the message unit is confirmed, that is, it can be determined whether the target message is stored in message unit 1 or message unit 2. After confirmation, the target message is read from the message partition of the message unit in the kafka cluster 402 and the message is consumed. When the message processing strategy 406 is a full consumption strategy, the target message is directly read from the message partition of the message topic A in the kafka cluster 402 and the message is consumed.

[0091] That is to say, in the kafka sending / consuming client SDK, under the original service-oriented implementation scheme, conditional control of sending and consuming strategies is added, so that the client generates different sending / consuming behaviors under different strategies. The specific strategy is: add message sending routing strategy 405 at the message sending end 401, and add message processing strategy 406 at the message consuming end 403. If the strategy is set to unit routing strategy and limited unit consumption strategy, messages can be sent and consumed according to a fixed message unit group according to the specified attribute information; if the strategy is set to random routing strategy and all consumption strategy, the message unit is ignored, and messages are directly sent and consumed randomly according to all message partitions under the message topic.

[0092] Figure 5 The following is a schematic diagram of the implementation of the message storage process in the Kafka cluster provided in the embodiment of the present application, as shown in FIG. Figure 5As shown in the figure, the specific implementation process is as follows: 01. Create a kafka application programming interface (API). The kafka API parameters can be message topic name, number of message partitions, replicas, and message units. 02. Calculate the total number of message partitions. The total number of message partitions for each message topic in the kafka cluster is determined by the number of message partitions partitionCount and message units in the kafka API parameters. The calculation method for the total number of message partitions is: the number of message partitions for each message unit * the number of message units. 03. Create message topics and create multiple message topics based on the total number of message partitions. 04. Metadata writing: write the metadata of the message topic (i.e. the above topic information) to the coordination service unit (zookeeper) directory, and adjust the storage directory structure to / brokers / topics / ${topic} / unit / partitions / ${partition}, which means that in the directory of the message topic under the root directory of the kafka server (i.e. / brokers / topics / ${topic}), add the directory of the message unit, and divide the directory of the message partition into the directory of the message unit. And adjust the value of the directory / brokers / topics / ${topic} in zookeeper, add attribute information under its original content structure, and the modified content is as follows: Figure 6 As shown, the bold font section is the newly added content structure (that is, the content structure corresponding to the message unit). 05. Message cache: cache the messages in the message topic into the message cache space in the Kafka cluster.

[0093] The technical solution of the present application divides the message partitions of the message topic in the Kafka cluster into units, and divides the multiple message partitions under a message topic into different message units according to the actual business needs (i.e., the message operating environment), and each message operating environment corresponds to a message unit. After the partition unit processing, the client can isolate and transmit messages according to the message unit dimension by deploying different message operating environments, without the need to isolate different message operating environments according to Kafka clusters or topics, reducing the occurrence of infinite expansion of the number of Kafka clusters or message topics during the operation and maintenance process, reducing the operation and maintenance costs to a certain extent, and improving resource utilization.

[0094] The following further describes an exemplary structure of the message processing device 455 provided in the embodiment of the present application implemented as a software module. In some embodiments, for example Figure 7As shown, the software modules in the message processing device 455 may include: a target message acquisition module 4551, used to obtain the target message from the message cache space in response to a message processing request sent by the first client; wherein the message cache space includes at least one message storage space, each message storage space includes an environment identifier of a message running environment; the message processing request carries the first environment identifier of the target message running environment; the target message is stored in the target message partition of the target message storage space corresponding to the first environment identifier; a message sending module 4552, used to send the target message to the first client.

[0095] In some embodiments, the target message acquisition module 4551 is also used to: determine the message sending strategy of the target message in response to the message processing request; the message sending strategy is used to characterize the sending method when the second client sends the target message to the server, and the second client is the message production end of the target message; based on the message sending strategy, determine the message processing strategy used to instruct message processing of the message stored in the target storage location; based on the message processing strategy, obtain the target message from the message cache space.

[0096] In some embodiments, the message processing request also carries a target message category; the target message acquisition module 4551 is also used to: when the message processing strategy is used to indicate message processing for the message stored in the target message partition, determine the target message topic corresponding to the target message category from the message cache space; determine the target message storage space corresponding to the first environment identifier under the target message topic; and obtain the target message from the target message partition of the target message storage space.

[0097] In some embodiments, the device 455 also includes a cache structure adjustment module, which is used to: determine the cache structure of the message cache space; the cache structure includes N message partitions; N is an integer greater than 1; based on a preset message running environment, the message partitions in the cache structure are adjusted to obtain an updated cache structure; the updated cache structure includes a message storage space located under the message topic and corresponding to the preset message running environment, and a message partition located in the message storage space.

[0098] In some embodiments, the cache structure adjustment module is also used to: obtain a second environment identifier of the preset message running environment; create a message storage space corresponding to the second environment identifier under the message topic; filter out a preset number of message partitions from the N message partitions; divide the filtered message partitions into the message storage space corresponding to the second environment identifier to obtain the updated cache structure.

[0099] In some embodiments, the cache structure adjustment module is also used to: determine the storage directory structure of the message cache space; the storage directory structure includes multiple hierarchical directories; under the first hierarchical directory corresponding to the message subject in the storage directory structure, create a second hierarchical directory, and add attribute information of the message storage space in the subject information of the message subject under the first hierarchical directory; the attribute information includes the second environment identifier; the second hierarchical directory is the hierarchical directory corresponding to the message storage space to be created in the storage directory structure; based on the second hierarchical directory and the attribute information, create a message storage space corresponding to the second environment identifier.

[0100] It should be noted that the description of the device of the embodiment of the present application is similar to the description of the above method embodiment, and has similar beneficial effects as the method embodiment, so it is not repeated. For technical details not disclosed in the embodiment of the device, please refer to the description of the method embodiment of the present application for understanding.

[0101] Correspondingly, an embodiment of the present application provides an electronic device, Figure 8 A schematic diagram of the structure of an electronic device provided in an embodiment of the present application is shown in FIG. Figure 8 As shown, the electronic device 1200 includes at least: a processor 1201 , a communication interface 1202 , and a storage medium 1203 configured to store executable instructions, wherein: the processor 1201 generally controls the overall operation of the electronic device 1200 .

[0102] The communication interface 1202 enables the electronic device to communicate with other terminals or servers through a network.

[0103] The storage medium 1203 is configured to store instructions and applications executable by the processor 1201, and can also cache data to be processed or processed by the processor 1201 and each module in the electronic device 1200, which can be implemented through flash memory (FLASH) or random access memory (Random Access Memory, RAM).

[0104] The present application embodiment provides a computer-readable storage medium in which computer-executable instructions are stored. When the computer-executable instructions are executed by a processor, the processor will execute the message processing method provided by the present application embodiment, for example, Figure 2 The message processing method is shown.

[0105] The embodiment of the present application provides a computer program product, which includes computer executable instructions, and the computer executable instructions are stored in a computer readable storage medium. The processor of the electronic device reads the computer executable instructions from the computer readable storage medium, and the processor executes the computer executable instructions, so that the electronic device executes the message processing method described above in the embodiment of the present application.

[0106] In some embodiments, the computer-readable storage medium may be a memory such as RAM, ROM, flash memory, magnetic surface memory, optical disk, or CD-ROM; or may be various devices including one or any combination of the above memories.

[0107] In some embodiments, computer executable instructions may be in the form of a program, software, software module, script or code, written in any form of programming language (including compiled or interpreted languages, or declarative or procedural languages), and may be deployed in any form, including as a stand-alone program or as a module, component, subroutine or other unit suitable for use in a computing environment.

[0108] As an example, computer-executable instructions may, but need not, correspond to a file in a file system, may be stored as part of a file that stores other programs or data, such as in one or more scripts in a HyperText Markup Language (HTML) document, in a single file dedicated to the program in question, or in multiple coordinated files (e.g., files storing one or more modules, subroutines, or code portions).

[0109] As an example, computer executable instructions may be deployed to be executed on one electronic device, or on multiple electronic devices located at one site, or on multiple electronic devices distributed at multiple sites and interconnected by a communication network.

[0110] The above is only an embodiment of the present application and is not intended to limit the protection scope of the present application. Any modifications, equivalent substitutions and improvements made within the spirit and scope of the present application are included in the protection scope of the present application.

Claims

1. A message processing method, characterized in that: The method comprises: Retrieving a target message from a message cache space in response to a message processing request sent by a first client; wherein the message cache space includes at least one message storage space, each message storage space including an environment identifier of a message execution environment; the message processing request carries a first environment identifier of the target message execution environment; and the target message is stored in a target message partition of the target message storage space corresponding to the first environment identifier; The target message is sent to the first client.

2. The method according to claim 1, characterized in that The acquiring of the target message from the message buffer space includes: Determining a message sending strategy for the target message; Determining, based on the message sending policy, a message processing policy for instructing to perform message processing on the message stored in the target storage location; Based on the message processing strategy, the target message is obtained from the message buffer space.

3. The method according to claim 2, characterized in that The message processing request also carries a target message category; and obtaining the target message from a message cache space based on the message processing policy includes: In a case where the message processing policy is used to instruct to perform message processing on the message stored in the target message partition, determining a target message topic corresponding to the target message category from the message cache space; Determining a target message storage space corresponding to the first environment identifier under the target message topic; The target message is obtained from the target message partition of the target message storage space.

4. The method according to claim 1, wherein Before responding to the message processing request sent by the first client, the method further includes: Determine a cache structure of the message cache space; the cache structure includes N message partitions; N is an integer greater than 1; Based on the preset message running environment, the message partition in the cache structure is adjusted to obtain an updated cache structure; the updated cache structure includes a message storage space located under the message topic and corresponding to the preset message running environment, and a message partition located in the message storage space.

5. The method according to claim 4, characterized in that The message partitions in the cache structure are adjusted based on the preset message running environment to obtain an updated cache structure, including: Obtaining a second environment identifier of the preset message execution environment; Creating a message storage space corresponding to the second environment identifier under the message topic; Filtering a preset number of message partitions from the N message partitions; The filtered message partitions are divided into the message storage space corresponding to the second environment identifier to obtain the updated cache structure.

6. The method according to claim 5, characterized in that The creating a message storage space corresponding to the second environment identifier under the message topic includes: Determine a storage directory structure of the message cache space; the storage directory structure includes multiple hierarchical directories; Creating a second-level directory under the first-level directory corresponding to the message subject in the storage directory structure, and adding attribute information of the message storage space to the subject information of the message subject under the first-level directory; the attribute information includes the second environment identifier; the second-level directory is the hierarchical directory corresponding to the message storage space to be created in the storage directory structure; Based on the second hierarchical directory and the attribute information, a message storage space corresponding to the second environment identifier is created.

7. A message processing device, characterized in that: The device comprises: A target message acquisition module is configured to acquire a target message from a message cache space in response to a message processing request sent by a first client; wherein the message cache space includes at least one message storage space, each message storage space including an environment identifier of a message execution environment; the message processing request carries the first environment identifier of the target message execution environment; and the target message is stored in a target message partition of the target message storage space corresponding to the first environment identifier; A message sending module is used to send the target message to the first client.

8. An electronic device, characterized in that: include: a memory for storing computer-executable instructions; The processor is configured to implement the message processing method according to any one of claims 1 to 6 when executing the computer-executable instructions stored in the memory.

9. A computer-readable storage medium, characterized in that Computer-executable instructions are stored, and when the computer-executable instructions are executed by a processor, the message processing method according to any one of claims 1 to 6 is implemented.

10. A computer program product, characterized in that The computer program product includes computer-executable instructions stored in a computer-readable storage medium; Wherein, when the processor of the electronic device reads the computer-executable instructions from the computer-readable storage medium and executes the computer-executable instructions, the message processing method according to any one of claims 1 to 6 is implemented.

Images