Data transmission method, device and storage medium based on message queue

Abstract

The present application discloses a data transmission method, device, computer equipment and storage medium based on a message queue, the method comprising: receiving message data sent by a message producer, the message data carrying the message generated by the message producer and the data source information associated with the message; determining the message queue associated with the data source information, and generating a target message subject according to the data source information; determining the target storage area associated with the target message subject in the message queue, and storing the target message subject and the message in the target storage area; after receiving a message subscription request carrying data source information sent by a message consumer, pulling the message corresponding to the target message subject from the target storage area of ​​the message queue according to the data source information and pushing the message to the message consumer. The present application can improve the convenience of storing data in the message queue and pulling data from the message queue, improve the efficiency of data transmission, and has high applicability.

Description

Technical Field

[0001] The present application relates to the field of computer technology, and in particular to a data transmission method, device and storage medium based on a message queue. Background Art

[0002] In the field of big data, message queues are important components in distributed systems. Message queues are a type of middleware that mainly solves problems such as application coupling, asynchronous messages, and traffic peak shaving. In existing producer and consumer applications based on message queues, upstream producer applications generate messages based on pre-defined message topics and send them to the message queue middleware, and downstream consumer applications consume the corresponding messages based on the message topics agreed upon in advance by the upstream producer applications. However, in such an application model, when the message topic in the message generated by the upstream producer changes, the downstream consumer application must be notified, and the developer must make code adjustments and secondary development on the consumer application to adapt to the new message topic generated by the upstream producer. In addition, when the message queue middleware is reselected or the technology is upgraded, the application architecture of both the upstream producer and the downstream consumer needs to be reconstructed and the code redeveloped to store messages in the message queue or pull data from the message queue, which is cumbersome, costly to implement, and poor applicability. Summary of the invention

[0003] The present application provides a data transmission method, device and storage medium based on a message queue, which can improve the convenience of storing data in a message queue and pulling data from a message queue, improve the efficiency of data transmission, and have high applicability.

[0004] In a first aspect, the present application provides a data transmission method based on a message queue, the method being applicable to a message manager, the method comprising:

[0005] The message manager receives the message data sent by the message producer, wherein the message data carries the message generated by the message producer and the data source information associated with the message;

[0006] The message manager determines the message queue associated with the data source information, and generates a target message topic according to the data source information;

[0007] The message manager determines a target storage area associated with the target message subject in the message queue, and stores the target message subject and the message in the target storage area;

[0008] When receiving a message subscription request sent by a message consumer and the message subscription request carries the data source information, the message manager determines the target message topic from the target storage area of ​​the message queue based on the data source information, pulls the message corresponding to the target message topic from the target storage area and pushes the message to the message consumer.

[0009] In combination with the first aspect, in a possible implementation, the message queue includes at least one storage area including the target storage area, one storage area is associated with a message topic and is used to store messages of the message topic, and messages of the same message topic are associated with the same data source information.

[0010] In combination with the first aspect, in a possible implementation manner, the data source information includes a data source name of the data source to which the message belongs and a project name of a project to which the data source belongs, and the project to which the data source belongs includes at least one data source including the data source;

[0011] The above-mentioned generating a target message topic according to the above-mentioned data source information includes:

[0012] A target message subject is generated according to the project name and the data source name, wherein one data source name corresponds to one message subject.

[0013] In combination with the first aspect, in a possible implementation, the above-mentioned message data includes a message mode identifier carried in the header information of the above-mentioned message, and the above-mentioned message mode identifier is used to identify the message mode of the above-mentioned message, so that the above-mentioned message consumer can consume the above-mentioned message based on the message mode.

[0014] In combination with the first aspect, in a possible implementation manner, after the message manager receives the message data sent by the message producer, the method further includes:

[0015] The message manager registers the message mode of the message to the message registration service center, and generates a message mode identifier corresponding to the message mode of the message through the message registration service center;

[0016] When a message mode acquisition request sent by the message consumer is received and the message mode acquisition request carries the message mode identifier, the message mode corresponding to the message mode identifier is acquired from the message registration service center and the message mode is sent to the message consumer.

[0017] In a second aspect, the present application provides a data transmission method based on a message queue, which is applicable to a message consumer, and the method includes:

[0018] The message consumer obtains the data source information of the data source created by the message producer;

[0019] When the message consumer requests to subscribe to the message generated by the message producer, a message subscription request is sent to the message manager. The message subscription request carries data source information, and the data source information is used to determine the target message topic that the message consumer requests to subscribe to.

[0020] The message consumer receives the message associated with the target message topic from the message manager, and parses and consumes the message.

[0021] In conjunction with the second aspect, in a possible implementation manner, the parsing and consuming the message includes:

[0022] The message consumer obtains the message mode identification information from the header information of the message, and determines the message mode of the message according to the message mode identification information;

[0023] The above message is parsed and consumed based on the message mode of the above message.

[0024] In conjunction with the second aspect, in a possible implementation manner, the parsing and consuming the message further includes:

[0025] The message consumer obtains the message mode identifier from the header information of the message, and sends a message mode acquisition request to the message manager, wherein the message mode acquisition request carries the message mode identifier;

[0026] The message mode corresponding to the message mode identifier is obtained from the message manager, and the message is parsed and consumed based on the message mode.

[0027] In a third aspect, a data transmission method based on a message queue is provided, the method being applicable to a message producer, the method comprising:

[0028] Create a data source, and generate a message and data source information associated with the message based on the data source;

[0029] Sending message data to a message manager, wherein the message data carries the message and the data source information.

[0030] In conjunction with the third aspect, in a possible implementation manner, after the message producer creates a data source, the method further includes:

[0031] Sending data source information of the above data source to a message manager and / or a message consumer;

[0032] The above data source information includes the data source name and the project name of the project to which the data source belongs.

[0033] In a fourth aspect, the present application provides a data transmission device based on a message queue, the data transmission device being applicable to a message manager, and the data transmission device comprising:

[0034] A receiving module, used to receive message data sent by a message producer, wherein the message data carries the message generated by the message producer and data source information associated with the message;

[0035] A determination module, used to determine the message queue associated with the data source information received by the message data receiving module, and generate a target message topic according to the data source information;

[0036] The determination module is further configured to determine a target storage area associated with the target message subject in the message queue, and store the target message subject and the message in the target storage area;

[0037] The receiving module is further used to receive a message subscription request sent by a message consumer, wherein the message subscription request carries the data source information;

[0038] The message push module is used to determine the target message topic from the target storage area of ​​the message queue according to the data source information carried in the message subscription request received by the receiving module, pull the message corresponding to the target message topic from the target storage area and push the message to the message consumer.

[0039] In combination with the fourth aspect, in a possible implementation, the message queue includes at least one storage area including the target storage area, one storage area is associated with a message topic and is used to store messages of the message topic, and messages of the same message topic are associated with the same data source information.

[0040] In conjunction with the fourth aspect, in a possible implementation, the data source information includes a data source name of the data source to which the message belongs and a project name of a project to which the data source belongs, and the project to which the data source belongs includes at least one data source including the data source; and the determination module is used to:

[0041] A target message subject is generated according to the project name and the data source name, wherein one data source name corresponds to one message subject.

[0042] In conjunction with the fourth aspect, in a possible implementation manner, the message queue-based message transmission device further includes:

[0043] A registration module, used to register the above message mode with a message registration service center, and generate a message mode identifier corresponding to the message mode of the above message through the above message registration service center;

[0044] The above message push module is also used for:

[0045] After receiving the message mode acquisition request sent by the above-mentioned message consumer and the above-mentioned message mode acquisition request carries the above-mentioned message mode identifier, the above-mentioned message mode corresponding to the above-mentioned message mode identifier is obtained from the above-mentioned message registration service center and the above-mentioned message mode is sent to the above-mentioned message consumer.

[0046] In a fifth aspect, the present application provides a data transmission device based on a message queue, the data transmission device being applicable to a message consumer, and the data transmission device comprising:

[0047] The acquisition module is used to obtain the data source information of the data source created by the message producer;

[0048] A sending module, used to send a message subscription request to a message manager when subscribing to a message generated by the message producer, wherein the message subscription request carries data source information, and the data source information is used to determine a target message subject for the subscription request;

[0049] A receiving module, used for receiving a message associated with the target message topic from the message manager;

[0050] The parsing and consuming module is used to parse and consume the above message received by the above receiving module.

[0051] In conjunction with the fifth aspect, in a possible implementation manner, the above-mentioned parsing and consumption module is used to:

[0052] Acquire message mode identification information from the header information of the message received by the above-mentioned receiving module, and determine the message mode of the above-mentioned message according to the above-mentioned message mode identification information;

[0053] The above message is parsed and consumed based on the message mode of the above message.

[0054] In conjunction with the fifth aspect, in a possible implementation manner, the above-mentioned parsing and consumption module is used to:

[0055] Obtaining a message mode identifier from the header information of the message received by the receiving module, and sending a message mode acquisition request to the message manager through the sending module, wherein the message mode acquisition request carries the message mode identifier;

[0056] The message mode corresponding to the message mode identifier is obtained from the message manager, and the message is parsed and consumed based on the message mode.

[0057] In a sixth aspect, the present application provides a data transmission device based on a message queue, the data transmission device being applicable to a message producer, and the data transmission device comprising:

[0058] Create a module to create a data source;

[0059] A message generation module, which generates a message and data source information associated with the message according to the data source created by the creation module;

[0060] The sending module is used to send message data to the message manager, wherein the message data carries the message generated by the message generating module and the data source information.

[0061] In conjunction with the sixth aspect, in a possible implementation manner, the sending module is further used to:

[0062] Sending data source information of the above data source to a message manager and / or a message consumer;

[0063] The above data source information includes the data source name and the project name of the project to which the data source belongs.

[0064] In a seventh aspect, the present application provides a computer device, including: a processor, a memory, a transceiver, and a network interface;

[0065] The processor is connected to a memory and a network interface, wherein the network interface is used to provide a data communication function, the memory is used to store a computer program, and the processor and the transceiver are used to call the computer program to execute the method executed by the above-mentioned first aspect and any possible implementation method of the first aspect in the present application.

[0066] In an eighth aspect, the present application provides a computer device, including: a processor, a memory, a transceiver, and a network interface;

[0067] The processor is connected to a memory and a network interface, wherein the network interface is used to provide a data communication function, the memory is used to store a computer program, and the processor and the transceiver are used to call the computer program to execute the method executed by the above-mentioned second aspect of the present application and any possible implementation method of the second aspect.

[0068] In a ninth aspect, the present application provides a computer device, including: a processor, a memory, a transceiver, and a network interface;

[0069] The processor is connected to a memory and a network interface, wherein the network interface is used to provide a data communication function, the memory is used to store a computer program, and the processor and the transceiver are used to call the computer program to execute the method executed by the above-mentioned third aspect of the present application and any possible implementation method of the third aspect.

[0070] In the tenth aspect, the present application provides a computer-readable storage medium, which stores a computer program, and the computer program includes program instructions. When the program instructions are executed by a processor, they execute the method performed by the above-mentioned first aspect of the present application and any possible implementation method of the first aspect.

[0071] In the eleventh aspect, the present application provides a computer-readable storage medium, which stores a computer program, and the computer program includes program instructions. When the program instructions are executed by a processor, they execute the method performed by the above-mentioned second aspect of the present application and any possible implementation method of the second aspect.

[0072] In the twelfth aspect, the present application provides a computer-readable storage medium, which stores a computer program, and the computer program includes program instructions. When the program instructions are executed by a processor, they execute the method performed by the above-mentioned third aspect of the present application and any possible implementation method of the third aspect.

[0073] In this application, the message producer can create the data source by itself when generating a message. After the data source is created, the message manager will operate to bind the corresponding message queue component and create a target storage area in the corresponding message queue that corresponds to the data source binding relationship. Message consumers and message producers can reuse this data source. The message manager can store or pull data from the target storage area corresponding to the data source, which can improve the convenience of the message manager storing data in the message queue and pulling data from the message queue, improve the efficiency of data transmission, and has high applicability. At this time, the message queue is transparent to both message producers and message consumers. The message producer does not need to care about the specific middleware component of the message queue used. They are related through the data source. As long as the message producer is bound to a data source, if the message producer changes the underlying message queue later, the message manager will create a new target storage area in the new message queue according to the data source bound to the message producer when switching the component type of the message queue. The application of the message producer can be switched seamlessly without modifying the program and configuration on the application side, which can reduce the cost of message producers or message consumers to learn different types of message queues, simplify the development model of message queue applications, reduce the difficulty of development technology, and help business parties to implement the business logic of upper-level industrial applications faster and more efficiently. BRIEF DESCRIPTION OF THE DRAWINGS

[0074] In order to more clearly illustrate the technical solution in the present application, the drawings required for use in the description of the embodiments will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present application. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying any creative work.

[0075] Figure 1 It is a structural diagram of the data transmission system architecture based on the message queue provided by this application;

[0076] Figure 2 It is an interactive schematic diagram of data transmission based on message queue provided by this application;

[0077] Figure 3 This is a flow chart of a data transmission method based on a message queue provided by the present application;

[0078] Figure 4 is another flow chart of the data transmission method based on the message queue provided by the present application;

[0079] Figure 5 is another flow chart of the data transmission method based on the message queue provided by the present application;

[0080] Figure 6 It is a structural diagram of a data transmission device based on a message queue provided by the present application;

[0081] Figure 7 is another structural schematic diagram of the data transmission device based on the message queue provided by the present application;

[0082] Figure 8 is another structural schematic diagram of the data transmission device based on the message queue provided by the present application;

[0083] Fig. 9 It is a structural schematic diagram of a computer device provided by this application;

[0084] Fig.10 is another structural schematic diagram of the computer device provided by the present application;

[0085] Fig.11 It is another structural schematic diagram of the computer device provided by this application. DETAILED DESCRIPTION

[0086] The following will be combined with the drawings in this application to clearly and completely describe the technical solutions in this application. Obviously, the described embodiments are only part of the embodiments of this application, not all of them. Based on the embodiments in this application, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of this application.

[0087] See also Figure 1 , Figure 1 This is a schematic diagram of the structure of a data transmission system architecture based on a message queue provided by this application. Figure 1 As shown, the network architecture may include a server 20 and a user terminal cluster, and the user terminal cluster may include multiple user terminals, such as Figure 1 As shown, it may specifically include user terminal 100a, user terminal 100b, user terminal 100c, ..., user terminal 100n.

[0088] It is understandable that in the optional embodiments of the present application, the message producer and the message consumer may be user terminals in the above-mentioned user terminal cluster, the message producer may be a user terminal that generates and stores messages in the message queue, and the message consumer may be a user terminal that subscribes to and consumes messages stored in the message queue. For the convenience of description, the following will be described by taking the message producer and the message consumer as an example. The above-mentioned user terminal may include but is not limited to mobile phones, tablet computers, laptops, PDAs, smart speakers, mobile Internet devices (mobile internet device, MID), points of sales (point of sales, POS), wearable devices (such as smart watches, smart bracelets, etc.). In the embodiments of the present application, the above-mentioned message manager, message queue, data source, and message registration service center may be part of the same server, or may be distributed in different servers. The above-mentioned server may be an independent physical server, or may be a server cluster or distributed system composed of multiple physical servers, or may be a cloud server that provides cloud services, cloud databases, cloud computing, cloud functions, cloud storage, network services, cloud communications, middleware services, domain name services, security services, CDN, and basic cloud computing services such as big data and artificial intelligence platforms. It can be determined according to the actual application scenario and is not limited here. For the convenience of description, the following will use the message manager as the server or message transmission center in the distributed system (such as Figure 1 The server 20 in the example is used for explanation. The message queue can be a middleware in a distributed system, which is used to store messages generated by message producers. The message manager can receive messages from message producers and store them in the message queue, or pull data from the message queue and push it to message consumers, or register message modes such as message types of messages with the message registration service center, or obtain message modes requested by message consumers from the message registration service center and send them to message consumers, etc.

[0089] In some possible implementations, such as Figure 1As shown, user terminal 100a, user terminal 100b, user terminal 100c, ..., user terminal 100n can be connected to the server 20 through the network connection, so that each user terminal can exchange data with the server 20 through the network connection. For example, any user terminal in the user terminal cluster that is loaded with a message production application (i.e., a message producer, such as user terminal 100a) sends message data carrying data source information to the server 20 (i.e., a message manager), or the server 20 sends the corresponding message data to the user terminal (such as user terminal 100b) when receiving a message subscription request carrying data source information sent by any user terminal in the user terminal cluster that is loaded with a message consumption application (i.e., a message consumer, such as user terminal 100b). The above-mentioned message production application and message consumption application can be a social application or an online shopping application, etc., and the above-mentioned message data can be a chat message sent by a user or an order message, which can be determined according to the actual use scenario and is not limited here.

[0090] See also Figure 2 , Figure 2 This is a schematic diagram of an interactive data transmission based on a message queue provided by this application. Figure 2 As shown, the message queue-based data transmission system provided by the present application may include a message producer 10a, a message manager 10b, a message consumer 10c, a data source 10d, a message registration center 10e, and a message queue 10f. Figure 1The data source 10d in the example is a data source created by the message producer 10a or the message consumer 10c and can be used by both, or a data source specified by the protocol, such as a data source bound to the message producer 10a, and then notified to the message consumer 10c through the protocol. The data source 10d here can include relational databases (such as Mysql, Oracle, Nosql databases) and can also include a data source such as "message cache". "Message cache" refers to a data source that can be reused, which is a logical concept. Here, the data source name can be customized by the user, and the data source name can be composed of all lowercase English letters, underscores or numbers. Creating a data source 10d also requires a project, which is a collection of data sources for different purposes created by the user. The project is the basic part of the real-time stream computing system. Before performing other operations, you must first create a project, and then manage the data sources, real-time computing, resources and members under each project in the project management. The source end of the data source 10d is mainly connected to databases (Mysql / PG / SQLServer / Oracle)\collection log files, JQData, Kafka and other types of databases. The data source 10d is used to connect with the message queue 10f, associate the message producer 10a and the message consumer 10c, and then push the data from the message queue 10f to the real-time stream computing system, and then use it for downstream computing tasks. Real-time computing mainly refers to real-time computing tasks that connect with the message queue 10f, such as flink or spark streaming tasks. When the message producer 10a or the message consumer 10c creates the data source 10d and reports it to the message manager 10b, the message manager 10b binds the message queue (such as message queue 10f) corresponding to the data source 10d, and generates the corresponding target message topic according to the above data source information. Among them, the corresponding relationship between the data source 10d and the message topic can be, message topic = project name + data source name. It can be understood that the corresponding message topic can be directly found in the message queue 10f through the data source 10d, and the message and the data source 10d can be associated through the corresponding relationship between the message topic and the message, so that the message topic in the message queue 10f can be found through the data source 10d, and the message can be found through the message topic. Therefore, the message manager 10b can find the message topic in the message queue 10f through the data source 10d, find the message through the message topic, and thus realize the production and consumption of data in the corresponding target storage area in the corresponding message queue 10f. The data source 10d created at this time can be used as a source input end or a sink output end, as a data source for various forms of real-time computing engines such as flink, or as the place where data flows after real-time computing.

[0091] in, Figure 2The message producer 10a in the figure is a user terminal that generates message data. When producing message data, the message producer 10a can first create a data source 10d by itself. The message manager 10b will bind the corresponding message queue 10f according to the data source 10d created by the message producer 10a, and automatically create a target storage area in the corresponding message queue 10b that also corresponds to the data source 10d. The message producer 10a sends the generated message data to the message manager 10b, and the message manager 10b stores the above-received message data in the target storage area in the message queue 10f. As long as the message producer 10a creates a data source 10d, if the middleware type of the message queue 10b is switched later, the message manager 10b will create a new target storage area in the new message queue according to the data source 10d created by the message producer 10a, which is used to store the message data sent to the message manager 10b according to the above-mentioned message producer 10a. In this way, the application of the message producer 10a can be switched seamlessly without modifying the program and configuration on the application side.

[0092] in, Figure 2 The message queue 10f in the distributed system is an important component, which mainly solves the problems of application coupling, asynchronous messages, and traffic peak shaving. In the embodiment of the present application, the message queue 10f is equivalent to a container for storing messages. Its characteristic is that the message data through the message queue 10f can only be advanced in and then out. The message manager 10b first delivers the message data carrying the message generated by the message producer 10a and the target message subject generated according to the data source information to the target storage area in the message queue 10f for storage through the general logical concept of the data source 10d. The message consumer 10c will monitor the data in the message queue 10f. If there is data available for consumption in the message queue 10f, it will also use the general logical concept of the above-mentioned data source 10d to send a message subscription request to the message manager 10b. The message manager 10b will determine the target message subject of the message subscription request from the target storage area in the message queue 10f according to the above-mentioned data source information, pull the message corresponding to the target message subject from the above-mentioned target storage area, and push the above-mentioned message to the message consumer 10c for consumption one by one. Because the performance of the message queue 10f in receiving data is very high, it will not affect the performance of the message producer 10a, and also ensures the timely efficiency of the message consumer 10c. In an optional embodiment of the present application, the message queue 10f is transparent to the message producer 10a and the message consumer 10b, and they are interconnected through the data source 10d. The above-mentioned message queue 10f can be a message queue middleware such as activeMQ, rabbitMQ, rocketMQ, zeroMQ, kafka, etc.

[0093] in, Figure 2The message consumer 10c in the figure is a user terminal that processes data. The message consumer 10c sends a message subscription request carrying data source information to the message manager 10b through the data source shared with the message producer 10a. The message manager 10b determines the message topic subscribed by the above message subscription request from the target storage area of ​​the message queue 10f based on the above data source information, pulls out the message corresponding to the target message topic from the above target storage area, and pushes the above message to the message consumer 10c. After receiving the returned subscription message, the message consumer 10c parses the message mode identification information in the message header information. The above message mode identification information is used to identify the message mode of the message. The message consumer 10c uses the corresponding parsing method to parse and consume the message according to the message mode.

[0094] In some feasible embodiments, if there is no message mode identification information in the message header information, the above-mentioned message consumer 10c can also send a message mode acquisition request carrying the message mode identification to the message manager 10b. After receiving the above request, the message manager 10b obtains the message mode corresponding to the above-mentioned message mode identification from the message registration service center 10e, and sends the above-mentioned message mode to the message consumer 10c. The message consumer 10c parses and consumes the message according to the above-mentioned message mode. In an optional embodiment of the present application, if the message consumer 10c needs to subscribe to the data in a certain data source, the message consumer can introduce an SDK package to call the data source subscription interface. The subscription service provides the function of saving the user's consumption points on the server side. Only simple configuration and processing are required to achieve a high-availability point storage service.

[0095] Among them, in an optional embodiment of the present application, Figure 2The data transmission system based on the message queue may also include a message registration service center 10e, and the message registration service center 10e is used to record the message mode registered by the message manager 10b. In an optional embodiment of the present application, after the message manager 10b receives the message data sent by the message producer 10a, the message manager 10b registers the message mode of the message to the message registration service center 10e, and generates a message mode identifier corresponding to the message mode of the message through the message registration service center 10e, and adds the message mode identifier information to the header message of the message and pushes it together with the message data. When the message manager 10b receives a message mode acquisition request sent by the message consumer 10c and the message mode request carries the message mode identifier, the message manager 10b obtains the message mode corresponding to the message mode identifier from the message registration service center 10e and sends the message mode to the message consumer 10c. Optionally, the message registration service center 10e can be a Spring Cloud component, which is an ordered collection of a series of frameworks, including all aspects of the microservice architecture, and a large number of encapsulations are performed on each component, thereby simplifying development.

[0096] The following will be combined Figure 3-Figure 5 The message queue-based data transmission method provided in the embodiment of the present application is illustrated by way of example. Figure 3 FIG. 1 is a flow chart of a data transmission method based on a message queue provided in an embodiment of the present application. Figure 3 As shown, the data transmission method based on the message queue provided in the embodiment of the present application includes the following steps:

[0097] S101, sending message data carrying data source information.

[0098] In some feasible implementations, the message manager receives message data sent by the message producer, and the message data carries the message generated by the message producer and the data source information associated with the message. Before sending the message data, the message producer first creates a data source, associates the created data source to a certain project, and generates a message and the data source information associated with the message according to the data source. After creating the data source, the message data carrying the message and data source information is sent to the message manager, and the data source information is sent to the message consumer. The message consumer adds the received data source information to the message subscription request and sends it to the message manager together. The message manager stores and pulls messages from the message queue according to the data source. Among them, the data source information includes the data source name and the data source type. Here, the data source name can be composed of all lowercase English letters, underscores, and numbers, such as "source_123". Optionally, the data source created above can be a data source such as "message cache", which is a logical concept and refers to a reusable data source. The project here is a data source set (including one or more data sources) for different purposes created by the user. In other words, one or more data sources can be associated with the same project, and then the data sources, real-time computing, resources and members under each project can be managed in the project management. Optionally, the message data may also include a message mode identifier carried in the header information of the message, and the message mode identifier is used to identify the message mode of the message. The message mode here is used for the message consumer to parse the message based on the message mode and consume it. It can be understood that the data source is used to establish the connection between the message generated by the message producer, the message subject and the data source. At the same time, the data source is also used to connect with the message queue, associate the message producer with the message consumer, and then push the data from the message queue to the real-time stream computing system for downstream computing tasks. After the data source is created, the data source information is added to the message data and sent to the message manager together with the header information of the message. Optionally, the message can be transmitted in Apache Avro format during message transmission, and the SDK can be used to uniformly encapsulate and manage this function. Optionally, the message producer sends message data carrying data source information, which may be a shopping application in the terminal. After the user places an order, the order process completes persistence processing and sends the user's order information to the message manager in the server.

[0099] S102, registering the message mode of the message data.

[0100] In some feasible implementations, after the message manager receives the message data sent by the message producer, the message manager registers the message mode of the above message to the message registration service center, and generates a message mode identifier corresponding to the message mode of the message through the message registration service center, and adds the above message mode identifier to the header message of the message and pushes it together with the message data. When the message consumer sends a message mode acquisition request to the message manager and the above message mode request carries the above message mode identifier, the message manager can obtain the message mode corresponding to the above message mode identifier from the message registration service center and send the above message mode to the message consumer. In an optional embodiment of the present application, the above message manager registers the message mode of the above message data to the message registration service center by registering the above message data to the spring cloud registration center, and generates a message mode identifier corresponding to the message mode of the message through spring cloud.

[0101] S103, generating a target message subject according to the data source information, and storing the message corresponding to the target message subject into a corresponding target storage area in the message queue.

[0102] In some feasible implementations, when the message manager receives the above message data, it will bind the corresponding message queue according to the above data source information, and generate the corresponding target message subject according to the above data source information, and the above data source and the target message subject are in a one-to-one correspondence. Optionally, the name of the target message subject is the project name plus the data source name, such as "program1+source123". It can be understood that the corresponding target message subject can be found in the message queue through the above data source, and the message and the data source can be associated through the correspondence between the target message subject and the message, so that the target message subject in the message queue can be found through the data source, and the message can be found through the target message subject. Therefore, the message producer and the message consumer can find the target message subject in the underlying message queue through the data source, find the message through the target message subject, and realize the production and consumption of data in the corresponding target storage area in the message queue. The above message queue includes at least one storage area including the above target storage area, and one storage area is associated with a message subject. The data source information associated with the message of the same message subject is also the same, and the above storage area is used to store the message of the above message subject. Optionally, when the above message data enters the corresponding storage area in the message queue, data formats such as CSV, JSON, and AVRO can be selected, without the need to set the data dictionary of the message subject separately. When the message producer or message consumer creates a data source, the message manager will operate to bind the corresponding message queue, and in the corresponding message queue, operate to bind a target storage area that is also bound to the data source to store messages of the message subject corresponding to the above data source. Message consumers and message producers can reuse this data source, or they can produce push data or consume pull data in the target storage area corresponding to the above data source. If the message queue type is switched later, the message manager will automatically create and bind a new target storage area in the new message queue according to the data source bound by the message producer. In this way, the application of the message producer can be seamlessly switched. At this time, the middleware such as the message queue is transparent to the message producer and the message consumer. The message producer and the message consumer do not need to care about which specific message queue is used, and they can push and pull data.

[0103] S104: Send a message subscription request carrying data source information.

[0104] In some feasible implementations, the message consumer sends a subscription request carrying data source information to the message manager. Since the message consumer and the message producer share the data source, when the message manager receives the message subscription request carrying the data source information sent by the message consumer, it will first parse the message subscription request and find the message of the corresponding message subject stored in the target storage area corresponding to the data source in the message queue according to the data source information carried by the message subscription request. The data source information includes the data source name, and the name of the message subject corresponding to the data source name is found according to the data source name to determine the message subject subscribed by the message subscription request. In an optional embodiment of the present application, the message consumer requests to subscribe to the message generated by the message producer, which can be the inventory management process in the shopping application in the terminal, and sends the user's order information subscription request to the message manager in the server through the agreed data source. When the inventory management process subscribes to the user's order information, it adopts a pull method to obtain the user's order information, and performs corresponding inventory operations according to the user's order information.

[0105] S105, returning the message in the corresponding storage area pulled according to the above data source information.

[0106] In some feasible implementations, after the message manager finds the name of the message topic corresponding to the above data source name based on the data source name in the above data source information, it pulls the messages subscribed by the message consumer from the target storage area storing the messages of the above message topic, and pushes the above message to the message consumer.

[0107] S106: Obtain a message mode identifier from the header information of the received message.

[0108] In some feasible implementations, when a message consumer receives a message from a message manager and stored in a target storage area in a message queue, a corresponding format parsing method is used for different data formats to parse a message mode identifier from the header information of the message. The message consumer determines the message mode of the message received according to the message subscription request based on the acquired message mode identifier. The message mode here is used by the message consumer to parse the message. If the message mode corresponding to the message generated by the message producer changes, the message consumer directly obtains the changed message mode identifier based on the header information of the message. Optionally, if there is no message mode identifier information in the header information of the message received by the above message consumer, the message consumer can execute step S107.

[0109] S107: Send a message mode request carrying a message mode identifier.

[0110] In some feasible implementations, the message consumer sends a message mode request carrying the message mode identifier to the message manager according to the message mode identifier parsed from the header information of the message.

[0111] S108, obtaining the message mode corresponding to the message mode identifier from the message registration service center.

[0112] In some feasible implementations, the message manager retrieves the corresponding message mode from the message registration service center according to the received message mode identifier, and sends it to the message consumer.

[0113] S109, sending the corresponding message mode obtained from the message registration service center.

[0114] In some feasible implementations, when a message consumer sends a message mode acquisition request to a message manager and the message mode request carries the message mode identifier, the message manager obtains the message mode corresponding to the message mode identifier from the message registration service center and sends the message mode to the message consumer.

[0115] S110, parsing and consuming the message based on the acquired message mode.

[0116] In some feasible implementations, the message consumer uses the corresponding parsing method to parse the message according to the message mode of different messages and then consumes it. The above message mode includes but is not limited to the format type of the message and the type of the message queue. Since the message consumer and the message producer share the common logical concept of data source, the management of the message subject data dictionary is avoided. The downstream message consumer only needs to parse the message mode of the received message according to the corresponding parsing method and then consume the message.

[0117] In this application, the message producer or message consumer creates the data source by themselves, and the message manager uniformly maintains the mapping relationship between the data source and the message subject in the underlying message queue, and the logical relationship between the message queue and the message producer and consumer. Only the abstracted bus data flow model is presented to the message consumer and message producer application side. The message consumer and message producer only need to understand the data source name and the target storage name to generate and consume messages. Here, the data source is used to connect with the message queue, associate the message producer with the message consumer, and then push the data from the message queue to the real-time stream computing system, and then use it for subsequent computing tasks. The message consumer can identify the message subject corresponding to the data source based on the data source information shared with the message producer. There is no need to manage the message subject data dictionary, and can use the corresponding parsing method to parse the message according to the message mode of different messages. The operation is simple and the applicability is high.

[0118] See also Figure 4 , Figure 4 This is another flow chart of a data transmission method based on a message queue provided by the present application. The method is applicable to the message manager side, and the method includes the following steps:

[0119] S201, a message manager receives message data sent by a message producer, wherein the message data carries the message generated by the message producer and data source information associated with the message.

[0120] The specific implementation of step S201 can be found in the above Figure 3 The description of step S101 in the corresponding embodiment will not be repeated here.

[0121] S202: The message manager determines the message queue associated with the data source information, and generates a target message topic according to the data source information.

[0122] The specific implementation of step S202 can be found in the above Figure 3 The description of step S103 in the corresponding embodiment will not be repeated here.

[0123] S203, the message manager determines a target storage area associated with the target message subject in the message queue, and stores the target message subject and the message in the target storage area.

[0124] The specific implementation of step S203 can be found in the above Figure 3 The description of step S103 in the corresponding embodiment will not be repeated here.

[0125] S204, when receiving a message subscription request sent by a message consumer and the message subscription request carries the data source information, the message manager determines the target message topic from the target storage area of ​​the message queue according to the data source information.

[0126] In some feasible implementations, the message consumer and the message producer share the data source, so when the message manager receives the message subscription request sent by the message consumer carrying the data source information, it will first parse the message subscription request and find the message of the corresponding message topic stored in the target storage area corresponding to the data source in the message queue according to the data source information carried by the message subscription request. The data source information includes the data source name, and the name of the message topic corresponding to the data source name is found according to the data source name to determine the message topic subscribed by the message subscription request.

[0127] S205: Pull the message corresponding to the target message topic from the target storage area and push the message to the message consumer.

[0128] In some feasible implementations, after the message manager finds the name of the message topic corresponding to the above data source name based on the data source name in the above data source information, it pulls the messages subscribed by the message consumer from the target storage area storing the messages of the above message topic, and pushes the above message to the message consumer.

[0129] In this application, the message producer can create the data source by itself when generating a message. After the data source is created, the message manager will operate to bind the corresponding message queue component and create a target storage area in the corresponding message queue that corresponds to the data source binding. As long as the message producer is bound to a data source, if the message producer changes the underlying message queue later, when the message manager switches the component type of the message queue, according to the data source bound by the message producer, a new target storage area will be created in the new message queue. The application of the message producer can be seamlessly switched without modifying the program and configuration on the application side. It can simplify the development model of message queue applications and improve the convenience of the message manager in storing data in the message queue and pulling data from the message queue.

[0130] See also Figure 5 , Figure 5 This is a flow chart of a data transmission method based on a message queue provided by the present application. The method is applicable to the message consumer side, and the method includes the following steps:

[0131] S301, a message consumer obtains data source information of a data source created by a message producer.

[0132] In some feasible implementations, the data source created by the message producer can be shared by message consumers. Optionally, the data source can also be created by message consumers for sharing by message producers. The data source information includes but is not limited to the data source name and data source type. In an optional embodiment of the present application, the data source created at this time is not only used as a source input terminal, but also as a sink output terminal, as a data source for various similar real-time computing engines such as flink, and can also be the place where data flows after real-time computing. The real-time computing here mainly refers to real-time computing tasks that are connected to the message queue component, such as flink or spark streaming tasks.

[0133] S302, when the message consumer requests to subscribe to the message generated by the message producer, a message subscription request is sent to the message manager. The message subscription request carries data source information, and the data source information is used to determine the target message topic that the message consumer requests to subscribe to.

[0134] In some feasible implementations, when a message consumer needs to subscribe to a message, the data source information obtained for determining the target message topic is sent to the message manager along with the message subscription request. The correspondence between the data source information and the target message topic here is specifically referred to in step S103, which will not be described in detail here.

[0135] S303, the message consumer obtains message mode identification information from the header information of the message, and determines the message mode of the message according to the message mode identification information.

[0136] In some feasible implementations, when a message consumer receives a message from a message manager and stored in a target storage area in a message queue, a corresponding format parsing method is used for different data formats to parse the message mode identification information from the header information of the message, and the above message mode identification information includes but is not limited to the message mode identification, message mode, etc. The message consumer determines which message mode the message received according to the message subscription request belongs to based on the acquired message identification mode information. If the message mode corresponding to the message generated by the message producer changes, the message consumer directly obtains the changed message mode based on the header information of the message, and parses and consumes the changed message based on the received message mode.

[0137] S304: Parse and consume the message based on the message mode of the message.

[0138] The specific implementation of step S304 can be found in the above Figure 3 The description of step S109 in the corresponding embodiment will not be repeated here.

[0139] In this application, the message producer or message consumer creates the data source by himself, and the message manager uniformly maintains the mapping relationship between the data source and the message subject in the underlying message queue, and the logical relationship between the message queue and the message producer and message consumer. Only the abstracted bus data flow model is presented to the message consumer side. The message consumer and message producer only need to understand the data source name and the target storage name to generate and consume messages. Here, the data source is used to connect with the message queue, associate the message producer with the message consumer, and then push the data from the message queue to the real-time stream computing system, and then use it for subsequent computing tasks. The message consumer can identify the message subject corresponding to the data source based on the data source information shared with the message producer. There is no need to manage the message subject data dictionary, and can use the corresponding parsing method to parse the message according to the message mode of different messages. The operation is simple and the applicability is high.

[0140] For further information, see Figure 6 , Figure 6 This is a structural diagram of a message queue-based data transmission device provided by the present application. The message queue-based data transmission device can be a user terminal corresponding to a message manager, or a computer program (including program code) running in a computer device. For example, the message queue-based data transmission device is an application software; the message queue-based data transmission device can be used to execute the corresponding steps in the method provided by the present application. Figure 6 As shown, the data transmission device 1 based on the message queue can be applied to a computer device. The data transmission device 1 based on the message queue can include: a receiving module 11, a determining module 12, a message pushing module 13, and a registration module 14.

[0141] The receiving module 11 is used to receive message data sent by the message producer, wherein the message data carries the message generated by the message producer and the data source information associated with the message;

[0142] A determination module 12, used to determine the message queue associated with the data source information received by the message data receiving module, and generate a target message topic according to the data source information;

[0143] The determination module 12 is further configured to determine a target storage area associated with the target message subject in the message queue, and store the target message subject and the message in the target storage area;

[0144] The receiving module 11 is further used to receive a message subscription request sent by a message consumer, wherein the message subscription request carries the data source information;

[0145] The message push module 13 is used to determine the target message topic from the target storage area of ​​the message queue according to the data source information carried in the message subscription request received by the receiving module, pull the message corresponding to the target message topic from the target storage area and push the message to the message consumer.

[0146] In a possible implementation, the message queue includes at least one storage area including the target storage area, one storage area is associated with one message subject and is used to store messages of the message subject, and messages of the same message subject are associated with the same data source information.

[0147] In a possible implementation, the data source information includes a data source name of a data source to which the message belongs and a project name of a project to which the data source belongs, and the project to which the data source belongs includes at least one data source including the data source; the determination module is used to:

[0148] A target message subject is generated according to the project name and the data source name, wherein one data source name corresponds to one message subject.

[0149] In a possible implementation manner, the message queue-based message transmission device further includes:

[0150] The registration module 14 is used to register the above message mode to the message registration service center, and generate a message mode identifier corresponding to the message mode of the above message through the message registration service center.

[0151] In a possible implementation manner, the message push module 13 is further used for:

[0152] After receiving the message mode acquisition request sent by the above-mentioned message consumer and the above-mentioned message mode acquisition request carries the above-mentioned message mode identifier, the above-mentioned message mode corresponding to the above-mentioned message mode identifier is obtained from the above-mentioned message registration service center and the above-mentioned message mode is sent to the above-mentioned message consumer.

[0153] The specific implementation methods of the receiving module 11, the determining module 12, the pushing module 13, and the registering module 14 can be referred to in the above Figure 3 The steps S101-S105 in the corresponding embodiment will not be described in detail here.

[0154] In this application, when a message producer generates a message, it can create a data source by itself. After the data source is created, the message manager will operate the corresponding message queue component to bind it, and create a target storage area in the corresponding message queue that corresponds to the data source binding relationship. Message consumers and message producers can reuse this data source, or they can produce and consume data or consume and pull data in the target storage area corresponding to the data source. At this time, the message queue is transparent to both message producers and message consumers. The message producer does not need to care about what component the middleware such as the specific message queue used is. They are associated through the data source. As long as the message producer is bound to a certain data source, if the message producer changes the underlying message queue later, the message manager will create a new target storage area in the new message queue according to the data source bound by the message producer when the component type of the message queue is switched. The application of the message producer can be seamlessly switched without modifying the program and configuration of the application end. It can reduce the cost of message producers or message consumers to learn different types of message queues, improve the convenience of storing data in message queues and pulling data from message queues, and improve the efficiency of data transmission.

[0155] Please also see Figure 7 , is a schematic diagram of the structure of another data transmission device based on a message queue provided by an embodiment of the present invention. Figure 7 As shown, the data transmission device 2 based on the message queue may be a user terminal corresponding to a message consumer, and the data transmission device 2 includes: an acquisition module 21 , a sending module 22 , a receiving module 23 , and a parsing and consuming module 24 .

[0156] An acquisition module 21 is used to acquire data source information of a data source created by a message producer;

[0157] The sending module 22 is used to send a message subscription request to the message manager when subscribing to the message generated by the message producer. The message subscription request carries data source information, and the data source information is used to determine the target message subject of the subscription request;

[0158] A receiving module 23, configured for the message consumer to receive the message associated with the target message topic from the message manager, parse and consume the message;

[0159] See also Figure 7 In a possible implementation manner, the above-mentioned parsing and consumption module 24 is used to:

[0160] Obtaining a message mode identifier from the header information of the message received by the receiving module, and sending a message mode acquisition request to the message manager through the sending module, wherein the message mode acquisition request carries the message mode identifier;

[0161] The message mode corresponding to the message mode identifier is obtained from the message manager, and the message is parsed and consumed based on the message mode.

[0162] In a possible implementation manner, the above-mentioned parsing and consumption module 24 is further used for:

[0163] Obtaining a message mode identifier from the header information of the message received by the receiving module, and sending a message mode acquisition request to the message manager through the sending module, wherein the message mode acquisition request carries the message mode identifier;

[0164] The message mode corresponding to the message mode identifier is obtained from the message manager, and the message is parsed and consumed based on the message mode.

[0165] The specific functional implementation of the acquisition module 21, the sending module 22, the receiving module 23, and the parsing and consumption module 24 can be found in the above Figure 5 The steps S301-S304 in the corresponding embodiment are not described in detail here.

[0166] Please also see Figure 8 , is a schematic diagram of the structure of another data transmission device based on a message queue provided by an embodiment of the present invention. Figure 8 As shown, the data transmission device 3 based on the message queue may be a user terminal corresponding to a message producer, and the data transmission device 3 includes: a creation module 31 , a message generation module 32 , and a sending module 33 .

[0167] A creation module 31, used to create a data source;

[0168] A message generating module 32, generating a message and data source information associated with the message according to the data source created by the creating module;

[0169] The sending module 33 is used to send message data to the message manager, wherein the message data carries the message generated by the message generating module and the data source information.

[0170] See also Figure 8 In a feasible implementation manner, the sending module 33 is further used for:

[0171] Sending data source information of the above data source to a message manager and / or a message consumer;

[0172] The above data source information includes the data source name and the project name of the project to which the data source belongs.

[0173] For further information, see Fig. 9 , Fig. 9 Schematic diagram of a computer device provided by this application. Fig. 9 As shown, the computer device 1000 can be suitable for a user terminal of a message manager, and the computer device 1000 may include: at least one processor 1001, such as a CPU, at least one network interface 1004, a transceiver 1003, a memory 1005, and at least one communication bus 1002. The communication bus 1002 is used to realize the connection and communication between these components. The network interface 1004 may optionally include a standard wired interface and a wireless interface (such as a WI-FI interface). The memory 1005 may be a high-speed random access memory (RAM) memory, or a non-volatile memory (non-volatile memory), such as at least one disk storage. The memory 1005 may optionally also be at least one storage device located away from the aforementioned processor 1001. As Fig. 9 As shown, the memory 1005 as a computer storage medium may include an operating system, a network communication module, a user interface module, and a device control application program.

[0174] exist Fig. 9 In the computer device 1000 shown, the network interface 1004 is mainly used for the user terminal to perform network communication, and the processor 1001 and the transceiver 1003 can be used to call the device control application stored in the memory 1005 to perform the following operations:

[0175] The transceiver 1003 is used to receive message data sent by a message producer, the message data carrying the message generated by the message producer and the data source information associated with the message;

[0176] The processor 1001 is used to determine the message queue associated with the data source information and generate a target message topic according to the data source information;

[0177] The processor 1001 is further configured to determine a target storage area associated with a target message subject in the message queue, and store the target message subject and the message in the target storage area;

[0178] When the transceiver 1003 receives a message subscription request sent by a message consumer and the message subscription request carries data source information, the processor 1001 is also used to determine the target message topic from the target storage area of ​​the message queue according to the data source information, pull the message corresponding to the target message topic from the target storage area and push the message to the message consumer.

[0179] In some feasible implementations, the message queue includes at least one storage area including the target storage area, one storage area is associated with one message subject and is used to store messages of the message subject, and messages of the same message subject are associated with the same data source information.

[0180] In some feasible implementations, the above-mentioned data source information includes the data source name of the data source to which the above-mentioned message belongs and the project name of the project to which the above-mentioned data source belongs, and the project to which the above-mentioned data source belongs includes at least one data source including the above-mentioned data source; the above-mentioned processor 1001 is also used to generate a target message subject according to the above-mentioned project name and the above-mentioned data source name, wherein one data source name corresponds to one message subject.

[0181] In some feasible implementations, the processor 1001 is further configured to register the message mode with a message registration service center, and generate a message mode identifier corresponding to the message mode of the message through the message registration service center;

[0182] The transceiver 1003 is also used to obtain the message mode corresponding to the message mode identifier from the message registration service center and send the message mode to the message consumer after receiving the message mode acquisition request sent by the message consumer and the message mode acquisition request carries the message mode identifier.

[0183] It should be understood that the computer device 1000 described in the present application can execute the above Figure 4 The description of the data transmission method based on the message queue in the corresponding embodiment can also be performed as described above. Figure 6 The description of the data transmission device 1 based on the message queue in the corresponding embodiment will not be repeated here. In addition, the description of the beneficial effects of the same method will not be repeated here either.

[0184] In addition, it should be pointed out here that: the present application also provides a computer-readable storage medium, and the computer-readable storage medium stores a computer program executed by the message queue-based data transmission device 1 mentioned above, and the computer program includes program instructions. When the processor executes the program instructions, it can execute the above Figure 4 The description of the message queue-based data transmission method in the corresponding embodiment will therefore not be repeated here.

[0185] For further information, see Fig.10 FIG. 1 is a schematic diagram of the structure of another computer device provided by an embodiment of the present invention. Fig.10As shown, the computer device 2000 can be applicable to a user terminal of a message consumer, and the computer device 2000 may include: at least one processor 2001, such as a CPU, at least one network interface 2004, a transceiver 2003, a memory 2005, and at least one communication bus 2002. The communication bus 2002 is used to realize the connection and communication between these components. The network interface 2004 may optionally include a standard wired interface and a wireless interface (such as a WI-FI interface). The memory 2005 may be a high-speed random access memory (RAM) memory, or a non-volatile memory (non-volatile memory), such as at least one disk storage. The memory 2005 may optionally also be at least one storage device located away from the aforementioned processor 2001. As Fig.10 As shown, the memory 2005 as a computer storage medium may include an operating system, a network communication module, a user interface module, and a device control application program.

[0186] exist Fig.10 In the computer device 2000 shown, the network interface 2004 is mainly used for the user terminal to perform network communication; and the processor 2001 and the transceiver 2003 can be used to call the device control application stored in the memory 2005 to perform the following operations:

[0187] The transceiver 2003 is used to obtain the data source information of the data source created by the message producer;

[0188] The transceiver 2003 is also used to send a message subscription request to the message manager when requesting to subscribe to the message generated by the message producer. The message subscription request carries data source information, and the data source information is used to determine the target message topic that the message consumer requests to subscribe to.

[0189] When the transceiver 2003 receives the message associated with the target message topic from the message manager, the processor 2001 is used to parse and consume the message.

[0190] In a feasible implementation manner, the transceiver 2003 is further configured to obtain message mode identification information from the header information of the message;

[0191] The processor 2001 is further configured to determine a message mode of the message according to the message mode identification information, and parse and consume the message based on the message mode of the message.

[0192] In a feasible implementation manner, the transceiver 2003 is further used to obtain a message mode identifier from the header information of the message, and send a message mode acquisition request to a message manager, wherein the message mode acquisition request carries the message mode identifier; and obtain a message mode corresponding to the message mode identifier from the message manager;

[0193] The processor 2001 is also used to parse and consume the message based on the message mode.

[0194] It should be understood that the computer device 2000 described in this application can execute the above Figure 5 The description of the data transmission method based on the message queue in the corresponding embodiment can also be performed as described above. Figure 7 The description of the data transmission device 2 based on the message queue in the corresponding embodiment will not be repeated here. In addition, the description of the beneficial effects of the same method will not be repeated here either.

[0195] In addition, it should be pointed out here that: the present application also provides a computer-readable storage medium, and the computer-readable storage medium stores a computer program executed by the message queue-based data transmission device 2 mentioned above, and the computer program includes program instructions. When the processor executes the program instructions, it can execute the above Figure 5 The description of the message queue-based data transmission method in the corresponding embodiment will therefore not be repeated here.

[0196] For further information, see Fig.11 FIG. 1 is a schematic diagram of the structure of another computer device provided by an embodiment of the present invention. Fig.11 As shown, the computer device 3000 can be suitable for a user terminal of a message producer, and the computer device 3000 may include: at least one processor 3001, such as a CPU, at least one network interface 3004, a transceiver 3003, a memory 3005, and at least one communication bus 3002. The communication bus 3002 is used to realize the connection and communication between these components. The network interface 3004 may optionally include a standard wired interface and a wireless interface (such as a WI-FI interface). The memory 3005 may be a high-speed random access memory (RAM) memory, or a non-volatile memory (non-volatile memory), such as at least one disk storage. The memory 3005 may optionally also be at least one storage device located away from the aforementioned processor 3001. As Fig.11 As shown, the memory 3005 as a computer storage medium may include an operating system, a network communication module, a user interface module, and a device control application program.

[0197] exist Fig.11 In the computer device 3000 shown, the network interface 3004 is mainly used for the user terminal to perform network communication; and the processor 3001 and the transceiver 3003 can be used to call the device control application stored in the memory 3005 to perform the following operations:

[0198] The processor 3001 is configured to create a data source, and generate a message and data source information associated with the message according to the data source;

[0199] The transceiver 3003 is used to send message data to the message manager, and the message data carries the message and the data source information.

[0200] In a feasible implementation, the transceiver 3003 is also used to send data source information of the data source to a message manager and / or a message consumer after the processor 3001 creates the data source, wherein the data source information includes the data source name and the project name of the project to which the data source belongs.

[0201] It should be understood that the computer device 3000 described in this application can execute the above Figure 3 The description of the data transmission method based on the message queue in the corresponding embodiment can also be performed as described above. Figure 8 The description of the data transmission device 3 based on the message queue in the corresponding embodiment will not be repeated here. In addition, the description of the beneficial effects of adopting the same method will not be repeated here either.

[0202] In addition, it should be pointed out here that: the present application also provides a computer-readable storage medium, and the computer-readable storage medium stores a computer program executed by the message queue-based data transmission device 3 mentioned above, and the computer program includes program instructions. When the processor executes the program instructions, it can execute the above Figure 3 The description of the message queue-based data transmission method in the corresponding embodiment will therefore not be repeated here.

[0203] In addition, the description of the beneficial effects of the same method will not be repeated. For technical details not disclosed in the computer-readable storage medium embodiment involved in this application, please refer to the description of the method embodiment of this application. As an example, program instructions can be deployed to be executed on one computing device, or on multiple computing devices located at one location, or on multiple computing devices distributed at multiple locations and interconnected by a communication network. Multiple computing devices distributed at multiple locations and interconnected by a communication network can constitute a blockchain system.

[0204] Those skilled in the art can understand that all or part of the processes in the above-mentioned embodiments can be completed by instructing the relevant hardware through a computer program. The above-mentioned program can be stored in a computer-readable storage medium. When the program is executed, it can include the processes of the embodiments of the above-mentioned methods. Among them, the above-mentioned computer-readable storage medium can be a data transmission device based on a message queue provided in any of the above-mentioned embodiments or an internal storage unit of the above-mentioned device, such as a hard disk or memory of an electronic device. The computer-readable storage medium can also be an external storage device of the electronic device, such as a plug-in hard disk, a smart memory card (smart media card, SMC), a secure digital (secure digital, SD) card, a flash card (flash card), etc. equipped on the electronic device. The above-mentioned computer-readable storage medium can also include a disk, an optical disk, a read-only memory (ROM) or a random access memory, etc. Further, the computer-readable storage medium can also include both an internal storage unit of the electronic device and an external storage device. The computer-readable storage medium is used to store the computer program and other programs and data required by the electronic device. The computer-readable storage medium can also be used to temporarily store data that has been output or is to be output.

[0205] The terms "including" and "having" and any variations thereof in the claims, specification and drawings of the present invention are intended to cover non-exclusive inclusions. For example, a process, method, system, product or device comprising a series of steps or units is not limited to the listed steps or units, but may optionally include steps or units that are not listed, or may optionally include other steps or units that are inherent to these processes, methods, products or devices. Reference to "embodiments" herein means that specific features, structures or characteristics described in conjunction with the embodiments may be included in at least one embodiment of the present invention. Displaying the phrase at various locations in the specification does not necessarily refer to the same embodiment, nor is it an independent or alternative embodiment that is mutually exclusive with other embodiments. It is explicitly and implicitly understood by those skilled in the art that the embodiments described herein may be combined with other embodiments. The term "and / or" used in the specification and the appended claims of the present invention refers to any combination of one or more of the associated listed items and all possible combinations, and includes these combinations.

[0206] Those skilled in the art will appreciate that the units and algorithm steps of each example described in conjunction with the embodiments disclosed herein can be implemented by electronic hardware, computer software, or a combination of the two. In order to clearly illustrate the interchangeability of hardware and software, the composition and steps of each example have been generally described in terms of function in the above description. Professional and technical personnel may use different methods to implement the described functions for each specific application, but such implementation should not be considered to be beyond the scope of the present invention.

[0207] The above disclosure is only the preferred embodiment of the present application, which certainly cannot be used to limit the scope of rights of the present application. Therefore, equivalent changes made according to the claims of the present application are still within the scope covered by the present application.

Claims

1. A data transmission method based on a message queue, characterized in that: The method comprises: The message manager receives the message data sent by the message producer, wherein the message data carries the message generated by the message producer and the data source information associated with the message. The message producer is a user terminal that generates and stores messages in a message queue. The data source information includes the data source name of the data source to which the message belongs and the project name of the project to which the data source belongs. The project to which the data source belongs includes at least one data source including the data source, and the data source is used to connect with the message queue. The message manager determines the message queue associated with the data source information, and generates a target message topic according to the data source information; The message manager determines a target storage area associated with the target message subject in the message queue, and stores the target message subject and the message in the target storage area; When receiving a message subscription request sent by a message consumer and the message subscription request carries the data source information, the message manager determines the target message topic from the target storage area of ​​the message queue according to the data source information, pulls the message corresponding to the target message topic from the target storage area and pushes the message to the message consumer, where the message consumer is a user terminal that subscribes to and consumes messages stored in the message queue.

2. The method according to claim 1, characterized in that The message queue includes at least one storage area including the target storage area. One storage area is associated with one message subject and is used to store messages of the message subject. Messages of the same message subject are associated with the same data source information.

3. The method according to claim 2, characterized in that Generating a target message topic according to the data source information includes: A target message subject is generated according to the project name and the data source name, wherein one data source name corresponds to one message subject.

4. The method according to any one of claims 1 to 3, characterized in that: After the message manager receives the message data sent by the message producer, the method further includes: The message manager registers the message mode of the message with a message registration service center, and generates a message mode identifier corresponding to the message mode of the message through the message registration service center; When a message mode acquisition request sent by the message consumer is received and the message mode acquisition request carries the message mode identifier, the message mode corresponding to the message mode identifier is acquired from the message registration service center and the message mode is sent to the message consumer.

5. A data transmission method based on a message queue, characterized in that: The method comprises: The message consumer obtains the data source information of the data source created by the message producer. The message producer is a user terminal that generates and stores messages in a message queue. The message consumer is a user terminal that subscribes to and consumes messages stored in the message queue. The data source information includes the data source name of the data source to which the message belongs and the project name of the project to which the data source belongs. The project to which the data source belongs includes at least one data source including the data source. The data source is used to connect to the message queue. When the message consumer requests to subscribe to the message generated by the message producer, a message subscription request is sent to the message manager, wherein the message subscription request carries the data source information, and the data source information is used to determine the target message topic that the message consumer requests to subscribe to; The message consumer receives the message associated with the target message topic from the message manager, and parses and consumes the message.

6. The method according to claim 5, characterized in that The parsing and consuming the message includes: The message consumer obtains message mode identification information from the header information of the message, and determines the message mode of the message according to the message mode identification information; The message is parsed and consumed based on the message mode of the message.

7. The method according to claim 5, characterized in that The parsing and consuming the message includes: The message consumer obtains a message mode identifier from the header information of the message, and sends a message mode acquisition request to the message manager, wherein the message mode acquisition request carries the message mode identifier; The message mode corresponding to the message mode identifier is obtained from the message manager, and the message is parsed and consumed based on the message mode.

8. A data transmission method based on a message queue, characterized in that: The method comprises: The message producer creates a data source, and generates a message and data source information associated with the message according to the data source. The message producer is a user terminal that generates and stores messages in a message queue. The data source information includes the data source name of the data source to which the message belongs and the project name of the project to which the data source belongs. The project to which the data source belongs includes at least one data source including the data source. The data source is used to connect to the message queue. The message producer sends message data to the message manager, and the message data carries the message and the data source information.

9. A computer device, characterized in that: include: transceivers, processors, memory, and network interfaces; The processor is connected to the transceiver, the memory, and a network interface, the network interface is used to provide a data communication function, the memory is used to store program code, and the transceiver and the processor are used to call the program code to execute the method described in any one of claims 1-8.

10. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores a computer program, wherein the computer program includes program instructions. When the program instructions are executed by a processor, the method according to any one of claims 1 to 8 is executed.

Images