Message consumption method, device, computer equipment and storage medium

Abstract

The present application relates to a message consumption method, device, computer equipment and storage medium. The method comprises: obtaining corresponding message consumption rules from a pre-stored consumption rule configuration file according to the name of a data source; the message consumption rules include the start and end time of the message to be consumed, and the start and end time are set according to the estimated time when the message arrives at the message middleware; according to the message consumption rules, searching the message middleware for the message that arrives at the message middleware within the start and end time, and consuming it; storing the consumed message in a target file, and setting the start and end time as the name of the target file. The use of this method can improve the efficiency of troubleshooting missed messages.

Description

Technical Field

[0001] The present application relates to the field of communication technology, and in particular to a message consumption method, apparatus, computer equipment and storage medium. Background Art

[0002] Message middleware includes message production and message consumption. The function of the message production end is to provide messages, and the function of the message consumption end is to connect with the message production end and output messages. The message is text data of some numbers, letters and symbols. Message middleware uses efficient and reliable message transmission mechanism to exchange data regardless of platform. There is a corresponding message consumption end for each platform. Message middleware integrates distributed systems based on data communication. By providing message transmission and message queuing models, it can expand communication between processes in a distributed environment.

[0003] However, when a message is missed, IT operation and maintenance personnel usually need to check historical messages (i.e., check whether the message is missed because the sender did not send the message or the receiver did not receive the message) from the first message of the message middleware until the last message is consumed, and store the consumed messages in a text file. The IT operation and maintenance personnel then search for the missed message from the text file. Since the message consumer cannot be interrupted midway, the number of messages in the message middleware is huge, resulting in a huge capacity of text files used to store messages, which can reach tens of G bytes to hundreds of G bytes. Finding a specific message to be checked from this large file is like looking for a needle in a haystack, and the hardware performance of the computer is very high when searching for messages in a large file, so the existing efficiency of checking missed messages is very low. Summary of the invention

[0004] Based on this, it is necessary to provide a message consumption method, device, computer equipment and storage medium that can improve the efficiency of message exclusion in response to the above technical problems.

[0005] A message consumption method, the method comprising:

[0006] Obtain corresponding message consumption rules from a pre-stored consumption rule configuration file according to the name of the data source; the message consumption rules include the start and end time of the message to be consumed, and the start and end time are set according to the estimated time when the message arrives at the message middleware;

[0007] According to the message consumption rule, searching the message middleware for the message that arrives at the message middleware within the start and end time, and consuming the message;

[0008] The consumed message is stored in a target file, and the start and end times are set as the name of the target file.

[0009] In one of the embodiments, the step of obtaining corresponding message consumption rules from a pre-stored consumption rule configuration file according to the name of the data source includes: obtaining a message input application interface, a message output application interface, and an executable file for processing the input and output of the message from the pre-stored consumption rule configuration file according to the name of the data source; wherein the message input application interface, the message output application interface, and the executable file for processing the input and output of the message are configured with corresponding message consumption rules; wherein the message output application interface is configured with the start and end time of the message to be consumed.

[0010] In one of the embodiments, the application program interface for message output is configured with a naming method for files storing the messages, a single file size for storing the messages, and a directory for files storing the messages; the step of storing the consumed messages in a target file and setting the start and end times as the names of the target files includes: storing the consumed messages in the target file according to the naming method for files storing the messages, a single file size for storing the messages, and a directory for files storing the messages; the naming method for the files storing the messages includes setting the start and end times as the names of the target files.

[0011] In one of the embodiments, the application program interface into which the message enters is configured with the communication protocol of the message to be consumed; the step of searching from the message middleware for the message that arrives at the message middleware within the start and end time according to the message consumption rule and consuming the message comprises: communicating with the corresponding message middleware according to the communication protocol of the message to be consumed; searching from the message middleware for the message that arrives at the message middleware within the start and end time according to the message consumption rule and consuming the message.

[0012] In one of the embodiments, the steps of searching the message middleware for messages that arrive at the message middleware within the start and end time according to the message consumption rule and consuming the messages include: searching the message middleware for the position of the message that starts consumption and the position of the message that ends consumption according to the message consumption rule; sorting the messages in the message middleware in the order of the time they arrive at the message middleware; and consuming the messages between the position of the message that starts consumption and the position of the message that ends consumption in sequence.

[0013] In one of the embodiments, after the step of sequentially consuming the messages between the position of the message where consumption starts and the position of the message where consumption ends, it includes: recording the size of each of the consumed messages and / or counting the number of the consumed messages.

[0014] In one of the embodiments, the step of storing the consumed messages into a target file and setting the start and end times as the name of the target file includes: storing the consumed messages into a text file according to a preset file size or a preset number of messages, the text file stores messages of a preset file size or a preset number of messages, and setting the start and end times as the name of the text file.

[0015] A message consumption device, the device comprising:

[0016] A message consumption rule acquisition module is used to acquire corresponding message consumption rules from a pre-stored consumption rule configuration file according to the name of the data source; the message consumption rules include the start and end time of the message to be consumed, and the start and end time are set according to the estimated time when the message arrives at the message middleware;

[0017] A consumption module, used to search the message middleware for messages arriving at the message middleware within the start and end time according to the message consumption rule, and consume the messages;

[0018] The storage module is used to store the consumed message into a target file and set the start and end time as the name of the target file.

[0019] A computer device comprises a memory and a processor, wherein the memory stores a computer program, and when the processor executes the computer program, the following steps are implemented:

[0020] Obtain corresponding message consumption rules from a pre-stored consumption rule configuration file according to the name of the data source; the message consumption rules include the start and end time of the message to be consumed, and the start and end time are set according to the estimated time when the message arrives at the message middleware;

[0021] According to the message consumption rule, searching the message middleware for the message that arrives at the message middleware within the start and end time, and consuming the message;

[0022] The consumed message is stored in a target file, and the start and end times are set as the name of the target file.

[0023] A computer-readable storage medium stores a computer program, which, when executed by a processor, implements the following steps:

[0024] Obtain corresponding message consumption rules from a pre-stored consumption rule configuration file according to the name of the data source; the message consumption rules include the start and end time of the message to be consumed, and the start and end time are set according to the estimated time when the message arrives at the message middleware;

[0025] According to the message consumption rule, searching the message middleware for the message that arrives at the message middleware within the start and end time, and consuming the message;

[0026] The consumed message is stored in a target file, and the start and end times are set as the name of the target file.

[0027] The above-mentioned message consumption method, device, computer equipment and storage medium configure corresponding message consumption rules for each data source through pre-stored consumption rule configuration files, so that the consumption of messages can be integrated into one system, reducing the development workload of configuring message consumption tools according to different data sources; at the same time, the consumed messages are stored in the target file according to the start and end time, which can narrow the scope of investigation of missed messages and improve the efficiency of investigation of missed messages. BRIEF DESCRIPTION OF THE DRAWINGS

[0028] Figure 1 An application environment diagram of a message consumption method in an embodiment;

[0029] Figure 2 A schematic diagram of a flow chart of a message consumption method in an embodiment;

[0030] Figure 3 A flowchart of the steps of searching for messages within a start and end time for consumption in one embodiment;

[0031] Figure 4 is a structural block diagram of a message consumption device in one embodiment;

[0032] Figure 5 A structural block diagram of a message consumption system in one embodiment;

[0033] Figure 6 FIG. 4 is a diagram showing the internal structure of a computer device in one embodiment. DETAILED DESCRIPTION

[0034] In order to make the purpose, technical solution and advantages of the present application more clearly understood, the present application is further described in detail below in conjunction with the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are only used to explain the present application and are not used to limit the present application.

[0035] The message consumption method provided in this application can be applied to Figure 1In the application environment shown. Among them, the first platform terminal 102 and the second platform terminal 106 communicate with the message middleware server 104 through the network. The first platform terminal 102 obtains the corresponding message consumption rules from the pre-stored consumption rule configuration file according to the name of the data source; the message consumption rules include the start and end time of the message to be consumed, and the start and end time are set according to the estimated time when the message arrives at the message middleware server 104; according to the message consumption rules, the message middleware server 104 searches for the message that arrives at the message middleware server 104 within the start and end time, and consumes it; the consumed message is stored in the target file, and the start and end time is set as the name of the target file; wherein the second platform terminal 106 sends the message to the message middleware server 104. Through the method described in this embodiment, the missed messages transmitted from the second platform terminal 106 to the first platform terminal 102 can be checked.

[0036] The first platform terminal 102 and the second platform terminal 106 may be, but are not limited to, various personal computers, laptops, smart phones, tablet computers, and portable wearable devices, and the server 104 may be implemented as an independent server or a server cluster consisting of multiple servers.

[0037] In one embodiment, Figure 2 As shown, a message consumption method is provided, which is applied to Figure 1 Taking the first platform terminal 102 in the example as an example, the following steps are included:

[0038] Step S110, obtaining corresponding message consumption rules from a pre-stored consumption rule configuration file according to the name of the data source; the message consumption rules include the start and end time of the message to be consumed, and the start and end time are set according to the estimated time when the message arrives at the message middleware.

[0039] The data source includes data in message middleware such as ActiveMQ, RabbitMQ, Kafka, RocketMQ, etc. The name of the data source is the name of the message middleware. The pre-stored consumption rule configuration file sets corresponding message consumption rules for each data source. The message consumption rules include: the application program interface for message input, the application program interface for message output, and the executable file for processing the input and output of the message; wherein, the executable file for processing the input and output of the message is the jar package that the custom message source depends on. The message is text data composed of some numbers, letters or symbols.

[0040] The start and end time of the consumed message is set according to the estimated time when the message arrives at the message middleware. The message middleware receives the message and stores the received message with a number. The message middleware may receive multiple messages within one second, that is, multiple messages may arrive at the message middleware at the same time. At this time, the time when these multiple messages arrive at the message middleware is the same. For example, if messages A, B, and C arrive at message middleware S at 12:30:15 Beijing time, then the time when messages A, B, and C arrive at message middleware S is 12:30:15 Beijing time. The message middleware stores messages A, B, and C according to 12:30:15 Beijing time. Of course, messages A, B, and C have corresponding numbers. If the message middleware estimates First, 1000 messages are stored, and the previously stored messages have been assigned numbers 1-1000. Then messages A, B, and C continue to be numbered according to the already assigned numbers, so messages A, B, and C can be numbered 1001, 1002, and 1003. The order of messages A, B, and C can also be messages B, A, and C, and the corresponding numbers of messages B, A, and C can be 1001, 1002, and 1003; and so on, messages A, B, and C can also be in other orders, and the corresponding numbers of messages A, B, and C in other orders can be 1001, 1002, and 1003.

[0041] Among them, the estimated time is the time when the message middleware notifies the message consumer (the first platform terminal 102) that the message has arrived. Since the time when the message actually arrives at the message middleware deviates from the estimated time, the start and end time is a time period including the estimated time. By searching for the message within the start and end time periods, it can be ensured that the message can be found even when there is a deviation from the estimated time. Of course, if the message does not exist in the message middleware and the message cannot be found, it is judged that the message middleware has missed the message.

[0042] Step S120: According to the message consumption rule, the message arriving at the message middleware within the start and end time is searched from the message middleware, and the message is consumed.

[0043] Among them, the process of message consumption is a process of taking messages from the message middleware. The message middleware will save the log of message consumption and the process of message consumption can be restored through the log.

[0044] Specifically, during the message consumption process of the message middleware, the terminal controls the start time and end time of message consumption through a timer, counts the consumed messages through a counter, and obtains the location of the message that starts consumption and the location of the message that ends consumption through a message location acquisition module.

[0045] In one embodiment, a timer controls the start and end time of acquiring a message, wherein each message has a time of arriving at the message middleware, and the start and end time of the acquired message is set according to the time when the message arrives at the message middleware. For example, if it is known that the time of missing a message (a message that needs to be checked) is about 9:00, then the start and end time of acquiring a message can be set to 9:00-10:00, and the missed message can be checked from the time period of 9:00-10:00. Among them, a counter controls the number and byte size of consumed messages. For example, for a whole segment of acquired messages, if the bytes are too large and not convenient for viewing, it is divided into multiple message segments as needed and the byte size of each message segment is set. Among them, a message position acquisition module controls the position of the acquired message in the middleware. For example, there are multiple messages in the message middleware, and each message has a corresponding position in the message middleware. The message position acquisition module can calculate the position of the message to be acquired in the message middleware in combination with the start and end time.

[0046] Step S130, storing the consumed message into a target file, and setting the start and end times as the name of the target file.

[0047] The target file may be a text file, through which the consumed messages can be conveniently viewed. Of course, the target file may also be a table file or other files.

[0048] In the above message consumption method, corresponding message consumption rules are set for each data source through the pre-stored consumption rule configuration file, which can integrate the consumption of messages into one system and reduce the development workload of configuring message consumption tools according to different data sources; at the same time, the consumed messages are stored in the target file according to the start and end time, which can narrow the scope of the missed message investigation and improve the efficiency of the missed message investigation.

[0049] In one of the embodiments, the step S110 includes: obtaining an application program interface for message input, an application program interface for message output, and an executable file for processing the input and output of the message from a pre-stored consumption rule configuration file according to the name of the data source; wherein the application program interface for message input, the application program interface for message output, and the executable file for processing the input and output of the message are configured with corresponding message consumption rules; wherein the application program interface for message output is configured with the start and end time of the message to be consumed.

[0050] Among them, the application program interface for message entry is mainly used to define a set of APIs (Application Programming Interface) for data source entry processing. The application program interface is a set of definitions, programs and protocols, and mutual communication between computer software is realized through the API. Among them, when consuming messages of various different message middleware, it is necessary to communicate with the message middleware according to the corresponding API. The consumption rule configuration file pre-stored in this embodiment can realize communication with multiple message middleware according to the type of message middleware and the corresponding configuration API.

[0051] For example, the message incoming application program interface is used to obtain the TCP connection of the incoming message. TCP (Transmission Control Protocol) is a connection-oriented, reliable, byte stream-based transport layer communication protocol.

[0052] The application programming interface for message output is used to define a set of data output APIs (Application Programming Interface). In this embodiment, when the message consumed by the message middleware is output, the message is stored according to the start and end time of the message, the naming method of the message, the size of the target file, and the directory of the target file output set in the pre-stored consumption rule configuration file.

[0053] In one embodiment, the application program interface for the message output configures the naming method of the file storing the message, the size of the single file storing the message, and the directory of the file storing the message. The step S130 includes: storing the consumed message in the target file according to the naming method of the message to be consumed, the size of the single file storing the message, and the directory of the file storing the message, and the naming method of the file storing the message includes setting the start and end times as the name of the target file.

[0054] Among them, the naming method of the file that saves the message is the naming method of the file that stores the message consumed from the message middleware. When the file used to store the consumed message is a text file, the start and end time are set to the name of the text file. For example, if the message consumed from the message middleware is from 7:55:00 on December 30, 2016 to 12:55:00 on December 30, 2016, the text file storing the consumed message will be named "20161230075500-20161230125500". Of course, when multiple files with the start and end times are included, the multiple files with the start and end times can be numbered in order of storage. For example, the message consumed from the message middleware is from 7:55:00 on December 30, 2016 to 12:55:00 on December 30, 2016, and the consumed messages are stored in two text files, which are named "20161230075500-20161230125500-01" and "20161230075500-20161230125500-02" respectively.

[0055] The size of a single file storing the message includes the capacity of the single file storing the message or the number of stored messages in the single file storing the message. When there are many messages acquired within the start and end time, the messages consumed within the start and end time need to be stored in multiple files. For example, the capacity of a single file can be set to 2M bytes, and each file can store 2M bytes of messages, or the number of stored messages in a single file is set to 100, and each file can store 100 messages.

[0056] The directory of the file storing the message is the storage location of the file storing the message, and configuring the directory of the file storing the message in the application program interface for outputting the message can facilitate searching of the message.

[0057] In one of the embodiments, the application program interface where the message enters is configured with the communication protocol of the message that needs to be consumed; the step 120 includes: communicating with the corresponding message middleware according to the communication protocol of the message that needs to be consumed; according to the message consumption rule, searching the message middleware for the message that arrives at the message middleware within the start and end time, and consuming it.

[0058] The communication protocol includes the TCP / IP protocol, that is, the network communication protocol. Different communication protocols can be connected to different message middleware to consume messages in the message middleware.

[0059] In one of the embodiments, the step S120 includes: step S121, according to the message consumption rule, querying the location of the message where consumption starts and the location of the message where consumption ends from the message middleware; the messages are sorted in the message middleware in the order of the time of arrival at the message middleware; step S122, consuming the messages between the location of the message where consumption starts and the location of the message where consumption ends in sequence.

[0060] Among them, the messages in the message middleware have corresponding numbers, and the unique corresponding message can be found in the message middleware according to the number, and there may be multiple messages corresponding to the start and end times, that is, there are multiple messages arriving at the message middleware at the same time. Therefore, it is necessary to obtain the position of the message that starts consumption and the position of the message that ends consumption according to the start and end times.

[0061] For example, the start time of the message to be consumed is 12:30:15 Beijing time. At 12:30:15 Beijing time, messages A, B, and C arrive at the message middleware S. Then, the time when messages A, B, and C arrive at the message middleware S is 12:30:15 Beijing time. The message middleware stores messages A, B, and C according to 12:30:15 Beijing time. Of course, messages A, B, and C have corresponding numbers. If the message middleware stores 1,000 messages in advance, the previously stored messages have been assigned numbers 1-1000. Then, messages A, B, and C continue to be numbered according to the assigned numbers. Then, messages A, B, and C can be numbered 1001, 1002, and 1003. The position of message A with the smallest number at the start time (12:30:15 Beijing time) is used as the position of the message to be consumed. For example, the end time of the message to be consumed is 13:30:15 Beijing time. At 13:30:15 Beijing time, messages L and I arrive at the message middleware S. Then the time when messages L and I arrive at the message middleware S is both 13:30:15 Beijing time. The message middleware stores messages L and I according to 13:30:15 Beijing time. Of course, messages L and I have corresponding numbers. If the message middleware has pre-stored 2000 messages, the previously pre-stored messages have been assigned numbers 1-2000. Then messages L and I continue to be numbered according to the assigned numbers. Then, messages L and I can be numbered 2001 and 2002, and the position of message I with the largest number at the end time (13:30:15 Beijing time) is used as the position of the message that ends consumption.

[0062] In one of the embodiments, after the step of sequentially consuming the messages between the position of the message where consumption starts and the position of the message where consumption ends, the step includes: recording the size of each of the consumed messages and / or counting the number of the consumed messages. The size of each of the consumed messages refers to the size of the consumed message itself. Recording according to the size of the message facilitates the division of the consumed messages into multiple files for storage. Similarly, counting the number of the consumed messages also facilitates the division of the consumed messages into multiple files for storage. For example, if the target file for storing the consumed messages is 2M bytes, the total size of the messages stored in the target file does not exceed 2M bytes, or if the target file for storing the consumed messages can store 100 messages, the total number of messages stored in the target file does not exceed 100.

[0063] In one embodiment, step 130 includes: storing the consumed messages in a text file according to a preset file size or a preset number of messages, the text file stores messages of a preset file size or a preset number of messages, and setting the start and end time as the name of the text file.

[0064] It should be understood that although Figure 2-3 The steps in the flowchart are shown in sequence as indicated by the arrows, but these steps are not necessarily executed in the order indicated by the arrows. Unless otherwise specified in this document, there is no strict order restriction for the execution of these steps, and these steps can be executed in other orders. Moreover, Figure 2-3 At least part of the steps may include multiple steps or multiple stages. These steps or stages are not necessarily performed at the same time, but can be performed at different times. The execution order of these steps or stages is not necessarily sequential, but can be performed in turn or alternately with other steps or at least part of the steps or stages in other steps.

[0065] In one embodiment, Figure 4 As shown, a message consumption device is provided, including: a message consumption rule acquisition module 210, a consumption module 220 and a storage module 230. Among them:

[0066] The message consumption rule acquisition module 210 is used to obtain the corresponding message consumption rules from the pre-stored consumption rule configuration file according to the name of the data source; the message consumption rules include the start and end time of the message to be consumed, and the start and end time are set according to the estimated time when the message arrives at the message middleware.

[0067] The consumption module 220 is used to search the message middleware for the message that arrives at the message middleware within the start and end time according to the message consumption rule, and consume the message.

[0068] The storage module 230 is used to store the consumed message into a target file and set the start and end time as the name of the target file.

[0069] In one of the embodiments, the message consumption rule acquisition module 210 is also used to obtain the message input application interface, the message output application interface, and the executable file for processing the input and output of the message from a pre-stored consumption rule configuration file according to the name of the data source; wherein the message input application interface, the message output application interface, and the executable file for processing the input and output of the message are configured with corresponding message consumption rules; wherein the message output application interface is configured with the start and end time of the message to be consumed.

[0070] In one embodiment, the application program interface for the message output is configured with the naming method of the file storing the message, the size of the single file storing the message, and the directory of the file storing the message. The storage module 230 is also used to store the consumed message in the target file according to the naming method of the message to be consumed, the size of the single file storing the message, and the directory of the file storing the message. The naming method of the file storing the message includes setting the start and end time as the name of the target file.

[0071] In one of the embodiments, the application program interface in which the message enters is configured with the communication protocol of the message to be consumed; the consumption module 220 includes: a communication unit, used to communicate with the corresponding message middleware according to the communication protocol of the message to be consumed; a consumption unit, used to search from the message middleware for the message that arrives at the message middleware within the start and end time according to the message consumption rules, and consume the message.

[0072] In one embodiment, the consumption module 220 includes: a message location query unit, which is used to query the location of the message where consumption starts and the location of the message where consumption ends from the message middleware according to the message consumption rule; the messages are sorted in the message middleware in the order of the time of arrival at the message middleware; and a consumption unit, which consumes the messages between the location of the message where consumption starts and the location of the message where consumption ends in sequence.

[0073] In one embodiment, the consumption module 220 further includes: a recording unit, configured to record the size of each of the consumed messages and / or count the number of the consumed messages.

[0074] In one embodiment, the storage module 230 is also used to store the consumed messages into a text file according to a preset file size or a preset number of messages, and the text file stores messages of a preset file size or a preset number of messages, and sets the start and end time as the name of the text file.

[0075] For the specific definition of the message consumption device, please refer to the definition of the message consumption method above, which will not be repeated here. Each module in the above-mentioned message consumption device can be implemented in whole or in part by software, hardware and a combination thereof. The above-mentioned modules can be embedded in or independent of the processor in the computer device in the form of hardware, or can be stored in the memory of the computer device in the form of software, so that the processor can call and execute the operations corresponding to the above modules.

[0076] In one embodiment, Figure 5 As shown, a message consumption system is provided, including: an input application program interface, an output application program interface, an adapter, a processor and a consumption rule configuration file. Among them, the input application program interface includes a communication protocol, a message type and a message configuration; the output application program interface includes a message consumer and a message output configuration; the adapter includes a message adapter, a message filter and a Java dependency package expansion unit; the processor includes a timer, a counter and a message location acquisition module.

[0077] Among them, the input application program interface is mainly used to define a set of APIs (Application Programming Interface) for data source entry processing. The application program interface is a set of definitions, programs and protocols, and mutual communication between computer software is realized through the API. When consuming messages of various message middlewares, it is necessary to communicate with the message middleware according to the corresponding API. Through the consumption rule configuration file, according to the type of message middleware, the corresponding configuration API can be used to realize communication with multiple message middlewares. For example, the communication protocol is used to obtain the TCP (Transmission Control Protocol) connection of the incoming message. TCP is a connection-oriented, reliable, byte stream-based transport layer communication protocol. The input application program interface is used to obtain the incoming message type, such as the message type includes kafka, rocketmq, activemq, rabbitmq, newdefinemq; the input application program interface obtains the incoming message configuration, such as the TCP connection string, connection parameters, extended dependency jar package and other data in the consumption rule configuration file. Among them, the jar package is a java executable file.

[0078] Among them, the output application program interface is mainly used to define a set of data output APIs (Application Programming Interface). In this embodiment, when the consumed message is output, the output message is stored through the start and end time of the message in the consumption rule configuration file, the naming method, the size of a single file, and the output directory. For example, the consumer of the message refers to the consumption thread created by the message processor module; the output configuration of the message is the start time of the message to be consumed, the end time of the message to be consumed, the naming method of the file that saves the message, the directory of the file that saves the message, and the size of the single file that saves the message in the message consumption rule.

[0079] Among them, the consumption rule configuration file is mainly used to configure message consumption rules. In the consumption rule configuration file, the corresponding message source entry API is set for various message middleware, the message output rules are defined when the message is output, and the corresponding jar package that depends on the message processing is configured. The jar package is a java executable file, which is designed according to the message source and executes the operation of consuming messages according to the consumption rules and storing messages according to the consumption rules. For example, the message consumption rules are used to configure the message source type, TCP connection string, connection parameters, the start time of the message to be consumed, the end time of the message to be consumed, the naming method of the file that saves the message, the directory of the file that saves the message, and the size of a single file that saves the message.

[0080] The adapter is mainly used to call the message adapter and initialize the message filter. For example, the adapter calls the message adapter, the message adapter calls the Java dependency package extension unit, and the Java dependency package extension unit calls the jar package that the message depends on for processing. The jar package is a java executable file. The adapter is used to initialize the message filter, that is, to assign values ​​to each attribute of the message filter, for example:

[0081] dataSourceType = 'kafka'

[0082] dataBeginTime='20190601120500'

[0083] dataEndTime='20190601180500'

[0084] dataName = 'premium'

[0085] Among them, the adapter initializes the message filter, that is, completes the settings of the message source type dataSourceType, the message start and end time (dataBeginTime, dataEndTime) and the data name dateName. The text file of the output message can be named according to the values ​​of each attribute set by the adapter.

[0086] Wherein, the processor is used to execute the process of consuming and storing messages according to the above-mentioned consumption rule configuration file. Wherein, the processor includes a timer, a counter, and a message position acquisition module. Wherein, the timer controls the start and end time of acquiring the message, wherein each message has a time to reach the message middleware, and the start and end time of the acquired message is set according to the time when the message arrives at the message middleware. For example, if it is known that the time of missing the message is about 9:00, then the start and end time of acquiring the message can be set to 9:00-10:00, and the missed message can be checked from the time period of 9:00-10:00; the counter controls the number and byte size of the acquired messages, wherein, for a whole segment of the acquired message, if the bytes are too large and not conducive to viewing, it is divided into multiple message segments as needed and the byte size of each message segment is set; the message position acquisition module pos controls the position of the acquired message in the middleware, wherein there are multiple messages in the message middleware, each message has a corresponding position in the message middleware, and the message position can be combined with the start and end time of the message to calculate the position of the message to be acquired in the message middleware.

[0087] In one embodiment, a computer device is provided. The computer device may be a terminal, and its internal structure diagram may be as follows: Figure 6 As shown. The computer device includes a processor, a memory, a communication interface, a display screen and an input device connected through a system bus. Among them, the processor of the computer device is used to provide computing and control capabilities. The memory of the computer device includes a non-volatile storage medium and an internal memory. The non-volatile storage medium stores an operating system and a computer program. The internal memory provides an environment for the operation of the operating system and the computer program in the non-volatile storage medium. The communication interface of the computer device is used to communicate with an external terminal in a wired or wireless manner, and the wireless manner can be achieved through WIFI, an operator network, NFC (near field communication) or other technologies. When the computer program is executed by the processor, a message consumption method is implemented. The display screen of the computer device can be a liquid crystal display screen or an electronic ink display screen, and the input device of the computer device can be a touch layer covered on the display screen, or a button, trackball or touchpad set on the computer device housing, or an external keyboard, touchpad or mouse, etc.

[0088] Those skilled in the art will understand that Figure 6The structure shown in the figure is only a block diagram of a part of the structure related to the solution of the present application, and does not constitute a limitation on the computer device to which the solution of the present application is applied. The specific computer device may include more or fewer components than those shown in the figure, or combine certain components, or have a different arrangement of components.

[0089] In one embodiment, a computer device is provided, including a memory and a processor, wherein a computer program is stored in the memory, and when the processor executes the computer program, the following steps are implemented:

[0090] Obtain corresponding message consumption rules from a pre-stored consumption rule configuration file according to the name of the data source; the message consumption rules include the start and end time of the message to be consumed, and the start and end time are set according to the estimated time when the message arrives at the message middleware;

[0091] According to the message consumption rule, searching the message middleware for the message that arrives at the message middleware within the start and end time, and consuming the message;

[0092] The consumed message is stored in a target file, and the start and end times are set as the name of the target file.

[0093] In one embodiment, a computer readable storage medium is provided, on which a computer program is stored, and when the computer program is executed by a processor, the following steps are implemented:

[0094] Obtain corresponding message consumption rules from a pre-stored consumption rule configuration file according to the name of the data source; the message consumption rules include the start and end time of the message to be consumed, and the start and end time are set according to the estimated time when the message arrives at the message middleware;

[0095] According to the message consumption rule, searching the message middleware for the message that arrives at the message middleware within the start and end time, and consuming the message;

[0096] The consumed message is stored in a target file, and the start and end times are set as the name of the target file.

[0097] Those of ordinary skill in the art can understand that all or part of the processes in the above-mentioned embodiment methods can be completed by instructing the relevant hardware through a computer program, and the computer program can be stored in a non-volatile computer-readable storage medium. When the computer program is executed, it can include the processes of the embodiments of the above-mentioned methods. Among them, any reference to memory, storage, database or other media used in the embodiments provided in this application can include at least one of non-volatile and volatile memory. Non-volatile memory can include read-only memory (ROM), magnetic tape, floppy disk, flash memory or optical memory, etc. Volatile memory can include random access memory (RAM) or external cache memory. As an illustration and not limitation, RAM can be in various forms, such as static random access memory (SRAM) or dynamic random access memory (DRAM).

[0098] The technical features of the above embodiments may be combined arbitrarily. To make the description concise, not all possible combinations of the technical features in the above embodiments are described. However, as long as there is no contradiction in the combination of these technical features, they should be considered to be within the scope of this specification.

[0099] The above-mentioned embodiments only express several implementation methods of the present application, and the descriptions thereof are relatively specific and detailed, but they cannot be understood as limiting the scope of the invention patent. It should be pointed out that, for a person of ordinary skill in the art, several variations and improvements can be made without departing from the concept of the present application, and these all belong to the protection scope of the present application. Therefore, the protection scope of the patent of the present application shall be subject to the attached claims.

Claims

1. A message consumption method, characterized in that: The method comprises: According to the name of the data source, the application program interface for message entry, the application program interface for message output, and the executable file for processing the input and output of the message are obtained from the pre-stored consumption rule configuration file, wherein the application program interface for message entry, the application program interface for message output, and the executable file for processing the input and output of the message are configured with corresponding message consumption rules; wherein the application program interface for message output is configured with the start and end time of the message to be consumed, and the start and end time are set according to the estimated time when the message arrives at the message middleware, the application program interface for message entry is configured based on the type of the message middleware, the application program interface for message output is configured with the naming method of the file for storing the message, the size of a single file for storing the message, and the directory of the file for storing the message, and the application program interface for message entry is configured with the communication protocol of the message to be consumed; Communicate with the corresponding message middleware according to the communication protocol of the message to be consumed, search for the message that arrives at the message middleware within the start and end time from the message middleware according to the message consumption rule, and consume the message; The consumed messages are stored in a target file according to the naming method of the files storing the messages, the size of the single files storing the messages and the directory of the files storing the messages. The naming method of the files storing the messages includes setting the start and end times as the name of the target file.

2. The method according to claim 1, characterized in that The step of searching the message middleware for messages arriving at the message middleware within the start and end time according to the message consumption rule and consuming the messages includes: According to the message consumption rule, the location of the message starting to be consumed and the location of the message ending to be consumed are queried from the message middleware; the messages are sorted in the message middleware in the order of the time when they arrive at the message middleware; The messages between the position of the message where consumption starts and the position of the message where consumption ends are consumed in sequence.

3. The method according to claim 2, characterized in that After the step of sequentially consuming the messages between the position of the message where consumption starts and the position of the message where consumption ends, the method further comprises: The size of each consumed message is recorded and / or the number of consumed messages is counted.

4. The method according to claim 1, characterized in that: Storing the consumed message to a target file and setting the start and end times to the name of the target file includes: The consumed messages are stored in a text file according to a preset file size or a preset number of messages, the text file stores messages of a preset file size or a preset number of messages, and the start and end times are set as the name of the text file.

5. A message consumption device, characterized in that: The device comprises: A message consumption rule acquisition module, used to obtain the message input application interface, message output application interface, and executable file for processing the message input and output from the pre-stored consumption rule configuration file according to the name of the data source, wherein the message input application interface, message output application interface, and executable file for processing the message input and output are configured with corresponding message consumption rules; wherein the message output application interface is configured with the start and end time of the message to be consumed, and the start and end time are set according to the estimated time when the message arrives at the message middleware, the message input application interface is configured based on the type of the message middleware, the message output application interface is configured with the naming method of the file storing the message, the size of the single file storing the message, and the directory of the file storing the message, and the message input application interface is configured with the communication protocol of the message to be consumed; A consumption module, used to communicate with the corresponding message middleware according to the communication protocol of the message to be consumed, and according to the message consumption rule, search the message middleware for the message that arrives at the message middleware within the start and end time, and consume it; A storage module is used to store the consumed messages into a target file according to the naming method of the files storing the messages, the size of the single files storing the messages and the directory of the files storing the messages. The naming method of the files storing the messages includes setting the start and end times as the name of the target file.

6. The device according to claim 5, characterized in that The consumption module also includes a message location query unit, which is used to query the location of the message that starts consumption and the location of the message that ends consumption from the message middleware according to the message consumption rule; the messages are sorted in the message middleware in the order of the time they arrive at the message middleware; the consumption unit consumes the messages between the location of the message that starts consumption and the location of the message that ends consumption in sequence.

7. The device according to claim 6, characterized in that The consumption module further includes a recording unit, which is used to record the size of each of the consumed messages and / or count the number of the consumed messages.

8. The device according to claim 5, characterized in that The storage module is also used to store the consumed messages into a text file according to a preset file size or a preset number of messages, the text file stores messages of a preset file size or a preset number of messages, and sets the start and end times as the name of the text file.

9. A computer device comprising a memory and a processor, wherein the memory stores a computer program, wherein: When the processor executes the computer program, the steps of the method according to any one of claims 1 to 4 are implemented.

10. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, the steps of the method according to any one of claims 1 to 4 are implemented.

Images