Message processing method and device

Abstract

The present application provides a message processing method and device, wherein the message processing method includes: obtaining a first write request from a producer, and establishing a file identifier at the starting position of a target queue file based on the first write request; determining a corresponding message byte according to the message to be executed carried in the first write request, and generating a corresponding length byte according to the length of the message byte; after the identification byte corresponding to the file identifier, sequentially splicing the length byte and the message byte as a response to the first write request, which enables a process of storing a message to be stored to omit an index file, thereby reducing the memory pressure of the system, and in the process of reading the message, it is also possible to omit the process of referencing the index file, making the process of message reading more concise and fast, and providing a good environment for reading and querying messages.

Description

Technical Field

[0001] The present application relates to the technical field of data storage, and in particular to a message processing method, a message reading method, a message processing device, a message reading device, a computing device, and a computer-readable storage medium. Background Art

[0002] With the development of Internet technology, information interaction has become more frequent and important, the message processing speed is getting faster and faster, the amount of message processing is getting larger and larger, and the importance is getting higher and higher. In the prior art, messages are stored in message queues, and the message location is determined by index files during the message writing process. However, as the amount of message processing increases, the number of index files used to associate each message increases, and the storage space occupied by index files has become a problem that cannot be ignored. The excessive storage space occupied by index files has seriously affected the operating rate of the computing device, and since each message writing requires the establishment of a corresponding index file, and each message reading requires the call of the index file, the operating process of the computing device becomes cumbersome, making the message processing process cumbersome and unable to be completed quickly. Summary of the invention

[0003] In view of this, the embodiment of the present application provides a message processing method to solve the technical defects existing in the prior art. The embodiment of the present application also provides a message processing device, a computing device, and a computer-readable storage medium.

[0004] According to a first aspect of an embodiment of the present application, a message processing method is provided, including:

[0005] Obtaining a first write request from a producer, and establishing a file identifier at a start position of a target queue file based on the first write request;

[0006] Determine a corresponding message byte according to the to-be-executed message carried in the first write request, and generate a corresponding length byte according to the length of the message byte;

[0007] After the identification byte corresponding to the file identifier, the length byte and the message byte are sequentially spliced ​​as a response to the first write request.

[0008] Optionally, it also includes:

[0009] receiving a second write request, wherein the second write request is a write request that is adjacent in time sequence and is subsequent to the first write request;

[0010] Obtaining a second length byte and a second message byte corresponding to the message to be executed carried in the second write request;

[0011] Determine whether the byte length of the second length byte and the second message byte plus the length of the written message in the current file is greater than a preset length threshold of the target queue file;

[0012] If not, writing the second length byte and the second message byte into the target queue file;

[0013] If so, create a new file, and write the to-be-executed message carried in the second write request into the new file.

[0014] Optionally, writing the second length byte and the second message byte into the target queue file comprises:

[0015] Determine a corresponding second message byte according to the to-be-executed message carried in the second write request, and generate a corresponding second length byte according to the length of the second message byte;

[0016] After the current message byte, the second length byte and the second message byte are sequentially spliced ​​as a response to the second write request.

[0017] Optionally, the creating a new file and writing the to-be-executed message carried in the second write request into the new file includes:

[0018] Obtain the target file name and the target file length of the target queue file;

[0019] The result of adding the target file name to the target queue length is used as the file name of the newly created file, wherein the target queue file is named based on preset target characters and file suffix.

[0020] Creating a new file, and establishing a second file identifier at a start position of the new file based on the second write request;

[0021] Determine a corresponding second message byte according to the to-be-executed message carried in the second write request, and generate a corresponding second length byte according to the length of the second message byte;

[0022] After the second file identifier, a second length byte and a second message byte are sequentially spliced ​​as a response to the second write request.

[0023] Optionally, establishing a file identifier at a start position of a target queue file based on the first write request includes:

[0024] Reading a queue file version character and ancillary information characters based on the to-be-executed message carried in the first write request;

[0025] Establishing the file identifier according to the queue file version character and the supplementary information character;

[0026] The queue version character is used to mark the version information of the current file, and the supplementary information character is used to mark a unique queue file in the message queue.

[0027] Optionally, determining a corresponding message byte according to the to-be-executed message carried in the first write request includes:

[0028] Determine a message header character string and a message content data character string corresponding to the message to be executed;

[0029] The message header character string and the message content data character string are concatenated to form the message bytes corresponding to the message to be executed.

[0030] Optionally, it also includes:

[0031] The target queue file is named based on preset target characters and file suffix.

[0032] Optionally, it also includes:

[0033] Lock the write interface of the message queue;

[0034] Correspondingly, obtaining the producer's write request includes:

[0035] Obtaining the write request submitted by the producer through the locked write interface;

[0036] Among them, if there are at least two producers submitting write requests through the locked write interface, each write request will be processed in sequence according to the write time.

[0037] According to a second aspect of an embodiment of the present application, a message reading method is provided, including:

[0038] Receive message reading requests from consumers;

[0039] Determining message location information based on the message reading request;

[0040] Determine the target task file corresponding to the message location information in the message queue;

[0041] In response to the message read request, the target message is read from the target task file and sent to the consumer.

[0042] Optionally, the reading a target message in the target task file in response to the message reading request comprises:

[0043] Reading a length byte of the target message in the target task file, and determining target length information of the target message based on the length byte;

[0044] Based on the target length information, a corresponding byte sequence is read from the target task file to form the target message.

[0045] Optionally, it also includes:

[0046] Obtain the file identifier of the target queue file;

[0047] Verifying the file version characters and the supplementary information characters in the file identifier;

[0048] If the verification is successful, the subsequent message reading tasks are executed sequentially;

[0049] If the verification fails, the process is terminated and an error message is returned to the consumer. Optionally, determining the target task file corresponding to the message location information in the message queue includes:

[0050] Determine the task file name contained in the message location information; determine the target task file in the message queue based on the task file name, or

[0051] Determine the byte position of the target message recorded in the message position information in the message queue; call the message queue program to determine the queue file names of the queue files in the message queue one by one; determine the byte range occupied by each queue file in the message queue according to the queue file name; determine the target task file where the target message is located based on the byte position and the byte range, or

[0052] Determine the target timestamp of the target message recorded in the message location information; the message queue program determines the creation time of the queue files in the message queue one by one; based on the target timestamp and the creation time of each queue file, determine the target task file where the target message is located.

[0053] According to a third aspect of an embodiment of the present application, a message processing device is provided, including:

[0054] an acquisition module configured to acquire a first write request from a producer, and to establish a file identifier at a start position of a target queue file based on the first write request;

[0055] a determination module configured to determine a corresponding message byte according to the to-be-executed message carried in the first write request, and to generate a corresponding length byte according to the length of the message byte;

[0056] The splicing module is configured to sequentially splice the length byte and the message byte after the identification byte corresponding to the file identifier as a response to the first write request.

[0057] According to a fourth aspect of an embodiment of the present application, a message reading device is provided, including:

[0058] A receiving module, configured to receive a message reading request from a consumer;

[0059] a location determination module, configured to determine message location information based on the message reading request;

[0060] A file determination module, configured to determine the target task file corresponding to the message location information in the message queue;

[0061] The reading module is configured to read the target message from the target task file in response to the message reading request and send the target message to the consumer.

[0062] According to a fifth aspect of an embodiment of the present application, there is provided a computing device, including:

[0063] Memory and processor;

[0064] The memory is used to store computer-executable instructions, and the processor implements the steps of the message processing method or the message reading method when executing the computer-executable instructions.

[0065] According to a sixth aspect of an embodiment of the present application, a computer-readable storage medium is provided, which stores computer-executable instructions, and when the instructions are executed by a processor, the steps of the message processing method or the message reading method are implemented.

[0066] According to a seventh aspect of an embodiment of the present application, a chip is provided, which stores a computer program, and when the computer program is executed by the chip, the steps of the message processing method or the message reading method are implemented.

[0067] The message processing method and message reading method provided by the present application receive a write request sent by a producer, and establish a file identifier at the beginning position of a queue file based on the content of the write request, determine the message byte and the length byte of the message byte according to the content of the message to be processed carried in the write request, and store the length byte and the message byte in sequence in the message queue. A message information to be stored is stored in this form. When reading a target message, the queue file corresponding to the target message can be directly found in combination with the file name of the queue file in the message queue, eliminating the index file and reducing the memory pressure of the system. In the process of reading the message, the process of referencing the index file can also be omitted, making the message reading process more concise and fast, and providing a good environment for reading and querying messages. BRIEF DESCRIPTION OF THE DRAWINGS

[0068] Figure 1 This is a flow chart of a message processing method provided by an embodiment of the present application;

[0069] Figure 2 This is a message queue diagram of a message processing method provided by an embodiment of the present application;

[0070] Figure 3 is a flow chart of a message reading method provided by an embodiment of the present application;

[0071] Figure 4 It is a processing flow chart of a message processing method applied to a data storage scenario provided by an embodiment of the present application;

[0072] Figure 5 It is a structural diagram of a message processing device provided by an embodiment of the present application;

[0073] Figure 6 It is a structural schematic diagram of a message reading device provided by an embodiment of the present application;

[0074] Figure 7 It is a structural block diagram of a computing device provided in one embodiment of the present application. DETAILED DESCRIPTION

[0075] Many specific details are described in the following description to facilitate a full understanding of the present application. However, the present application can be implemented in many other ways than those described herein, and those skilled in the art can make similar generalizations without violating the connotation of the present application, so the present application is not limited by the specific implementation disclosed below.

[0076] The terms used in one or more embodiments of the present application are only for the purpose of describing specific embodiments, and are not intended to limit one or more embodiments of the present application. The singular forms of "a", "said" and "the" used in one or more embodiments of the present application and the appended claims are also intended to include plural forms, unless the context clearly indicates other meanings. It should also be understood that the term "and / or" used in one or more embodiments of the present application refers to and includes any or all possible combinations of one or more associated listed items.

[0077] It should be understood that, although the terms first, second, etc. may be used to describe various information in one or more embodiments of the present application, such information should not be limited to these terms. These terms are only used to distinguish the same type of information from each other. For example, without departing from the scope of one or more embodiments of the present application, the first may also be referred to as the second, and similarly, the second may also be referred to as the first.

[0078] First, the terms involved in one or more embodiments of the present invention are explained.

[0079] Read-write lock: A special spin lock that divides the access to shared resources into readers and writers. Readers only read shared resources, while writers need to write to shared resources. It allows multiple readers to access shared resources at the same time, and the maximum number of readers is the actual number of logical CPUs. Writers are exclusive. A read-write lock can only have one writer or multiple readers (related to the number of CPUs) at the same time, but not both readers and writers at the same time.

[0080] In the present application, a message processing method is provided. The present application also relates to a message processing device, a computing device, and a computer-readable storage medium, which are described in detail one by one in the following embodiments. Figure 1 A flowchart of a message processing method provided according to an embodiment of the present application is shown, which specifically includes the following steps:

[0081] S102: Obtain a first write request from the producer, and establish a file identifier at the start position of the target queue file based on the first write request.

[0082] Applied to message queues, the message queue program receives the producer's write request and executes subsequent task steps.

[0083] The producer can be a data collector, sensor or other device that has message storage requirements. The message is stored in the queue file in the message queue. The current queue file where the producer writes the message information is called the target queue file. The file identifier exists in the header of the queue file. When reading the message, the consumer can verify the file identifier to ensure that there is no data anomaly in the queue file.

[0084] Specifically, in the process of the producer writing a message, it is first necessary to send a write request. The message queue program receives the write request and the write process starts. After the queue file for writing the message is selected in the message queue, a file identifier is established at the header of the target queue file based on the message queue and the specific circumstances of the message being written.

[0085] Based on this, when a producer writes a message to a message queue, a queue file with a file identifier can be obtained. When a consumer consumes a message in a message queue, after receiving the consumption request, the message queue program first needs to read the file identifier at the head of the queue file, and then can verify the queue file with the file identifier, which greatly avoids the phenomenon of being unable to read due to data anomalies or inconsistency between the reading method and the storage method during the queue file reading process.

[0086] Furthermore, considering that when a consumer reads a message in a message queue, a file identifier containing file version information and auxiliary information is established at the beginning of the queue file, the queue file in the message queue can be checked during the message reading process to reduce message reading anomalies caused by mismatch between the reading method and the queue version. In this embodiment, the specific implementation method is as follows:

[0087] Based on the to-be-executed message carried in the first write request, the queue file version character and the supplementary information character are read; the file identifier is established according to the queue file version character and the supplementary information character; wherein the queue version character is used to mark the version information of the current file, and the supplementary information character is used to mark a unique queue file in the message queue.

[0088] Specifically, by reading the queue file version character, the version information of the current queue file can be obtained, and by reading the supplementary information character, whether the current queue file is a special queue file in the message queue can be obtained.

[0089] Based on this, the version information of the file determines the way messages are written and read in the queue file. The writing and reading methods between queue files of different versions will be affected by the version information. With the optimization of the reading or storage structure of the version, it is impossible to read messages in the message queue by relying on the old message reading method, or the part of the version related to the reading or storage structure has not changed after the optimization, and the message can still be read. Therefore, considering the impact of the version on message reading, when creating a file identifier, the version information of the queue message in the message queue can be combined. If the writing and reading methods do not match the version, the messages written and read may be lost or wrong. The attached information character contains a string used to mark the unique queue file in the message queue, making the reading of messages in the message queue more targeted.

[0090] For example, in the APP log processing scenario, to obtain the user's login information, the APP needs to store the user's login information. At this time, the APP sends a write request to the message queue used for storage. The version character B_i and the attribute information character F_i of the queue file are read based on the user login message carried in the write request. The version character B_i and the attribute information character F_i are combined to create a 16-bit text identifier W_16 for subsequent identification.

[0091] In summary, based on this, using the queue file version character to mark the version of the queue file can not only avoid the above problems, but also reserve space for the upgrade of the queue version. The supplementary information makes the process of reading messages in the message queue more accurate. The file version character and the supplementary information character are used to create a file identifier, which can take into account the characteristics of the version and supplementary information to achieve the uniqueness of the file identifier.

[0092] Furthermore, when multiple producers write to the message queue at the same time, the messages in the message queue are in a dynamic process. If more than one producer writes to the same position in the message queue, the messages recorded in the message queue may be wrong due to the different contents written by the producers. Therefore, locking can be implemented at the write and read interface of the message queue. In this embodiment, the specific implementation method is as follows:

[0093] The write interface of the message queue is locked; the write request submitted by the producer through the locked write interface is obtained; if there are at least two producers submitting write requests through the locked write interface, each write request is processed in sequence according to the write time.

[0094] Specifically, locking is to provide read-write locks for consumers who read message queues and producers who write message queues. Read locks and write locks are mutually exclusive and cannot be performed at the same time; that is, when the writing thread is writing, the reading thread cannot read the message queue; write locks are also mutually exclusive in the message queue, and there can only be one write lock in the message queue at the same time; there are no restrictions between read locks, and multiple read locks can be executed concurrently to read the message queue. According to the above rules, when the messages in the message queue are written and read, if multiple producers send write requests to the message queue, that is, multiple writing threads request write locks, then write locks need to be granted in sequence according to the time sequence of multiple writing threads requesting locks. After a writing thread completes writing, the write lock is released, and other writing threads can request locks according to the time sequence at this time; if multiple writing and reading threads exist at the same time, the read lock and write lock are obtained in the read-write order, and the reading and writing processes are performed separately. When the write lock is locked, the read lock will be blocked, and only write but not read is realized; multiple reading threads exist at the same time, and read locks are obtained at the same time to concurrently read the message information in the message queue.

[0095] The write lock and the read lock are exclusive, which realizes the separation between reading and writing. When the producer writes messages to the message queue, the messages in the message queue are changing and dynamic. At this time, synchronous reading operations are likely to cause reading errors, or the read messages are before the latest written messages, making the read messages lose their real-time nature. The separation of reading and writing can solve this problem well.

[0096] During the writing process, the messages in the message queue are dynamic, and multiple writing threads are implemented simultaneously, so it is easy for errors to occur in the written messages. The mutual exclusion of the write lock ensures that there is only one writing thread at the same time, making the writing clearer and reducing the occurrence of errors.

[0097] When reading messages from a message queue, the message content is static and will not change. At this time, there are no requirements for the reading thread like the writing thread. At this time, concurrent reading of the read lock can greatly increase the reading speed and improve work efficiency.

[0098] In summary, by locking the write and read interfaces of the message queue, only one write is implemented at the same time, and there is no read during the writing process, thereby reducing the write and read errors of messages in the message queue.

[0099] In addition, in the actual application of writing messages in the message queue, it is rare to write a single message. More often, multiple messages need to be written into the message queue. In this case, multiple messages are written sequentially, that is, after the current write request is completed, the next write request starts to be executed. In this embodiment, the specific implementation method is as follows:

[0100] If the message is written successfully, the write position and write length of the written message will be returned to the current producer; if the writing fails, the failure information will be returned to the current producer.

[0101] Specifically, when a message is written successfully, the location and length of the written message are returned to the current producer, which can help the current writer to quickly locate the message the next time it is written, increase writing speed, and improve work efficiency. When a message fails to be written, the failure information is returned to the current producer, which is convenient for the current producer to analyze the cause of the write failure. Regardless of whether the write is successful or failed, the current producer will receive feedback information. After receiving the feedback information, the current producer will immediately release the write lock and end the write process. Subsequent producers continue to write, making the write process of multiple producers smoother and without jamming.

[0102] Using the above example, in the APP log processing scenario, the user's login information and the user's logout information are obtained. At this time, these two information need to be stored in the message queue for storing the user status. If the two information are written at the same time, the user status will be unclear. Through the lock processing of writing, the user login and logout time are sorted. If the user logs in first and logs out later, the writing thread of the user login message will first obtain the write lock. Due to the mutual exclusion of the write lock, two write locks will not appear at the same time, so the writing thread of the user login information will write the information in the message queue. After the writing is completed, the user login information is written to the position of the message queue and the write length is returned, and then the user's logout information begins to be written. The writing thread of the user logout information also needs to obtain the write lock first. After obtaining the write lock and writing the user logout information to the message queue, the user logout information is written to the position of the message queue and the write length is returned. The above method writes the user's login and logout information to the message queue, and the writing time of the two is staggered to avoid data conflicts.

[0103] In summary, when continuously writing messages in the message queue, the message queue is locked so that two consecutive writing threads in the message queue will not write at the same time, and the writing thread and the reading thread will not be performed at the same time, ensuring that the two writing threads will not write messages out of limit or mess up the writing messages due to simultaneous writing, and the reading thread will not read incorrectly because the writing thread rewrites the message in the message queue. After the writing thread finishes its work, it returns feedback information to the producer, and the subsequent producers continue to write, making the writing process of multiple producers smoother and without jamming.

[0104] S104: Determine a corresponding message byte according to the to-be-executed message carried in the first write request, and generate a corresponding length byte according to the length of the message byte.

[0105] Specifically, after the establishment of the file identifier based on the write request as mentioned above, on this basis, in order to complete the writing of the message without creating an index file, and to meet the reading needs of consumers when reading the message, and to avoid message confusion and congestion problems, the corresponding message bytes can be determined according to the to-be-executed message carried in the write request, and the corresponding length bytes can be generated according to the length of the message bytes, so that the subsequent message writing operation can be carried out based on this.

[0106] The pending message specifically refers to the message that needs to be written into the message queue in the write request. The message byte specifically refers to the fixed-format message byte that is generated based on the information of the pending message and is stored and read in the message queue. The length byte is the byte that records the length of the message byte.

[0107] Based on this, the producer can write messages to the message queue using a predetermined format, and the generated length byte can determine the range of each stored message in the message queue, greatly reducing the difficulty for consumers to read messages.

[0108] Furthermore, in order to successfully write the message into the queue without maintaining the index file, the subsequent message writing operation can be implemented by determining the message bytes. In this embodiment, the specific implementation is as follows:

[0109] Determine the message header string and message content data string corresponding to the message to be executed; and concatenate the message header string and the message content data string to form the message bytes corresponding to the message to be executed. The message header string includes part or all of the message ID, message type, message code, message generation time, queue program reception time, checksum, and message content data string length.

[0110] Specifically, the message header string contains the attribute information of the message, and the message content string records the specific content of the message. The message header string and the message content string together constitute the message bytes. The message header string can contain a variety of attribute information, which does not have to appear in the message header string. These attribute information will be randomly selected according to the message content stored in the message queue or the different requirements of the read and write requests of the message queue. The appearance of one or more attribute information in the message header string is not fixed, so as to adapt to the message reading and writing requirements in different environments.

[0111] Based on this, at least one of the message ID, message type, message code, message generation time, queue program reception time, check code, and message content data string length of the information to be processed is obtained according to the message to be processed to form a message header string, and the message header string and the message content string formed by the message content are concatenated to form message bytes.

[0112] For example, in the processing scenario of APP logs, the user's login information is written to the message queue. If during this writing process, the message header string needs to record the message ID, message type, message generation time, queue program reception time, and message content data string length of the user login information, first determine that the message ID of the written user login information is ID_i, the message type is S_i, the message generation time is Tb, the queue program reception time is Tr, and the message content data string length is Lc_i. At this time, ID_i, S_i, Tb, Tr, and Lc_i are sequentially concatenated to form a message header string of the user login information, and then this message header string is concatenated with the message content string recording the user login message to generate the message bytes of the login message.

[0113] In actual applications, since the message header string corresponds to different contents, different combinations can be selected as the message header string based on different scenarios, and together with the message content string, they constitute the message bytes. This message byte is suitable for different scenarios and can extend many different reading and writing functions. The selection of the message header string can be determined according to the actual application scenario, and this embodiment does not make any limitation here.

[0114] In summary, the header of the message bytes stored in the above form contains a large amount of information about the message content. In the process of consumers reading messages, many calculation and search processes are saved, which greatly improves work efficiency. In particular, when consumers can read in parallel, multiple readings can be carried out simultaneously, which also saves computing power costs.

[0115] S106: After the identification byte corresponding to the file identifier, sequentially concatenate the length byte and the message byte as a response to the first write request.

[0116] Specifically, in the above process, message bytes and length bytes are created. To write the message completely into the message queue, the two need to be spliced ​​in sequence. The length byte identifies the length of the message bytes, so the length byte should be spliced ​​in front of the message bytes. In the process of message reading, the range of the message bytes can be determined first to avoid reading errors.

[0117] Furthermore, in order to ensure the continuous writing of messages in the message queue, after the current writing process ends, it is necessary to continue to execute the writing of the message queue by the subsequent writing thread. In this embodiment, the specific implementation method is as follows:

[0118] Receive a second write request, wherein the second write request is a write request that is adjacent in time sequence and after the first write request; obtain a second length byte and a second message byte corresponding to the message to be executed carried in the second write request; determine whether the byte length of the second length byte and the second message byte plus the length of the write message in the current file is greater than a preset length threshold of the target queue file; if not, write the second length byte and the second message byte to the target queue file; if so, create a new file, and write the message to be executed carried in the second write request into the new file.

[0119] Specifically, the first write request and the second write request are two corresponding write requests, the first write request is identified in front and the second write request is in the back, and the producer who makes the second write request can be the previous producer or another producer. The preset length threshold of the target queue file is the maximum length of the target queue file. When the length of the written message exceeds this value, the currently written message will be written into the next queue file.

[0120] Based on this, after the first write request is processed, the subsequent write tasks are executed sequentially. After receiving the second write request, the attribute information of the message to be executed is extracted according to the message to be executed carried in the second write request, and the required attribute information is combined into a message header string, and the message header string is spliced ​​with the message content to form message bytes, and then the length of the message bytes is calculated according to the message bytes, and the length bytes of a fixed format are generated according to this length, and then spliced ​​in the order of length bytes first and message bytes later. Then the total length of the spliced ​​string is calculated, and the character length already occupied by the current queue file is added to determine whether it is greater than the length threshold of the message queue. If it is greater, it means that the message to be executed cannot be stored in the current queue file, and a new file needs to be created to write it; if it is not greater, it means that the message to be executed can be stored in the current queue file, and the message to be executed can be written sequentially after the last written message.

[0121] For example, in the APP log processing scenario, the user's login information and the information about the user's purchase of membership functions in the APP are written into the message queue in sequence. After the user's login information is written, before the information about the user's purchase of membership functions in the APP is continued to be written, it is necessary to determine whether the space of the current queue file in the message queue can store the information about the user's purchase of membership functions in the APP. At this time, add the total length of the message bytes and length bytes of the information about the user's purchase of membership functions in the APP, plus the length that has been written in the current queue file. If it is greater than the set queue file length threshold of 100M, a new file is created for storage. If it does not exceed 100M, it is stored in the current file.

[0122] In summary, a method for processing continuous message write requests is provided, so that messages are written into the message queue continuously. The length threshold of the target queue file is set so that the length of the message written into the queue file is not infinite. The messages written into the message queue can be distributed more reasonably, making the structure of the queue file in the message queue more reasonable, and consumers can read the messages in the message queue more quickly and orderly.

[0123] Furthermore, when the message queue is continuously written, the messages to be written are written one by one into the queue file in the message queue. To avoid the queue file being too long, the storage space of the queue file needs to be limited. Then, in the case of continuous writing of messages, if the space of the current queue file is sufficient to store the message bytes and length bytes corresponding to the second write request, the message to be executed carried in the second write request is written into the current queue file. In this embodiment, the specific writing process of the second write request is as follows:

[0124] Determine the corresponding second message byte according to the to-be-executed message carried in the second write request, and generate the corresponding second length byte according to the length of the second message byte; after the current message byte, sequentially splice the second length byte and the second message byte as a response to the second write request.

[0125] Specifically, the second message byte is a message byte generated according to the to-be-executed message carried in the second write request, and the second length byte is the length byte of the second message byte. When the character space in the queue file is sufficient to store the newly written message, the newly written message is stored in the queue file. The specific storage method is the same as the previous message storage method, and both need to determine the corresponding message byte of the message content and the length byte of the message byte, and sequentially splice and write them.

[0126] Based on this, after the first write request has been processed, the subsequent write tasks are carried out in sequence. After receiving the second write request, the attribute information of the message to be written is extracted according to the pending message carried in the write request, and the required attribute information is combined into a message header string, and the message header string is spliced ​​with the message content to form message bytes. Then, based on the message bytes, the length of the message bytes is calculated, and this length is stored in the length bytes of a fixed format, and then spliced ​​in the order of length bytes first and message bytes later, and then the total length of the spliced ​​string is calculated, plus the character length already occupied by the current queue file, and it is judged to be less than the length threshold of the message queue, and the string spliced ​​by the above length bytes and message bytes is written into the current queue file. Finally, the address information of the written message is returned to the producer.

[0127] For example, in the processing scenario of APP logs, the user's login information is successfully stored in the message queue, and then the information of the user purchasing the membership function in the APP is written into the message queue. First, it is necessary to determine the message byte M_i of the information of the user purchasing the membership function in the APP and the length byte Ls_i that records this message byte. If the total length of Mi_i and Ls_i plus the length of the bytes already occupied in the queue file add up to 99M, and this total length is less than the preset threshold length of 100M of the queue file, then the information of the user purchasing the membership function in the APP is sufficient to be stored in the current queue file. Next, the message byte M_i and the length byte Ls_i of the information of the user purchasing the membership function in the APP are sequentially spliced ​​into the message queue.

[0128] In contrast, in the case of continuous writing operations to the message queue as described above, if the space of the current queue file is insufficient to store the message bytes and length bytes corresponding to the second write request, in this embodiment, the specific writing process of the second write request is as follows:

[0129] Create a new file, and establish a second file identifier at the starting position of the new file based on the second write request; determine the corresponding second message byte according to the to-be-executed message carried in the second write request, and generate the corresponding second length byte according to the length of the second message byte; after the second file identifier, sequentially splice the second length byte and the second message byte as a response to the second write request.

[0130] Specifically, when the character space in the queue file is insufficient to store the newly written message, it is necessary to first create a new queue file. At the beginning of the newly created queue file, it is still necessary to establish a file identifier to identify the version of the queue file and other information. The subsequent process of storing the newly written message into the newly created message queue is similar to the content described in the above embodiment, and this embodiment will not be repeated here.

[0131] Based on this, after the first write request has been processed, the subsequent write tasks are carried out in sequence. After receiving the second write request, the attribute information of the message to be written is extracted according to the to-be-executed message carried in the write request, and the required attribute information is combined into a message header string, and the message header string is spliced ​​with the message content to form message bytes, and then the length of the message bytes is calculated according to the message bytes, and the length is stored in the length byte of the fixed format. Then, the splicing is carried out in the order of length bytes first and message bytes later, and then the total length of the spliced ​​string is calculated, and the length of the character already occupied by the current queue file is added. It is judged that it is greater than the length threshold of the message queue, and a new queue file is created. The newly created queue file is named according to the predetermined naming rules, and a file identifier is established at the head of the newly created queue file according to the predetermined rules, and then the string spliced ​​by the above-mentioned length bytes and message bytes is written into the newly created queue file after the file identifier. Finally, the address information of the written message is returned to the producer.

[0132] For example, in the processing scenario of APP logs, the user's login information is successfully stored in the message queue, and then the information about the user's purchase of membership functions in the APP is written into the message queue. At the beginning, it is necessary to determine the message byte M_i of the information about the user's purchase of membership functions in the APP and the length byte Ls_i that records this message byte. If the total length of Mi_i and Ls_i plus the length of the bytes already occupied in the queue file add up to 101M, and this total length is greater than the preset threshold length of 100M of the queue file, then the information about the user's purchase of membership functions in the APP is not enough to be stored in the current queue file. A new queue file needs to be created, and a 16-bit file identifier W_16 is established at the header of the new queue file. Next, the message byte M_i and the length byte Ls_i of the user's purchase of membership functions in the APP are sequentially spliced ​​into the message queue.

[0133] In summary, by calculating the remaining storage space of the current queue file in the message queue and comparing it with the length of the message written by the subsequent writing process, the storage of the queue file is ensured not to exceed the preset threshold length, which ensures that the queue file will not be lengthy and avoids the slow writing and reading process and the system running not smooth due to the lengthy queue file.

[0134] Furthermore, when creating a new file, there is no obvious distinction between the files. This means that during the data reading process, the queue files can only be read one by one in sequence, which greatly slows down the running time of the system. Based on this, it is necessary to name each queue file. In this embodiment, the specific implementation method is as follows:

[0135] Obtain a target file name and a target file length of the target queue file; and use the result of adding the target file name and the target queue length as the file name of the newly created file.

[0136] The target queue file length is the number of bytes occupied by the target queue file in the message queue.

[0137] Based on this, from the naming process of the newly created files mentioned above, it can be seen that the starting position of each queue file in the message queue is the file name of this queue file plus 1, and the ending position of each queue file in the message queue is the file name of the next queue file. The file names of two adjacent queue files can be used to easily obtain the range of characters occupied by the queue file in the message queue, and the location of the queue file can be easily found. Among them, the naming of the first queue file is a supplement to the scheme. For example, the preset target character can be 0 to ensure that the naming of each queue file does not deviate from the above design ideas. For specific naming formats, please see the attached figure of the instruction manual. Figure 2 , where the first queue file is named 0.pqdata, and the byte length occupied by the first queue file is 104857600, then the second queue file next to the first queue file is named 104857600.pqdata, and the naming of subsequent files is similar.

[0138] For example, in the APP log processing scenario, the user's login information is successfully stored in the message queue, and then the information about the user's purchase of the membership function in the APP is written into the message queue. If the information about the user's purchase of the membership function in the APP is not enough to be stored in the current queue file, assuming that the byte length occupied by the current queue file is 104857600, when a new queue file is created to store the information about the user's purchase of the membership function in the APP, the file name of the new file is the file name of the previous queue file 0 plus the length of the previous queue file 104857600, plus the suffix .pqdata, the file name of the new file can be obtained as 104857600.pqdata.

[0139] In summary, the position of the queue file in the message queue can be determined by the file name of the queue file, and combined with the position of the message in the queue file, the position of the message in the message queue can be accurately located. In this way, the message position is searched without the guidance of a separate location information file or redundant index files. This makes the message search process simple, greatly improves the system's operating speed, and reduces unnecessary memory usage.

[0140] Furthermore, in the process of naming files, since the newly created files are named based on the file name and file length information of the previous file, the first queue file in the message queue does not have a previous queue file, so its naming method is not applicable to the above method. In this embodiment, the specific method to solve this problem is as follows:

[0141] The target queue file is named based on preset target characters and file suffix.

[0142] Specifically, the first queue file in the message queue has a special name because there is no previous queue file. Here, the purpose of naming the queue file is to identify the position of the queue file in the message queue. The first queue file in the message queue can be named 0 or 1. Naming it 0 indicates that there are 0 bytes in the message queue before the first byte of this queue file; naming it 1 indicates the position of the first byte of this message queue file in the message queue.

[0143] For example, in the APP log processing scenario, the user's login information is stored in the message queue. Since the user's login information is the first information stored in the message queue, the queue file is named 0.pqdata.

[0144] In summary, the first queue file in the message queue can be named by the preset target character, which solves the problem that the first queue file cannot be named because there is no previous queue file.

[0145] The message processing method provided by the present application receives a write request sent by a producer, and establishes a file identifier at the beginning of a queue file based on the content of the write request, determines the message byte and the length byte of the message byte according to the content of the message to be processed carried in the write request, and stores the length byte and the message byte in the message queue in sequence, and a message information to be stored is stored in this form. When reading a target message, the queue file corresponding to the target message can be directly found in combination with the file name of the queue file in the message queue, eliminating the index file and reducing the memory pressure of the system. In the process of reading the message, the process of referencing the index file can also be omitted, making the message reading process more concise and fast, and providing a good environment for reading and querying messages.

[0146] This embodiment also provides an embodiment of a message reading method, which is specifically described as follows:

[0147] Figure 3 A flowchart of a message reading method provided by an embodiment of the present application is shown. Figure 3 As shown, the specific implementation of the message reading method is as follows:

[0148] S302: Receive a message reading request from a consumer.

[0149] This embodiment provides a message reading method corresponding to the above embodiment, which is used to read the message corresponding to the writing after the message is written using the above message processing method, and is specifically implemented as follows:

[0150] After the message is written, it is saved in the message queue for subsequent consumption by consumers. When consumers read messages in the message queue, they need to rely on the file naming and file identifier established when the producer writes to the message queue. When reading messages in the message queue, the message queue program receives the message read request sent by the consumer, determines the specific location of the message to be read in the message queue based on the message location information carried in the message read request, and then reads the message to be read from the message queue based on this specific location, and returns the content of the message to be read and the location information of the message to be read to the consumer.

[0151] This solution is applied to message queues, where the message queue program receives the consumer's read request and executes subsequent task steps.

[0152] Among them, the consumer is a device, program or process that consumes the messages in the message queue.

[0153] Specifically, when a consumer consumes a message queue, it first needs to send a message read request. The message queue program receives the message read request and the reading process begins. In the message read request, the consumer has its own demand for reading messages in the message queue, and this demand can be included in the message read request sent by the consumer.

[0154] It should be noted that the message reading method provided in this embodiment corresponds to the message processing method provided in the above embodiment, wherein the description contents that are the same or corresponding to those in the above embodiment can be referred to the above embodiment, and this embodiment will not be repeated here.

[0155] S304: Determine message location information based on the message reading request.

[0156] Specifically, after receiving the consumer's message reading request as mentioned above, the message queue program can perform corresponding message reading according to the consumer's message reading demand in the message reading request issued by the consumer. It should be noted that the consumer's ultimate goal is to consume the messages in the message queue, so the consumer's demand for message reading can be reflected in the message position selection for the message to be read in the message reading request.

[0157] Among them, the message location information is the information that can specify the location of the message to be read in the message queue during the message reading process. After the message queue file receives the message reading request sent by the consumer, it can obtain the message location information based on this message reading request.

[0158] Based on this, after obtaining the message location information, the message queue program can search the queue file where the message to be read is located in the message queue, and finally determine the specific location of the message to be read and read it.

[0159] S306: Determine the target task file corresponding to the message location information in the message queue.

[0160] Specifically, after the message location information is determined through the consumer's message reading request, the message queue program can search the queue file in the message queue to confirm the location of the message to be read according to the message location information.

[0161] Furthermore, when a consumer needs to consume a message queue file in a whole queue file storing a specific message in the message queue during the process of consuming the message queue, the message queue program can verify the additional information characters in the file identifiers of all queue files in the message queue, or directly search according to the location information of the required queue file. In this embodiment, the specific method for solving this problem is as follows:

[0162] Determine the task file name contained in the message location information; and determine the target task file in the message queue based on the task file name.

[0163] Specifically, the consumer needs to read a complete queue file. At this time, the file name of the queue file to be read is put into the message reading request. After receiving the message reading request, the message queue program will directly determine the target task file in the message queue according to the queue file name carried therein.

[0164] For example, in the processing scenario of APP logs, the user's login information is stored in the message queue. When the real-time computing program needs to read the user's login information, the real-time computing program sends a consumption request, and the message queue program accepts the consumption request. The consumption request includes the message location information of the message to be read, where the user login information is stored in the first queue file in the message queue. Assume that in this example, the file name of the first queue file is 0.pqdata. The consumer places the file name 0.pqdata in the message reading request. After receiving the message reading request, the message queue program determines the first queue file as the target task queue according to the file name 0.pqdata.

[0165] In summary, this consumption process eliminates the process of calling index files, reduces the storage pressure of the system, speeds up task processing, and improves the work efficiency of the system.

[0166] In addition, when a consumer sends a message read request based on the byte range in the message queue to be read during the consumption of the message queue, the message queue program needs to determine the target task file where the message to be read is located based on the byte range carried in the message read request when receiving the message read request. In this embodiment, the specific method to solve this problem is as follows:

[0167] Determine the byte position of the target message recorded in the message location information in the message queue; call the message queue program to determine the queue file names of the queue files in the message queue one by one; determine the byte range occupied by each queue file in the message queue according to the queue file name; based on the byte position and the byte range, determine the target task file where the target message is located.

[0168] Specifically, the byte position represents the byte occupied by the message to be read in the message queue.

[0169] Based on this, the message queue program determines the byte position of the message to be read in the message queue according to the information recorded in the message position information. The message queue program determines the file names of the queue files in the message queue one by one. Because the file name of the queue file is the file name of the next queue file equal to the file name of the previous queue file plus the byte length of the queue file of the previous message queue, then the file name of the previous queue file plus 1 is the starting byte position of the previous file in the message queue, and the file name of the next queue file is the ending byte position of the previous queue file. The byte range occupied by the previous queue file in the message queue can be obtained through the file names of the previous and next two queue files. Through this rule combined with the byte position of the message to be read, the target task file where the message to be read is located can be found.

[0170] Using the above example, in the APP log processing scenario, the real-time computing program needs to read a certain login information of the user. At this time, the message position information can be the position of the first byte of the message to be read in the message queue, or the byte range occupied by the message to be read in the message queue. Assuming that in this example, the position of the first byte of the message to be read in the message queue is the message position information, and this position information is 101. According to the message queue program, the file name of the queue file in the message queue is searched, and the file name of the first queue file is found to be 0, and the file name of the second queue file is 104857600. According to the naming method of the queue files in the above message queue, it can be known that the byte range occupied by the first queue file in the message queue is 1-104857600, and the position information 101 is within 1-104857600. At this time, it can be known that the queue file where the message to be read is located is the first queue file, and the first queue file is determined as the target task queue.

[0171] In summary, this consumption process eliminates the process of calling index files, reduces the storage pressure of the system, speeds up task processing, and improves the work efficiency of the system.

[0172] In addition, when consumers consume message queues for real-time calculations, they often need to go back to a certain time point to re-consume data to achieve the purpose of recalculating data. Therefore, this queue program provides a consumption position closest to the specified message receiving time. In this embodiment, the specific implementation is as follows:

[0173] Determine the target timestamp of the target message recorded in the message location information; the message queue program determines the creation time of the queue files in the message queue one by one; based on the target timestamp and the creation time of each queue file, determine the target task file where the target message is located.

[0174] Specifically, the target timestamp is the writing time information of the message in the consumption queue to be read by the consumer in the process of consuming the message queue in a time-tracing manner; the target time file is the queue file where the message information corresponding to the target timestamp is located.

[0175] Based on this, when consumers use the time backtracking method to consume the message queue, they first send a consumption request. The target timestamp carried in this consumption request records the time information of the message to be read. After receiving the consumption request, the message queue program traverses the queue files in the message queue and obtains the time information of each queue file in the message queue. The time information of the queue file is the time point when the last message information is written in this queue file. The queue file whose time information is closest to the target timestamp and is greater than or equal to the time information recorded by the target timestamp is found in each queue file. At this time, it can be determined that the message to be read is stored in this file. First, select the queue file whose time information is closest to the time information recorded by the target timestamp, and narrow the scope to two or one queue files. In the case of one queue file, the message to be read is stored in the first message queue. The time information of all queue files is greater than or greater than or equal to the time information recorded in the target timestamp. Selecting the closest queue file directly selects the first queue file.

[0176] In the case of two queue files, the time information of the selected queue files is one greater than the time information recorded in the target timestamp, and the other is less than, because the time information of the queue file is the time point of the last message information written to this file. If the message corresponding to the time information recorded in the target timestamp exists in a queue file, then the time point of the last message information written to the queue file should be greater than or equal to the write time of the message information to be read in this queue file, which means that the time point of the last message written to the queue file is greater than the target timestamp information.

[0177] For example, in the processing scenario of APP logs, the user's login information is stored in the message queue. After the APP runs for a period of time, the user uses a function in the APP. This function requires a certain login information in the user's historical login information. At this time, it is necessary to use the time backtracking method to read the required user login information. After the message queue program receives the consumer's consumption request, it obtains the time information T1 recorded in the target timestamp in the consumption request. The message queue program traverses the queue files in the message queue and obtains the various time information of the queue files in the message queue. The queue file with the time information closest to the target timestamp and greater than or equal to the time information recorded by the target timestamp is the target time file.

[0178] In summary, in the process of consumers consuming messages in the message queue, consuming through time backtracking makes the consumption method more diversified, can meet different consumption environments, and adapt to more scenarios.

[0179] Furthermore, before reading the message to be read in the queue file, it is also necessary to select the reading method. It is necessary to ensure that the reading method matches the version of the queue file to ensure that the message read in the queue file is correct. Therefore, it is necessary to read the version information of the queue file. In this embodiment, the specific implementation method is as follows:

[0180] Obtain the file identifier of the target queue file; verify the file version character and the attached information character in the file identifier; if the verification passes, execute the subsequent message reading tasks in sequence; if the verification fails, end the process and return the error information to the consumer.

[0181] Specifically, the inspection of the file identifier includes a query of the file version information. According to the indication of the file version character, the version information of the queue file can be clearly understood, which facilitates the determination of the reading method for this version file and avoids errors caused by inconsistency between the reading method and the queue file version.

[0182] If the verification is successful, it proves that the queue file meets the reading conditions and can continue to be read. If the verification fails, it means that an unknown problem has occurred in the queue file. At this time, continuing to read the messages in the queue file cannot guarantee the correct reading of the messages, so the process is terminated immediately and the error information is returned to the consumer to facilitate subsequent inspection and maintenance.

[0183] For example, in the APP log processing scenario, the user's login information is stored in the message queue, and the user's login information needs to be read at this time. After the message queue program receives the consumer's consumption request, it determines the queue file where the user's login information is stored based on the message location information in the consumption request and the file name of the queue file. In the determined queue file, the first 16 bits are read sequentially from the header of the queue file. These 16 bits are the file identifier of this queue file. After that, the file identifier is verified to be correct, and the reading method is determined based on the file version information in the file identifier.

[0184] To summarize, by receiving the write request sent by the producer, and establishing a file identifier at the beginning position of the queue file based on the content of the write request, the message byte and the length byte of the message byte are determined according to the content of the message to be processed carried in the write request, and the length byte and the message byte are stored in sequence in the message queue. A message information to be stored is stored in this form, which eliminates the index file, reduces the memory pressure of the system, makes the message reading process more concise and fast, and provides a good environment for reading and querying messages.

[0185] S308: Read the target message in the target task file in response to the message read request, and send it to the consumer.

[0186] When reading messages in a message queue, the message queue program determines the target task file where the message to be read is located. This is only a preliminary step. It is also necessary to determine the specific location of the message to be read in the target task file and return it to the consumer after reading it.

[0187] Further, the queue file where the message to be processed is located and the specific location in the queue file can be determined through the location information of the message to be read. However, when reading the message to be read, the read message cannot be expressed by a location information. The message to be read is stored in a space in the queue file. If the message to be read is to be read, the content in the queue file needs to be specifically identified. In this embodiment, the specific implementation method is as follows:

[0188] The length byte of the target message is read in the target task file, and the target length information of the target message is determined based on the length byte; based on the target length information, a corresponding byte sequence is read in the target task file to form the target message.

[0189] In step S306, two methods for determining the target task file where the message to be read is located are described. Based on this, two methods for determining the specific location of the message to be read in the target task file are also extended. The specific methods are as follows:

[0190] If the target task file where the message to be read is located is determined based on the byte range of the message to be read in the message reading request, when reading the message to be read in the target task file, it is also necessary to continue to query the specific location information of the message to be read in the target task file based on this byte range information.

[0191] Specifically, after verifying the file identifier in the queue file header and determining the method of reading the queue file according to the version information of the queue file, bytes of a predetermined length of bytes are sequentially read after the file identifier to determine the range of the current message in the queue file, and this range is compared with the byte range of the message to be processed. If they are different, it proves that this message is not a message to be processed, and this message is skipped. The above verification process is repeated until the message to be processed is selected, the message to be processed is read, and the latest location information is returned to the consumer.

[0192] For example, in the processing scenario of APP logs, the user's login information is stored in the message queue. At this time, the real-time computing program needs to consume the user's login information. The message queue program receives the consumption request of the real-time computing program. After determining the queue file where the user's login information is stored, the message queue program first reads the first 16 bits of the queue file in this queue file. These 16 bits are the file identifier of the queue file, which is used to determine the version information of the queue file. According to the queue file version information in the file identifier, the reading method is selected. After determining the reading method that matches the version of the current queue file, 4 bits are read after the 16-bit file identifier. These 4 bits are the length bytes of the first message information stored in the queue file. This length byte determines the range of the message bytes. If this range is not the address of the message to be read, the byte length marked by the length byte is skipped after the length byte, that is, the message bytes of this message are skipped, and the next message is read until the message to be read is selected. The message byte range of the message to be read is determined according to the length byte of the message to be read, and the message bytes are read according to this range and returned to the real-time computing program.

[0193] If the target task file where the message to be read is located is determined based on the target timestamp of the message to be read in the message reading request, when reading the message to be read in the target task file, it is also necessary to continue to query the specific location information of the message to be read in the target task file based on this target timestamp.

[0194] Specifically, after selecting the target time file, the message queue program traverses each message stored in the target time file. First, it needs to verify the file identifier of the queue file and select a reading method that matches the queue file version based on the file identifier. Then, it reads the length byte of the first message to determine the storage range of the first message. After determining the range, the message bytes are read sequentially. The message generation time is determined in the message header string of the message bytes and compared with the time information recorded in the target timestamp. If they are the same, the message is determined to be the message to be read; if they are different, the message is skipped and subsequent messages are read sequentially until the message information with the same time information as the target timestamp is found in the queue file. After finding the message to be read, the message to be read is read, and the message content information, message location information, and time information of the message to be read are returned to the consumer.

[0195] For example, in the processing scenario of APP logs, if the required user login information needs to be read in a time-backward manner. In the determined queue file, the first 16 bits are read sequentially from the head of the queue file. These 16 bits are the file identifier of this queue file. Verify whether the file identifier is correct, and determine the reading method based on the file version information in the file identifier. Then continue to read 4 bits after the 16-bit file identifier. These 4 bits are the length bytes of the first message information stored in the queue file. This length byte determines the range of the message bytes. Continue to read the message bytes after the length byte. The header of the message bytes is the message header string. If the first 16 bits of the message header string record the message generation time of this message, then read these 16 bits and compare them with the time information T1 recorded by the target timestamp to determine whether this message is the user login information to be read. If not, skip the byte length marked by the length byte after the length byte, that is, skip the message bytes of this message, read the next message, until the message to be read is selected, and determine the message byte range of the message to be read based on the length byte of the message to be read. Read the message bytes based on this range and return to the real-time calculation program, and return the position information and time information of this message.

[0196] In summary, the length byte of the target message shows the specific character range of the message stored in the queue file. According to this range, the target message can be accurately located, which reduces the process of message position indexing and improves the accuracy of indexing.

[0197] The message reading method provided in the present application receives a message reading request from a consumer, determines a target task file corresponding to the message location information in a message queue based on the message location information in the message reading request, reads a target message from the target task file in response to the message reading request, and sends the message to the consumer. When reading the target message, the queue file corresponding to the target message can be directly found by using the byte range of the message to be read in the message queue or by using a time backtracking method, thereby eliminating the need for an index file, reducing the memory pressure of the system, and making the message reading process more concise and fast.

[0198] This embodiment combines Figure 4 , taking the entire process of a game data collector sending received game data to a message queue as an example, the message processing method is further explained. Figure 4 This is a processing flow chart of a message processing method applied to a data storage scenario provided by an embodiment of the present application, which specifically includes the following steps:

[0199] S402: Locking the write interface of the message queue.

[0200] Specifically, locking can be implemented using Java's built-in read-write lock ReentrantReadWriteLock. This lock can achieve that when the write lock is locked, the read lock will be blocked. After the write lock is released, multiple read threads can concurrently acquire read locks and concurrently read message queue files. Multiple consumers can concurrently read message queues. When multiple write threads request write locks, they are granted write locks in sequence according to the order in which they request locks. Write locks are mutually exclusive, and only one write thread has a write lock at the same time. After writing is completed, the write lock is released, and other write lock request threads or read lock request threads can continue to request locks at this time. Read locks and write locks are mutually exclusive. There can be multiple concurrent read locks, but there is only one write lock.

[0201] S404: Obtain the producer's write request.

[0202] Specifically, the game data collector collects the user login information in the game. At this time, the user login information needs to be written into the message queue. The write request must first be sent through the locked message queue write interface.

[0203] S406: Read the queue file version character and the supplementary information character based on the to-be-executed message carried in the write request, and establish the file identifier according to the queue file version character and the supplementary information character.

[0204] Specifically, the queue file version character can be a special string of 16 characters. The first 8 bytes record the additional information character, which is a special string of 8 characters, such as: WhaleQQQ, which is used to mark the queue file unique to this message queue program. The last 8 bytes record the file version character. The first 4 bytes of the last 8 bytes are of int type, representing the file version number. Each time the queue program reads a file part that changes, this part is increased by 1, starting from 1. The last 4 bytes are also of int type, reserved for subsequent upgrades and expansions, and the value is temporarily set to -1.

[0205] The file identifier is located at the beginning of each queue file in the message queue. For details, please see the attached figure of the specification. Figure 2 .

[0206] S408: Determine the corresponding message byte according to the to-be-executed message carried in the write request.

[0207] Specifically, determine the message header string and the message content data string corresponding to the user login information, and concatenate the message header string and the message content data string to form the message bytes corresponding to the message to be executed, wherein the message header string includes at least one of the following: message ID, message type, message code, message generation time, queue program receiving time, checksum, and message content data string length. The message content data string is the specific content of the written user login information.

[0208] S410: Generate a corresponding length byte according to the length of the message byte.

[0209] Calculate the length of the above message byte, that is, the number of bytes occupied by the message byte, and input the calculated number of characters occupied by the message byte into the length byte occupying 4 bytes.

[0210] S412: After the identification byte corresponding to the file identifier, sequentially concatenate the length byte and the message byte as a response to the write request.

[0211] The length byte is before the message byte, which is convenient for reading the message. The length byte is read first to determine the range of the message byte. For details, please see the attached figure of the manual. Figure 2 .

[0212] S414: Determine the message writing result.

[0213] Specifically, if the message is written successfully, the writing position and writing length of the written message will be returned to the current game data collector; if the writing fails, the failure information will be returned to the current game data collector.

[0214] S416: If the current write request is successful, a subsequent second write request is executed sequentially.

[0215] Specifically, a write request for a second user login message is received, and a second length byte and a second message byte corresponding to the message to be executed carried in the second write request are obtained.

[0216] S418: Determine whether the space in the current queue file can store the to-be-executed information carried in the second write request.

[0217] Specifically, it is determined whether the byte length of the second length byte and the second message byte plus the length of the written message in the current file is greater than a preset length threshold of the target queue file; the length threshold is manually set, and is set to 100M here. If the space in the queue file can be stored, step S420 is executed; if the space in the queue file cannot be stored, step S424 is executed.

[0218] S420: Write the second length byte and the second message byte into the target queue file.

[0219] Specifically, the corresponding second message byte is determined according to the to-be-executed message carried in the second write request, and the corresponding second length byte is generated according to the length of the second message byte; after the current message byte, the second length byte and the second message byte are sequentially spliced ​​as a response to the second user login information write request, wherein the process of writing the to-be-executed message carried in the second write request to the queue file is similar to the previous steps, except that the generated second length byte and the second message byte are sequentially spliced ​​after the previous write request response. For details, please refer to the attached figure of the specification. Figure 2 .

[0220] S422: Before creating the file, assign a file name to the file.

[0221] Specifically, obtain the target file name and the target file length of the target queue file; add the target file name and the target queue length as the file name of the newly created file, wherein the file name of the first queue file in the message queue can set the preset target character to the Long type number 0, and the suffix is ​​.pqdata. The file name of the subsequent queue file is determined according to the file name of the previous queue file and the file length of the previous file. For details on the naming of each queue file, please refer to the attached figure of the specification. Figure 2 .

[0222] S424: Create a new file, and write the to-be-executed message carried in the second write request into the new file.

[0223] Specifically, a new file is created, and a second file identifier is established at the starting position of the new file based on the second write request; a corresponding second message byte is determined according to the message to be executed carried in the second write request, and a corresponding second length byte is generated according to the length of the second message byte; after the second file identifier, the second length byte and the second message byte are sequentially spliced ​​as a response to the second write request.

[0224] Compared with step S420, the second write request at this time needs to be written into the newly created file, and the length byte and message byte generated according to the message to be written carried in the second write request are sequentially spliced ​​after the second file identifier of the newly created file. For details, please see the attached figure of the specification Figure 2 How the first message in a queue file is stored except for the first file 0.pqdata.

[0225] It should be noted that the game data collector writes the collected data into the message queue in sequence. The handle of the current write file is kept open inside the message queue. After the current message is written successfully, if it is determined that there is no need to create a new queue file, the next message can be written using the file handle written last time. In other words, there is no need to find the next write position based on the message position fed back after the successful writing of the message. It can be written sequentially directly after the current write position.

[0226] After writing the message to be processed into the queue file, a specific message reading part is also included. In this embodiment, the message is read by the real-time computing program as a consumer, and the message information in the message queue is read sequentially. When reading a message in the message queue, the position information of the current message in the message queue is recorded, and the next time the message is read, the reading continues from the position marked by the position information. The specific reading content is as follows:

[0227] S426: Receive the consumer's message reading request.

[0228] Specifically, the real-time computing program acts as a consumer to read the user login message in the message queue, and first obtains the read lock by locking the read interface.

[0229] S428: Determine message receiving location information based on the message reading request;

[0230] Specifically, after the message reading request of the real-time computing program is received, the message queue program starts to read the user login message in the message queue. When the message queue program reads the message in the message queue for the first time, it starts reading data from 0.pqdata. If the real-time computing program is not consuming the data in the message queue for the first time, the next message in the message queue is read sequentially in the message queue according to the position indicated by the message position information saved during the last consumption.

[0231] In actual application, it is assumed that the first queue file in the message queue is 0.pqdata, the first 16 bytes are the file identifier, the 17th to 20th bytes are the length bytes of the first message, if the value of this byte length is 80, then the 21st to 100th bytes are the message bytes of the first message, and the following 4 bytes, namely the 101st to 104th bytes are the length bytes of the second message, if the value of this byte length is 96, then the 105th to 200th bytes are the message bytes of the second message. If the real-time computing program consumes the message in the message queue for the first time, the message queue program that receives the consumption request of the real-time computing program starts reading from the 0.pqdata file, first reads the first 16 bits of the file identifier and verifies it, then reads the 17th to 20th bytes, determines that the byte interval occupied by the first message in the queue file is 17-100 bytes through the value 80 of the 4-bit length byte, reads the message bytes in the range of 21st to 100 bytes, and returns the message information and message position to the real-time computing program.

[0232] The real-time computing program consumes the next message sequentially. At this time, the real-time computing program is obviously no longer consuming the data in the message queue for the first time. According to the message position 17-100 bytes of the last consumed message saved, the next message is consumed sequentially. After the message queue program receives the consumption request from the real-time computing program, the message queue program starts reading from byte 101, first reading the four bytes in the range of 101-104 bytes. According to the byte length of the second message recorded in these four bytes, if the value of these four bytes is 96, then the message bytes of the second message occupy the byte range of 105-200 in the queue file. The message byte content of the second message in bytes 105-200 is read, and the content and position of the second message are returned to the real-time computing program.

[0233] S430: Determine the target task file corresponding to the message location information in the message queue;

[0234] Specifically, after determining the message location information, the real-time computing program sends the consumption location information to the message queue program. Then the message queue program needs to traverse the file names of the queue files in the message queue. Since the file names of the queue files record the character interval occupied by the queue files in the message queue, it is determined that the position indicated by the message location information exists within the character interval occupied by which queue file, and this queue file is determined as the target task file.

[0235] In actual applications, assuming that the file name of the first file in the message queue is 0.pqdata and the file name of the second file is 104857600.pqdata, according to the file names of the two files, the byte range occupied by the first queue file in the message queue is 1-104857600 bytes. If the message location information identifies that the bytes occupied by the message in the message queue are 101-200, the range is obviously within the byte range of the first queue file. It can be known that the message exists in the first queue file, and the first queue file is determined to be the target task file.

[0236] S432: Find the message at the specified position from the queue file

[0237] Specifically, the message queue program reads the queue file sequentially. During the reading process, the first 16 bits of the queue file are first read sequentially to verify the file identifier of the queue file, determine that the queue file is normal and select the reading method according to the version of the queue file, and then read the messages to be read sequentially according to the message position information saved in the real-time computing program. After the reading is completed, the latest message position and message length are returned to the real-time computing program.

[0238] It should be noted that the reading and writing of messages are similar. During the continuous message reading process, the queue program also keeps the handle of each consumer's read file open. When it is determined that the consumption position returned after the previous message reading is the same as the current reading position, there is no need to start from the beginning to find which file and position the message is located in. Subsequent messages can be read sequentially. Only when it is determined that the returned consumption position is different from the current reading position, it is necessary to re-determine the file and the message position in the file.

[0239] In actual applications, if the real-time computing program executes another task in the middle of consuming the queue file, and continues to consume the message queue file after the other task is completed, the message queue program queries the message position of the last read message saved in the real-time computing program as 17-100. After judgment, it is determined that it is in the first queue file of the message queue. First, the first 16 bits of the message queue are read, the file identifier of the queue file is verified and the file version is determined. The reading method is determined according to the file version, and then the 101st-104th bits of the message queue are read to determine the byte range occupied by the message to be read. If the byte occupation range is 101-200, the message content within this range is read and the message content and message length are returned to the real-time computing program.

[0240] In addition to the above message consumption methods, the real-time computing program also has a method of consuming messages based on time backtracking consumption positions. When the real-time computing program needs to go back to a certain point in time to re-consume data, the message reading operation can be performed according to the time backtracking consumption position in response to the real-time computing program going back to a certain point in time to re-consume.

[0241] For example, the user login information in the message queue is still consumed, but at this time it is found that the user login is abnormal. It is necessary to use the time-based backtracking message position to read the previous user login information to analyze the cause of the user login abnormality.

[0242] Specifically, the real-time computing program is the consumer. After the message queue program receives the message read request, it first obtains the read lock by locking the read interface, and then obtains the target timestamp T1 according to the consumer request, and selects the queue file closest to the target timestamp T1 and whose last modification time TX is greater than the target timestamp T1 from all queue files.

[0243] Next, read the first 16 bits of the queue file, verify the file identifier, and select a reading method based on the file version information in the last 8 bits of the file identifier. If the verification is successful, read the messages in the queue file one by one.

[0244] In the process of reading the messages in the queue file one by one, the first 4 bytes are read first to determine the length byte of the message, and the range of characters occupied by the message in the queue file is 17-100. Then the message header string in the message bytes is read to find the message receiving time information T2 in the message header string, and compare the message receiving time information T2 with the target timestamp T1. If the time recorded in the message receiving time information is less than the time of the target timestamp, skip this message. Read 101-104 bytes, these 4 bytes are the length bytes of the second message. If the value of this length byte is 96, it is determined that the character range occupied by the second message in the queue file is 101-200. Then read the message header string in the message bytes of the second message, find the message receiving time information in the message header string, if the time T3 recorded in the message receiving time information is greater than the time T1 of the target timestamp, return the position of the previous message of this message. The previous message is the user login information found by the real-time computing program. The message queue program reads this user login information, and returns the message content and position information to the real-time computing program after reading.

[0245] In summary, a message to be stored is stored and read in this form, which saves the index file and reduces the memory pressure of the system. In the process of reading the message, it also saves the process of referencing the index file, making the message reading process more concise and fast, and providing a good environment for reading and querying the message.

[0246] Corresponding to the above method embodiment, the present application also provides a message processing device embodiment, Figure 5 FIG. 1 shows a schematic diagram of the structure of a message processing device provided by an embodiment of the present application. Figure 5 As shown, the device comprises:

[0247] An acquisition module 502 is configured to acquire a first write request from a producer, and establish a file identifier at a start position of a target queue file based on the first write request;

[0248] The determination module 504 is configured to determine a corresponding message byte according to the to-be-executed message carried in the first write request, and generate a corresponding length byte according to the length of the message byte;

[0249] The splicing module 506 is configured to sequentially splice the length byte and the message byte after the identification byte corresponding to the file identifier as a response to the first write request.

[0250] In an optional embodiment, the message processing device further includes:

[0251] A write module is configured to receive a second write request, wherein the second write request is a write request that is adjacent in time sequence and follows the first write request; obtain a second length byte and a second message byte corresponding to the message to be executed carried in the second write request; determine whether the byte length of the second length byte and the second message byte plus the length of the write message in the current file is greater than a preset length threshold of the target queue file; if not, write the second length byte and the second message byte into the target queue file; if so, create a new file and write the message to be executed carried in the second write request into the new file.

[0252] In an optional embodiment, the writing module is further configured to:

[0253] Determine the corresponding second message byte according to the to-be-executed message carried in the second write request, and generate the corresponding second length byte according to the length of the second message byte; after the current message byte, sequentially splice the second length byte and the second message byte as a response to the second write request.

[0254] In an optional embodiment, the writing module is further configured to:

[0255] Obtain the target file name and the target file length of the target queue file;

[0256] The result of adding the target file name to the target queue length is used as the file name of the newly created file, wherein the target queue file is named based on preset target characters and file suffix.

[0257] Create a new file, and establish a second file identifier at the starting position of the new file based on the second write request; determine the corresponding second message byte according to the to-be-executed message carried in the second write request, and generate the corresponding second length byte according to the length of the second message byte; after the second file identifier, sequentially splice the second length byte and the second message byte as a response to the second write request.

[0258] In an optional embodiment, the acquisition module 502 is further configured to:

[0259] Based on the to-be-executed message carried in the first write request, the queue file version character and the supplementary information character are read; the file identifier is established according to the queue file version character and the supplementary information character; wherein the queue version character is used to mark the version information of the current file, and the supplementary information character is used to mark a unique queue file in the message queue.

[0260] In an optional embodiment, the determining module 504 is further configured to:

[0261] Determine a message header string and a message content data string corresponding to the message to be executed; and concatenate the message header string and the message content data string to form a message byte corresponding to the message to be executed.

[0262] In an optional embodiment, the message processing device further includes:

[0263] The locking module is configured to lock the write interface of the message queue; accordingly, obtaining the producer's write request includes: obtaining the write request submitted by the producer through the locked write interface; wherein, if there are at least two producers submitting write requests through the locked write interface, each write request is processed in sequence according to the write time.

[0264] The message processing device provided by the present application receives a write request sent by a producer, and establishes a file identifier at the beginning of a queue file based on the content of the write request, determines the message byte and the length byte of the message byte according to the content of the message to be processed carried in the write request, and stores the length byte and the message byte in sequence in the message queue. A message information to be stored is stored in this form, and when reading a target message, the queue file corresponding to the target message can be directly found in combination with the file name of the queue file in the message queue, eliminating the index file and reducing the memory pressure of the system. In the process of reading the message, the process of referencing the index file can also be omitted, making the message reading process more concise and fast, and providing a good environment for reading and querying messages.

[0265] It should be noted that the technical solution of the message processing device and the technical solution of the above-mentioned message processing method belong to the same concept. For details not described in detail in the technical solution of the message processing device, please refer to the description of the technical solution of the above-mentioned message processing method. In addition, the various components in the device embodiment should be understood as functional modules that must be established to implement each step of the program flow or each step of the method, and each functional module is not an actual functional division or separation. The device claim defined by such a group of functional modules should be understood as a functional module architecture that mainly implements the solution through the computer program recorded in the specification, and should not be understood as a physical device that mainly implements the solution through hardware.

[0266] Corresponding to the above method embodiment, the present application also provides a message reading device embodiment, Figure 6 FIG. 1 is a schematic diagram showing the structure of a message reading device provided by an embodiment of the present application. Figure 6 As shown, the device comprises:

[0267] The receiving module 602 is configured to receive a message reading request from a consumer;

[0268] A location determination module 604 is configured to determine message location information based on the message reading request;

[0269] The file determination module 606 is configured to determine the target task file corresponding to the message location information in the message queue;

[0270] The reading module 608 is configured to read the target message from the target task file in response to the message reading request and send the target message to the consumer.

[0271] In an optional embodiment, the reading module 608 is further configured to:

[0272] The length byte of the target message is read in the target task file, and the target length information of the target message is determined based on the length byte; based on the target length information, a corresponding byte sequence is read in the target task file to form the target message.

[0273] In an optional embodiment, the message reading device further includes:

[0274] The identifier verification module obtains the file identifier of the target queue file; verifies the file version characters and the attached information characters in the file identifier; if the verification passes, sequentially executes subsequent message reading tasks; if the verification fails, ends the process and returns an error message to the consumer.

[0275] In an optional embodiment, the file determination module 606 is further configured to:

[0276] Determine the task file name contained in the message location information; determine the target task file in the message queue based on the task file name, or

[0277] Determine the byte position of the target message recorded in the message position information in the message queue; call the message queue program to determine the queue file names of the queue files in the message queue one by one; determine the byte range of each queue file in the message queue according to the queue file name; determine the target task file or

[0278] Determine the target timestamp of the target message recorded in the message location information; call the message queue program to determine the creation time of the queue files in the message queue one by one; based on the target timestamp and the creation time of each queue file, determine the target task file where the target message is located.

[0279] It should be noted that the technical solution of the message reading device and the technical solution of the above-mentioned message reading method belong to the same concept. For details not described in detail in the technical solution of the message reading device, please refer to the description of the technical solution of the above-mentioned message reading method. In addition, the various components in the device embodiment should be understood as functional modules that must be established to implement each step of the program flow or each step of the method, and each functional module is not an actual functional division or separation. The device claim defined by such a group of functional modules should be understood as a functional module architecture that mainly implements the solution through a computer program recorded in the specification, and should not be understood as a physical device that mainly implements the solution through hardware.

[0280] Figure 7The structure block diagram of a computing device 700 provided according to an embodiment of the present application is shown. The components of the computing device 700 include but are not limited to a memory 710 and a processor 720. The processor 720 is connected to the memory 710 via a bus 730, and the database 750 is used to store data.

[0281] The computing device 700 also includes an access device 740 that enables the computing device 700 to communicate via one or more networks 760. Examples of these networks include a public switched telephone network (PSTN), a local area network (LAN), a wide area network (WAN), a personal area network (PAN), or a combination of communication networks such as the Internet. The access device 740 may include one or more of any type of network interface (e.g., a network interface card (NIC)) whether wired or wireless, such as an IEEE 802.11 wireless local area network (WLAN) wireless interface, a World Wide Interoperability for Microwave Access (Wi-MAX) interface, an Ethernet interface, a universal serial bus (USB) interface, a cellular network interface, a Bluetooth interface, a near field communication (NFC) interface, and the like.

[0282] In one embodiment of the present application, the above components of the computing device 700 and Figure 7 Other components not shown in the figure may also be connected to each other, for example, via a bus. It should be understood that Figure 7 The computing device structure block diagram shown is only for the purpose of illustration, and is not intended to limit the scope of the present application. Those skilled in the art may add or replace other components as needed.

[0283] The computing device 700 may be any type of stationary or mobile computing device, including a mobile computer or mobile computing device (e.g., a tablet computer, a personal digital assistant, a laptop computer, a notebook computer, a netbook, etc.), a mobile phone (e.g., a smart phone), a wearable computing device (e.g., a smart watch, smart glasses, etc.), or other types of mobile devices, or a stationary computing device such as a desktop computer or a PC. The computing device 700 may also be a mobile or stationary server. The processor 720 is used to implement the message processing method when executing the following computer executable instructions.

[0284] The above is a schematic scheme of a computing device of this embodiment. It should be noted that the technical scheme of the computing device and the technical scheme of the above-mentioned message processing method or message reading method belong to the same concept, and the details not described in detail in the technical scheme of the computing device can be referred to the description of the technical scheme of the above-mentioned message processing method or message reading method.

[0285] An embodiment of the present application further provides a computer-readable storage medium storing computer instructions, which are used for the message processing method or the message reading method when executed by a processor.

[0286] The above is a schematic scheme of a computer-readable storage medium of this embodiment. It should be noted that the technical scheme of the storage medium and the technical scheme of the message processing method or message reading method described above belong to the same concept, and the details not described in detail in the technical scheme of the storage medium can be referred to the description of the technical scheme of the message processing method or message reading method described above.

[0287] An embodiment of the present application further provides a chip storing a computer program, which implements the steps of the message processing method or the message reading method when executed by the chip.

[0288] The above describes specific embodiments of the present application. Other embodiments are within the scope of the appended claims. In some cases, the actions or steps recorded in the claims can be performed in an order different from that in the embodiments and still achieve the desired results. In addition, the processes depicted in the accompanying drawings do not necessarily require the specific order or continuous order shown to achieve the desired results. In some embodiments, multitasking and parallel processing are also possible or may be advantageous.

[0289] The computer instructions include computer program codes, which may be in source code form, object code form, executable files or some intermediate forms, etc. The computer-readable medium may include: any entity or device capable of carrying the computer program code, recording medium, USB flash drive, mobile hard disk, magnetic disk, optical disk, computer memory, read-only memory (ROM), random access memory (RAM), electric carrier signal, telecommunication signal and software distribution medium, etc. It should be noted that the content contained in the computer-readable medium may be appropriately increased or decreased according to the requirements of legislation and patent practice in the jurisdiction. For example, in some jurisdictions, according to legislation and patent practice, computer-readable media do not include electric carrier signals and telecommunication signals.

[0290] It should be noted that, for the above-mentioned method embodiments, for the sake of simplicity of description, they are all expressed as a series of action combinations, but those skilled in the art should be aware that the present application is not limited by the described action sequence, because according to the present application, certain steps can be performed in other sequences or simultaneously. Secondly, those skilled in the art should also be aware that the embodiments described in the specification are all preferred embodiments, and the actions and modules involved are not necessarily required by the present application.

[0291] In the above embodiments, the description of each embodiment has its own emphasis. For parts that are not described in detail in a certain embodiment, reference can be made to the relevant descriptions of other embodiments.

[0292] The preferred embodiments of the present application disclosed above are only used to help explain the present application. The optional embodiments do not describe all the details in detail, nor do they limit the invention to the specific implementation methods described. Obviously, many modifications and changes can be made according to the content of the present application. The present application selects and specifically describes these embodiments in order to better explain the principles and practical applications of the present application, so that those skilled in the art can understand and use the present application well. The present application is only limited by the claims and their full scope and equivalents.

Claims

1. A message processing method, characterized in that: include: Obtaining a first write request from the producer, and establishing a file identifier at the start position of a target queue file based on the first write request, wherein the end position of the target queue file in the message queue is the file name of the next queue file of the target queue file, and the end position of the target queue file in the message queue is determined by adding a target file name of the target queue file and a target file length of the target queue file, and the target file length is the number of bytes occupied by the target queue file in the message queue; Determine a corresponding message byte according to the to-be-executed message carried in the first write request, and generate a corresponding length byte according to the length of the message byte; After the identification byte corresponding to the file identifier, the length byte and the message byte are sequentially spliced ​​as a response to the first write request.

2. The method according to claim 1, characterized in that After the step of sequentially splicing the length byte and the message byte as a response to the first write request is performed, the step further includes: receiving a second write request, wherein the second write request is a write request that is adjacent in time sequence and is subsequent to the first write request; Obtaining a second length byte and a second message byte corresponding to the message to be executed carried in the second write request; Determine whether the byte length of the second length byte and the second message byte plus the length of the written message in the current file is greater than a preset length threshold of the target queue file; If not, writing the second length byte and the second message byte into the target queue file; If so, create a new file, and write the to-be-executed message carried in the second write request into the new file.

3. The method according to claim 2, characterized in that Writing the second length byte and the second message byte into the target queue file comprises: Determine a corresponding second message byte according to the to-be-executed message carried in the second write request, and generate a corresponding second length byte according to the length of the second message byte; After the current message byte, the second length byte and the second message byte are sequentially spliced ​​as a response to the second write request.

4. The method according to claim 2, characterized in that: The step of creating a new file and writing the to-be-executed message carried in the second write request into the new file includes: Obtain the target file name and the target file length of the target queue file; The result of adding the target file name and the target file length is used as the file name of the newly created file, wherein the target queue file is named based on preset target characters and file suffix; Creating a new file, and establishing a second file identifier at a start position of the new file based on the second write request; Determine a corresponding second message byte according to the to-be-executed message carried in the second write request, and generate a corresponding second length byte according to the length of the second message byte; After the second file identifier, a second length byte and a second message byte are sequentially spliced ​​as a response to the second write request.

5. The method according to claim 1, characterized in that The step of establishing a file identifier at a start position of a target queue file based on the first write request comprises: Reading a queue file version character and ancillary information characters based on the to-be-executed message carried in the first write request; Establishing the file identifier according to the queue file version character and the supplementary information character; The queue version character is used to mark the version information of the current file, and the supplementary information character is used to mark a unique queue file in the message queue.

6. The method according to claim 1, characterized in that The determining the corresponding message byte according to the to-be-executed message carried in the first write request includes: Determine a message header character string and a message content data character string corresponding to the message to be executed; The message header character string and the message content data character string are concatenated to form the message bytes corresponding to the message to be executed.

7. The method according to claim 1, characterized in that Before the step of obtaining the first write request of the producer is executed, the method further includes: Lock the write interface of the message queue; Correspondingly, obtaining the producer's write request includes: Obtaining the write request submitted by the producer through the locked write interface; Among them, if there are at least two producers submitting write requests through the locked write interface, each write request will be processed in sequence according to the write time.

8. A message reading method, characterized in that: include: Receive message reading requests from consumers; Determining message location information based on the message reading request; Determine a target task file corresponding to the message position information in a message queue, wherein the termination position of the target task file in the message queue is the file name of the next queue file of the target task file, and the termination position of the target task file in the message queue is determined by adding the file name of the target task file and the length of the target task file, and the length of the target task file is the number of bytes occupied by the target task file in the message queue; In response to the message read request, the target message is read from the target task file and sent to the consumer.

9. The method according to claim 8, characterized in that The step of reading a target message in the target task file in response to the message reading request comprises: Reading a length byte of the target message in the target task file, and determining target length information of the target message based on the length byte; Based on the target length information, a corresponding byte sequence is read in the target task file to form the target message.

10. The method according to claim 8, characterized in that After determining the target task file corresponding to the message location information in the message queue, the method further includes: Obtaining a file identifier of the target task file; Verifying the file version characters and the supplementary information characters in the file identifier; If the verification is successful, the subsequent message reading tasks are executed sequentially; If the verification fails, the process ends and the error message is returned to the consumer.

11. The method according to claim 8, characterized in that The step of determining the target task file corresponding to the message location information in the message queue includes: Determine the task file name contained in the message location information; determine the target task file in the message queue based on the task file name, or Determine the byte position of the target message recorded in the message position information in the message queue; call the message queue program to determine the queue file names of the queue files in the message queue one by one; determine the byte range occupied by each queue file in the message queue according to the queue file name; determine the target task file where the target message is located based on the byte position and the byte range, or Determine the target timestamp of the target message recorded in the message location information; call the message queue program to determine the creation time of the queue files in the message queue one by one; based on the target timestamp and the creation time of each queue file, determine the target task file where the target message is located.

12. A message processing device, characterized in that: include: an acquisition module, configured to acquire a first write request from a producer, and to establish a file identifier at a start position of a target queue file based on the first write request, wherein the end position of the target queue file in the message queue is the file name of a queue file next to the target queue file, and the end position of the target queue file in the message queue is determined by adding a target file name of the target queue file and a target file length of the target queue file, and the target file length is the number of bytes occupied by the target queue file in the message queue; a determination module configured to determine a corresponding message byte according to the to-be-executed message carried in the first write request, and to generate a corresponding length byte according to the length of the message byte; The splicing module is configured to sequentially splice the length byte and the message byte after the identification byte corresponding to the file identifier as a response to the first write request.

13. A message reading device, characterized in that: include: A receiving module, configured to receive a message reading request from a consumer; a location determination module, configured to determine message location information based on the message reading request; A file determination module is configured to determine a target task file corresponding to the message position information in a message queue, wherein the termination position of the target task file in the message queue is the file name of the next queue file of the target task file, and the termination position of the target task file in the message queue is determined by adding the file name of the target task file and the length of the target task file, and the length of the target task file is the number of bytes occupied by the target task file in the message queue; The reading module is configured to read the target message from the target task file in response to the message reading request and send the target message to the consumer.

14. A computing device, characterized in that include: Memory and processor; The memory is used to store computer-executable instructions, and the processor is used to execute the computer-executable instructions to implement the steps of the message processing method or message reading method described in any one of claims 1-7 or 8-11.

15. A computer-readable storage medium storing computer instructions, characterized in that: When the instruction is executed by the processor, the steps of the message processing method or message reading method described in any one of claims 1-7 or 8-11 are implemented.

16. A computer program product, characterized in that It includes computer instructions, which, when executed by a processor, implement the steps of the message processing method or message reading method described in any one of claims 1-7 or 8-11.

Images