[0061] The technical solutions in the embodiments of the present invention will be clearly and completely described below in conjunction with the accompanying drawings in the embodiments of the present invention. Obviously, the described embodiments are only a part of the embodiments of the present invention, rather than all the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those of ordinary skill in the art without creative work shall fall within the protection scope of the present invention.
[0062] The embodiment of the present invention provides an index file generation method, device and system. The following first describes the index file generation method provided in the embodiment of the present invention in detail.
[0063] In the first aspect, the index file generation method provided by the embodiment of the present invention is applied to a main device in an index file generation system, and the index file generation system further includes at least two service devices. The above-mentioned main device and service device may be the server of the server, or of course, the electronic device of the client. Introduce from another angle, the above-mentioned main equipment and service equipment can be a physical machine installed with an embedded system to realize a certain function, or a general-purpose machine.
[0064] It should be noted that the index file generation method provided by the embodiment of the present invention is applied to the main device in the index file generation system, and specifically, can be applied to the index file generation apparatus in the main device. When the main device is an electronic device of a client, the above-mentioned index file generating device can be specialized message management software or a functional plug-in of related software; when the main device is a server, the above-mentioned index file generating device can be a system that provides message management services , It can also be a system plug-in for related systems.
[0065] Such as figure 1 As shown, an index file generation method provided by an embodiment of the present invention, applied to a main device in an index file generation system, includes the following steps:
[0066] S101: Obtain all local log files used to store messages;
[0067] When multiple applications need to cooperate to complete a certain function, messages will be transmitted between the applications. If the transmitted messages cannot be processed in time, they will be stored in the log file. In the embodiment of the present invention, when the index file of the message needs to be generated, the master device can obtain the log file saved in the local disk.
[0068] Wherein, the obtained log file may be the original file of the storage directory. For example, the aforementioned log file can be a db.log file in the directory of KahaDB (the default persistent storage method of ActiveMQ); of course, it can also be a file in the backed-up directory after the aforementioned storage directory is backed up. For example, the db.log file in the KahaDB directory that was backed up.
[0069] The above message may be the content of mutual communication between applications on the same device, and of course it may also be the content of mutual communication between applications on different devices.
[0070] S102: group the log files according to preset rules and send them to each service device, so that each service device synchronously sends the messages in the received log file back to the main device;
[0071] Optionally, the foregoing preset rules may be related to the number of service devices, and may also be related to the operating status of the service devices, and the specific rules are preset by the staff according to actual needs.
[0072] In one implementation, the step of grouping log files according to preset rules may include: according to the number of service devices, dividing the log files into corresponding number of log groups; sending each group of log files to each service. equipment. For example, if the number of log files is 3000 and the number of service devices is 3, namely Device A, Device B, and Device C, the log files can be divided into 3 groups equally, each of which is 1000 log files in the first group, 1000 log files in the second group and 1000 log files in the third group. Then, the 1000 log files of the first group can be sent to device A, the 1000 log files of the second group can be sent to device B, and the 1000 log files of the third group can be sent to device C.
[0073] It should be noted that, in actual applications, there are situations where the above log files cannot be grouped equally, for example, 3100 log files are divided into 3 groups evenly. In this case, you can add the extra log files after the average group to any group. For example, after dividing 3100 log files into 3 groups (each group includes 1033 log files), one log file remains. In this case, you can add the remaining 1 log file to any group, so that the group includes 1034 log files.
[0074] In another implementation manner, the step of grouping log files according to preset rules may include: obtaining the current load of each service device, and according to the obtained current load of each service device and the total number of log files , Determine the number of log files to be sent to each service device; group the above log files according to the determined number of log files to be sent to each service device; send the log group containing the corresponding number of log files to the corresponding service device .
[0075] The load of each server mentioned above may be embodied as a CPU (Central Processing Unit, central processing unit) usage rate of each service device. The method of grouping log files can be: when the current load of a certain service device exceeds a certain value, a smaller number of logs can be sent to it. For example, when the utilization rate of device A is higher than 90%, the determined number of log files in the group sent to device A is divided into the first group and the second group, and the log files of the first group are sent to the device A. Send all (average grouping) of the log files of the second group to a device other than A (group corresponding number).
[0076] The method of grouping log files may also be: setting the proportion of the remaining load of each service device to be consistent with the proportion of the number of files sent to each device. For example, the index file includes five service devices A, B, C, D, and E, and the current remaining load is A: 40%, B: 90%, C: 30%, D: 50%, E: 20%, Add the current remaining load values of the above-mentioned service devices, and calculate the total value to be 230%, then the proportion of the remaining load of device A among all five service devices is which is Then the number of files allocated for device A accounts for all log files of the main device The proportion of the remaining load of device B among all five service devices is which is The number of files allocated to device B accounts for the total number of log files of the main device Similarly, the number of files allocated to device C accounts for all log files of the main device The number of files allocated for device D accounts for all log files of the main device The number of files allocated to device E accounts for all log files of the main device
[0077] After the log files are grouped, the main device sends the log files in each group to the corresponding service device. The process of sending log files by the main device is actually the process of file transfer. Specifically, qpress (a computer system used in Tools for file backup and migration) to move.
[0078] Each service device receives the log file sent by the main device, and synchronously sends the messages in the received log file back to the main device. Specifically, the service device can send the messages in the received log file back to the main device in the following ways: store the messages in the received log file, and generate an index file of the messages in the log file; When the message migration instruction is used, the producer is used to send the message back to the main device. The producer can be an application that can be used to produce messages, communicate with the master device, and send the produced messages to the master device.
[0079] The embodiment of the present invention does not limit the sending device of the message migration instruction, and only needs to ensure that each service device receives the message migration instruction at the same time, which further guarantees the synchronization of the message sent by each service device.
[0080] S103: Receive and store the messages sent by each service device, and at the same time generate an index file of the messages sent by each service device.
[0081] It can be understood that, since the steps in S102 enable each service device to synchronously send messages to the main device, the main device will receive multiple messages at the same time. For example, if 4 service devices send messages synchronously, the main device will receive 4 messages at the same time, so the index of 4 messages can be generated at the same time to obtain the index file.
[0082] Optionally, the method of generating the index file may be: receiving and storing the messages sent by each service device; obtaining the offset address of each message in parallel, and establishing a B-tree according to the obtained offset address; storing the B-tree in a new file , And determine the newly created file as an index file. Among them, B-tree is a non-linear data structure in computer science, used for data indexing.
[0083] The method of establishing the B-tree according to the offset address may be: grouping the messages according to the magnitude of the offset address value, and storing the messages in each grouping to the lowest node of different branches of the B-tree. For example, the offset address of the obtained message is shown in Table 1.
[0084] Table 1
[0085] news
[0086] When a value of 4 is preset as the dividing standard, messages a, b, c, and d with offset addresses not greater than 4 can be divided into the first group, and messages e, message f, and messages with offset addresses greater than 4 Message g and message h are divided into the second group. Store the above two sets of messages to the lowest node of the B-tree to get image 3 B-tree shown. In this way, when the message index is performed, it is assumed that the offset address 2 of the message b is obtained. When the message b is indexed according to the B tree, 2 is compared with the value 4 of the top node to obtain 2 <4, traverse message b in the node on the left (look for message b from 4 messages).
[0087] After the B-tree is established, it is stored in a file to obtain an index file.
[0088] An index file generating method provided by an embodiment of the present invention includes: allocating a log file for storing messages on a main device to at least two service devices in an index file generating system, so that each service device synchronizes all of its The received messages stored in the log file grouped by the master device are re-sent back to the master device. Since each service device sends messages to the master device synchronously, the master device receives the same amount (or more than the service device) at the same time. After receiving and storing the above-mentioned number of messages, the master device can establish an index of the same number of messages at the same time. Compared with the prior art method of scanning the messages in the daily files one by one and building indexes one by one, the method provided by the embodiment of the present invention can speed up the generation of index files, shorten the time for generating the index files, and avoid the cost of ActiveMQ in the prior art The problem of creating index files for a long time and affecting the normal sending and receiving of messages ensures the real-time performance of ActiveMQ services.
[0089] In the second aspect, the embodiment of the present invention also provides an index file generation method, which is applied to any service device in the index file generation system, and the index file generation system further includes a main device. For the introduction of main equipment and service equipment, please refer to the introduction in the first aspect.
[0090] It should be noted that the index file generation method provided by the embodiment of the present invention is applied to the service equipment in the index file generation system, and specifically, can be applied to the index file generation apparatus in the main equipment. When the above-mentioned service device is an electronic device of a client, the above-mentioned index file generating device may be specialized message management software or a functional plug-in of related software; when the above-mentioned service device is a server, the above-mentioned index file generating device may provide information The service management system can also be a system plug-in of the related system.
[0091] Such as figure 2 As shown, an index file generation method provided by an embodiment of the present invention, applied to a service device in an index file generation system, includes the following steps:
[0092] S201: Receive a log file sent by the main device in the index file generation system, where the log file is the main device grouping all log files locally used for storing messages according to a preset rule and then sending it to the service equipment;
[0093] When there is a need to generate an index file, the main device in the index file generation system groups the log files storing messages and sends them to each service device, so that each service device synchronously sends the messages in the received log file back to the main device . Therefore, each service device in the above-mentioned system receives the log file to send the message in the received log file back to the main device, and the main device can store the message again.
[0094] The received log file may be the original file in the storage directory on the main device. For example, the log file may be the db.log file in the KahaDB (ActiveMQ default persistent storage mode) directory stored on the main device. ; Of course, it can also be the files in the backed-up directory after the main device backs up the above-mentioned storage directory, for example, the db.log file in the backed-up KahaDB directory.
[0095] The above message may be the content of mutual communication between applications on the same device, and of course it may also be the content of mutual communication between applications on different devices.
[0096] The preset rules on which the main equipment is grouped may be related to the number of service equipment, and may also be related to the operating status of the service equipment. The specific rules are preset by the staff according to actual needs.
[0097] S202: Receive a message migration instruction, and send the message to be migrated targeted by the message migration instruction in the received log file to the master device, so that the master device stores the message to be migrated.
[0098] Each service device receives the message migration instruction to send the message to be migrated targeted by the message migration instruction back to the main device. Specifically, it may include the following steps: receiving a message migration instruction, and obtaining producers and consumers of messages to be migrated targeted by the message migration instructions, where the messages to be migrated are messages in the received log file, and consumers are acquiring messages and The obtained message is sent to the application program of the producer, and the producer is the application program that receives the message sent by the consumer, produces the message and sends the produced message to the main device; runs each producer and consumer to make each The producer produces messages and sends the produced messages to the master device.
[0099] Among them, the producer and the consumer can be an application on the service device, or can be an application on other devices that communicate with the service device. The process of the consumer deleting (consuming) the message and the producer producing the message is based on the prior art, which is not limited in the embodiment of the present invention.
[0100] The message migration instruction can be for a certain message or for multiple messages.
[0101] In an index file generation method provided by an embodiment of the present invention, after receiving a message migration instruction, a service device in the index file generation system sends the message to be migrated in the received log file targeted by the message migration instruction back to the host Device, because the index file generation system includes at least two service devices and, therefore, at least two service devices send messages to the main device at the same time, and the main device can receive and store at least two messages at the same time. Create an index of at least two messages. Compared with the prior art method of scanning the messages in the log file one by one and establishing the index of the messages one by one, the method provided by the embodiment of the present invention can speed up the generation speed of the index file, shorten the time for generating the index file, and avoid the prior art ActiveMQ takes a long time to build index files, and the problem that affects the normal sending and receiving of messages ensures the real-time nature of ActiveMQ services.
[0102] In order to prevent the service device from repeatedly sending messages to be migrated (omitting to send messages to be migrated), in the embodiment of the present invention, after the message is sent to the master device, it further includes: receiving a migration message deletion instruction, and deleting the target according to the migration message deletion instruction Message, where the target message is a message successfully sent to the master device.
[0103] The foregoing migration message deletion instruction may be a deletion instruction for a certain message, or a deletion instruction for more than one message. For example, a certain service device in the index file generation system stores message a, message b, and message c of the main device. When the received message migration instruction is a message migration instruction for message a, the producer will produce message a to the main device Then, delete message a according to the delete instruction for message a; when the received message migration instruction is a message migration instruction for message b, after the producer produces the above message b to the master device, delete the message according to the delete instruction for message b b, this cycle. For another example, when the received message migration instruction is a message migration instruction for all messages (including message a, message b, and message c), after the producer produces all messages to the above-mentioned master device, according to the delete instruction instruction for all messages Delete all messages.
[0104] Optionally, the migration message deletion instruction may be sent by the consumer (application program used to delete the corresponding message on the service device). The number of consumers can correspond to the number of producers producing messages.
[0105] It should be noted that the above-mentioned consumer and the above-mentioned producer for producing messages may be an application on a service device that executes the production and deletion steps, or may be an application on another device other than the above-mentioned service device.
[0106] Furthermore, in order to reduce the time delay caused by the establishment of a connection between each producer and the service device, and to speed up the speed of the service device sending messages to the main device, the number of producers, the number of consumers and the messages to be migrated in the embodiment of the present invention The number of destination objects can be the same, and consumers and producers corresponding to each destination object can send messages back to the main device in parallel, that is, a server device can send multiple messages to the main device at the same time, further shortening the index The time occupied by the file further ensures the real-time nature of the ActiveMQ service. Wherein, the destination object is the object targeted by the message to be migrated, and multiple messages to be migrated are targeted at one destination object. In other words, messages with the same destination object share a producer.
[0107] Corresponding to the method embodiment of the first aspect, the embodiment of the present invention also provides an index file generating device, such as Figure 4 As shown, the device is applied to a main device in an index file generation system, the index file generation system further includes at least two service devices, and the device includes:
[0108] The obtaining module 410 is used to obtain all local log files for storing messages;
[0109] The sending module 420 is configured to group the log files according to preset rules and send them to each service device, so that each service device synchronously sends the messages in the received log file back to the main device;
[0110] The receiving module 430 is used to receive and store messages sent by each service device, and at the same time generate an index file of the messages sent by each service device.
[0111] An index file generating device provided by an embodiment of the present invention distributes log files used to store messages on a main device to at least two service devices in an index file generating system, so that each service device synchronizes the received information , The messages stored in the log file grouped by the master device are re-sent back to the master device. Since each service device sends messages to the master device synchronously, the master device receives the same number as the service device (or more than the service device) at the same time. ), and the master device can establish an index of the same number of messages at the same time after receiving and storing the above-mentioned number of messages. Compared with the prior art scanning the messages in the daily files one by one and building indexes one by one, the device provided by the embodiment of the present invention can speed up the generation of index files, shorten the time for generating index files, and avoid the long cost of ActiveMQ in the prior art Time to build index files, and affect the normal sending and receiving of messages, to ensure the real-time performance of ActiveMQ service.
[0112] Optionally, the sending module 420 includes:
[0113] The first division sub-module is configured to divide the log files into corresponding number of log groups evenly according to the number of service devices;
[0114] The first sending sub-module is used to send each group of log files to each service device.
[0115] Optionally, the sending module 420 includes:
[0116] The first determining submodule is used to obtain the current load of each service device, and determine the number of log files to be sent to each service device according to the obtained current load of each service device and the total number of log files;
[0117] The second division sub-module is configured to group the log files according to the determined number of log files to be sent to each service device;
[0118] The second sending sub-module is used to send log packets containing a corresponding number of log files to the corresponding service device.
[0119] Optionally, the receiving module 430 includes:
[0120] The receiving sub-module is used to receive and store the messages sent by each service device;
[0121] Establish a sub-module to obtain the offset address of each message in parallel, and establish a B-tree according to the obtained offset address;
[0122] The second determining sub-module is used to store the B-tree in a new file and determine the new file as an index file.
[0123] Corresponding to the method embodiment of the second aspect, the embodiment of the present invention also provides an index file generating device, such as Figure 5 As shown, the device is applied to any service device in an index file generation system, the index file generation system further includes a main device, and the device includes:
[0124] The receiving module 510 is configured to receive a log file sent by the main device, where the log file is the main device grouping all log files locally used for storing messages according to a preset rule and then sending it to the service equipment;
[0125] The generating module 520 receives the message migration instruction, and sends the message to be migrated targeted by the message migration instruction in the received log file to the master device, so that the master device stores the message to be migrated.
[0126] According to an index file generating device provided by an embodiment of the present invention, after receiving a message migration instruction, the service device in the index file generating system sends back the message to be migrated in the received log file targeted by the message migration instruction. Device, because the index file generation system includes at least two service devices and, therefore, at least two service devices send messages to the main device at the same time, and the main device can receive and store at least two messages at the same time. Create an index of at least two messages. Compared with the prior art scanning the messages in the log file one by one and establishing the index of the messages one by one, the device provided by the embodiment of the present invention can speed up the generation of index files, shorten the time for generating the index files, and avoid ActiveMQ in the prior art. It takes a long time to build the index file, and the problem that affects the normal sending and receiving of messages ensures the real-time performance of ActiveMQ service.
[0127] Optionally, the generating module 520 includes:
[0128] The obtaining sub-module receives a message migration instruction, and obtains the producers and consumers of the messages to be migrated, where the messages to be migrated are messages in the received log file, and the consumers are to obtain messages and Sending the obtained message to the application program of the producer, the producer receiving the message sent by the consumer and producing the message, and sending the produced message to the application program of the main device;
[0129] The production module is used to run each producer and consumer, so that each consumer sends the message to the producer of the message, so that the producer produces the message and sends the produced message to the master device .
[0130] Optionally, the number of producers is the same as the number of destination objects of the message to be migrated, wherein the destination object is an object targeted by the message to be migrated, and multiple messages to be migrated are targeted at one destination object.
[0131] The embodiment of the present invention also provides an index file generation system, such as Image 6 As shown, the system includes a main device 610 and at least two service devices 620;
[0132] The main device 610 is configured to execute the index file generation method corresponding to each invention embodiment of the first aspect;
[0133] Each of the service devices 620 is configured to execute the index file generation method corresponding to each invention embodiment of the second aspect.
[0134] An index file generation system provided by an embodiment of the present invention allocates a log file used to store messages on a main device to at least two service devices in the index file generation system, so that each service device assigns the log file to The message in is re-sent back to the main device. Since each service device sends messages to the main device at the same time, the main device receives and stores at least two messages at the same time. Therefore, the index of at least two messages can be established in parallel, compared with the prior art scanning one by one. The messages in the log file are indexed one by one. The system provided by the embodiment of the present invention can speed up the generation of the index file, shorten the time for generating the index file, and avoid the ActiveMQ in the prior art from taking a long time to create the index file , And the problem that affects the normal sending and receiving of messages ensures the real-time nature of ActiveMQ services.
[0135] As for the device/system embodiment, since it is basically similar to the method embodiment, the description is relatively simple, and for related parts, please refer to the part of the description of the method embodiment.
[0136] It should be noted that in this article, relational terms such as first and second are only used to distinguish one entity or operation from another entity or operation, and do not necessarily require or imply one of these entities or operations. There is any such actual relationship or order between. Moreover, the terms "include", "include" or any other variants thereof are intended to cover non-exclusive inclusion, so that a process, method, article or device including a series of elements not only includes those elements, but also includes those that are not explicitly listed Other elements of, or also include elements inherent to this process, method, article or equipment. If there are no more restrictions, the element defined by the sentence "including a..." does not exclude the existence of other same elements in the process, method, article, or equipment including the element.
[0137] Each embodiment in this specification is described in a related manner, and the same or similar parts between the various embodiments can be referred to each other, and each embodiment focuses on the difference from other embodiments.
[0138] The foregoing descriptions are only preferred embodiments of the present invention, and are not used to limit the protection scope of the present invention. Any modification, equivalent replacement, improvement, etc. made within the spirit and principle of the present invention are all included in the protection scope of the present invention.
