File processing method and device applied to flow file and electronic equipment

Abstract

The embodiment of the invention discloses a file processing method and device applied to a pipeline file and electronic equipment. A specific embodiment of the method comprises the following steps: generating a file segmentation message; performing master-slave synchronization confirmation according to the file segmentation message; responding to successful master-slave synchronization confirmation, and generating a main stream water file; taking the file segmentation message as a service message, and writing the service message into a mainstream water file; in response to the received snapshot message flow or the service request message flow, performing master-slave synchronization confirmation on the snapshot message flow and the service request message flow; and in response to successful master-slave synchronization confirmation, writing a snapshot message included in the snapshot message flow or a service request message included in the service request message flow into the updated main stream water file. According to the implementation mode, the fault risk that the storage space of a disk is possibly exhausted due to the fact that the file size of a continuously-increased single pipeline file is too large under the condition that the system continuously operates for 7 * 24 hours is reduced, and meanwhile subsequent pipeline file backup, migration and replay are facilitated.

Description

Technical Field

[0001] Embodiments of the present disclosure relate to the field of computer technology, and in particular to a file processing method, device, and electronic device applied to a pipeline file. Background Art

[0002] In a low-latency, high-concurrency information processing system (for example, a financial transaction record processing system), all important data of the system is stored in the memory of the core information processing engine. To ensure data integrity and accurately restore the data status, the system will record the original business messages that cause all data status changes in the engine in order into the disk journal file. Restoring the information processing process (for example, the transaction record process) of a period of time in the journal file is one of the effective means of data correctness verification and problem troubleshooting. At present, the common information processing systems mentioned above often use a single journal file to continuously record the original business messages.

[0003] However, the inventors found that when the above method is adopted, there are the following technical problems:

[0004] To ensure effective recording of information, information processing systems often operate in an uninterrupted manner (for example, 24 / 7). This may result in a single stream file being too large and may lead to the risk of disk storage space exhaustion.

[0005] The above information disclosed in this Background section is only for enhancement of understanding of the background of the inventive concept and therefore it may contain information that does not form the prior art that is already known to a person of ordinary skill in the art. Summary of the invention

[0006] The content of this disclosure is used to introduce concepts in a brief form, which will be described in detail in the detailed implementation section below. The content of this disclosure is not intended to identify the key features or essential features of the technical solution claimed for protection, nor is it intended to limit the scope of the technical solution claimed for protection.

[0007] Some embodiments of the present disclosure propose a file processing method, device, and electronic device applied to a pipeline file to solve one or more of the technical problems mentioned in the above background technology section.

[0008] In a first aspect, some embodiments of the present disclosure provide a file processing method for a stream file, the method comprising: generating a file segmentation message, wherein the file segmentation message is a fence message for a stream file generated by a main queue machine; performing master-slave synchronization confirmation according to the file segmentation message; in response to a successful master-slave synchronization confirmation, generating a mainstream stream file, wherein the mainstream stream file is a newly generated stream file corresponding to the main queue machine; writing the file segmentation message as a service message into the mainstream stream file to obtain an updated mainstream stream file; in response to receiving a snapshot message stream or a service request message stream, performing master-slave synchronization confirmation on the snapshot message stream and the service request message stream, wherein the snapshot message stream is a message stream including snapshot messages copied by a memory database mirror, and the service request message is a message stream including service request messages sent by a service gateway; in response to a successful master-slave synchronization confirmation, writing the snapshot message included in the snapshot message stream or the service request message included in the service request message stream into the updated mainstream stream file.

[0009] In a second aspect, some embodiments of the present disclosure provide a file processing device for pipeline files, the device comprising: a first generating unit, configured to generate a file segmentation message, wherein the file segmentation message is a fence message for pipeline files generated by a master queue machine; a first master-slave synchronization confirmation unit, configured to perform master-slave synchronization confirmation according to the file segmentation message; a second generating unit, configured to generate a mainstream pipeline file in response to a successful master-slave synchronization confirmation, wherein the mainstream pipeline file is a newly generated pipeline file corresponding to the master queue machine; a first writing unit, configured to use the file segmentation message as a service message , write the above-mentioned mainstream water file to obtain the updated mainstream water file; the second master-slave synchronization confirmation unit is configured to perform master-slave synchronization confirmation on the above-mentioned snapshot message flow and the business request message flow in response to receiving the snapshot message flow or the business request message flow, wherein the above-mentioned snapshot message flow is a message flow including snapshot messages copied by the memory database mirror, and the above-mentioned business request message is a message flow including business request messages sent by the business gateway; the second writing unit is configured to write the snapshot message included in the above-mentioned snapshot message flow or the business request message included in the above-mentioned business request message flow into the above-mentioned updated mainstream water file in response to the successful master-slave synchronization confirmation.

[0010] In a third aspect, some embodiments of the present disclosure provide an electronic device comprising: one or more processors; a storage device on which one or more programs are stored, and when the one or more programs are executed by the one or more processors, the one or more processors implement the method described in any implementation manner of the above-mentioned first aspect.

[0011] In a fourth aspect, some embodiments of the present disclosure provide a computer-readable medium having a computer program stored thereon, wherein when the program is executed by a processor, the method described in any implementation manner of the above-mentioned first aspect is implemented.

[0012] The above-mentioned various embodiments of the present disclosure have the following beneficial effects: through the file processing method applied to the running water file of some embodiments of the present disclosure, the flexible segmentation of the running water file is realized, and the failure risk of the disk storage space being exhausted due to the excessive file size of a single running water file is avoided. Specifically, the reason for the failure risk is that in order to ensure the business continuity of the system (such as maintenance and update without stopping the service), the information processing system often runs in an uninterrupted manner and continuously saves the original business message to the running water file effectively, which will cause the file size of a single running water file to be too large, and there may be a failure risk of the disk storage space being exhausted. Based on this, the file processing method applied to the running water file of some embodiments of the present disclosure, first, generates a file segmentation message, wherein the above-mentioned file segmentation message is a fence message for the running water file generated by the main queue machine. By generating the file segmentation message, the segmentation point for the running water file is determined. Secondly, the master-slave synchronization confirmation is performed according to the above-mentioned file segmentation message. Through the master-slave synchronization confirmation, the consistency of the data is guaranteed. Then, in response to the successful confirmation of master-slave synchronization, a mainstream water file is generated, wherein the mainstream water file is a newly generated water file corresponding to the above-mentioned main queue machine. In this way, the purpose of truncating and segmenting the water file is achieved. Further, the above-mentioned file segmentation message is written into the above-mentioned mainstream water file as a business message to obtain the updated mainstream water file. By writing the file segmentation message into the mainstream water file as a file header, subsequent file replay is facilitated. In addition, in response to receiving a snapshot message stream or a business request message stream, the above-mentioned snapshot message stream and the business request message stream are subjected to master-slave synchronization confirmation, wherein the above-mentioned snapshot message stream is a message stream including snapshot messages copied by the memory database mirror, and the above-mentioned business request message is a message stream including business request messages sent by the business gateway. In this way, data consistency is ensured. Finally, in response to the successful confirmation of master-slave synchronization, the snapshot message included in the above-mentioned snapshot message stream or the business request message included in the above-mentioned business request message stream is written into the above-mentioned updated mainstream water file. In this way, the segmentation of the stream file is achieved, that is, a single large stream file is divided into multiple small stream files, which can reduce the risk of failure due to the excessive file size of a single stream file and the exhaustion of disk storage space, and facilitate the subsequent replay of the stream file. BRIEF DESCRIPTION OF THE DRAWINGS

[0013] The above and other features, advantages and aspects of the embodiments of the present disclosure will become more apparent with reference to the following detailed description in conjunction with the accompanying drawings. Throughout the accompanying drawings, the same or similar reference numerals represent the same or similar elements. It should be understood that the drawings are schematic and that components and elements are not necessarily drawn to scale.

[0014] Figure 1 is a flowchart of some embodiments of a file processing method applied to a pipeline file according to the present disclosure;

[0015] Figure 2 It is a relationship diagram of historical slave water flow files, historical mainstream water flow files, slave water flow files and mainstream water flow files;

[0016] Figure 3 It is a schematic diagram of the generation process of the updated mainstream water file;

[0017] Figure 4 It is a schematic diagram of the interaction process between the service gateway, the master queuing machine, the slave queuing machine and the information processing engine;

[0018] Figure 5 It is a schematic diagram of the message flow structure of the original business message flow of the system;

[0019] Figure 6 It is a schematic diagram of the process of an information processing engine replaying a pipeline file;

[0020] Figure 7 It is a structural schematic diagram of some embodiments of the file processing device of the pipeline file according to the present disclosure;

[0021] Figure 8 It is a schematic diagram of the structure of an electronic device suitable for implementing some embodiments of the present disclosure. DETAILED DESCRIPTION

[0022] Embodiments of the present disclosure will be described in more detail below with reference to the accompanying drawings. Although certain embodiments of the present disclosure are shown in the accompanying drawings, it should be understood that the present disclosure can be implemented in various forms and should not be construed as being limited to the embodiments set forth herein. On the contrary, these embodiments are provided to provide a more thorough and complete understanding of the present disclosure. It should be understood that the drawings and embodiments of the present disclosure are only for exemplary purposes and are not intended to limit the scope of protection of the present disclosure.

[0023] It should also be noted that, for ease of description, only the parts related to the invention are shown in the drawings. In the absence of conflict, the embodiments and features in the embodiments of the present disclosure can be combined with each other.

[0024] It should be noted that the concepts such as "first" and "second" mentioned in the present disclosure are only used to distinguish different devices, modules or units, and are not used to limit the order or interdependence of the functions performed by these devices, modules or units.

[0025] It should be noted that the modifications of "one" and "plurality" mentioned in the present disclosure are illustrative rather than restrictive, and those skilled in the art should understand that unless otherwise clearly indicated in the context, it should be understood as "one or more".

[0026] The names of the messages or information exchanged between multiple devices in the embodiments of the present disclosure are only used for illustrative purposes and are not used to limit the scope of these messages or information.

[0027] The present disclosure will be described in detail below with reference to the accompanying drawings and in conjunction with embodiments.

[0028] refer to Figure 1 , shows a process 100 of some embodiments of the file processing method applied to a running file according to the present disclosure. The file processing method applied to a running file comprises the following steps:

[0029] Step 101, generate a file segmentation message.

[0030] In some embodiments, the execution subject (e.g., a computing device) of the file processing method applied to a running file may generate a file segmentation message. The file segmentation message is a fence message generated by the main queue machine for the running file. The running file may be a file for storing business messages generated in a time sequence. In practice, business messages may include, but are not limited to: transaction records and business operations. For example, transaction records may record transaction data such as changes in funds, changes in orders, changes in positions, etc. that occur during a transaction. Business operations may represent corresponding operations for virtual items, such as business operations may include: liquidation operations, position addition operations, closing operations, etc. For example, virtual items may be stocks, funds, bonds, etc. The fence message represents the truncation position or restart position for the running file.

[0031] It should be noted that the above-mentioned computing device can be hardware or software. When the computing device is hardware, it can be implemented as a distributed cluster composed of multiple servers or terminal devices, or it can be implemented as a single server or a single terminal device. When the computing device is embodied as software, it can be installed in the hardware devices listed above. It can be implemented as multiple software or software modules for providing distributed services, for example, or it can be implemented as a single software or software module. No specific limitation is made here. It should be understood that the number of computing devices can have any number according to the implementation needs. Specifically, the above-mentioned computing device can be a file processing system composed of a master queue machine, a slave queue machine, a service gateway and an information processing engine, that is, the above-mentioned file processing method applied to the running water file is applied to the above-mentioned file processing system. Among them, the master queue machine and the slave queue machine are serialized sequencers of the master-slave relationship. For example, the master queue machine and the slave queue machine can be implemented using Kafka queues. The service gateway can be a gateway set between the client and the above-mentioned computing device. The information processing engine can be a component for processing business messages.

[0032] In some optional implementations of some embodiments, the execution subject generating the file segmentation message may include the following steps:

[0033] In the first step, the operating states of the master queuing machine and the slave queuing machine are determined respectively to generate a first operating state and a second operating state.

[0034] Among them, the first operating state represents the working state of the above-mentioned main queue machine. The second operating state represents the working state of the above-mentioned slave queue machine. In practice, the operating states corresponding to the main queue machine and the slave queue machine include: abnormal operating state, working state. Among them, the abnormal operating state represents that the main queue machine or the slave queue machine is in an abnormal state, that is, it cannot provide queue service normally. The working state represents that the main queue machine or the slave queue machine is in a normal operating state. Specifically, since the main queue machine and the slave queue machine adopt a dual-machine hot standby operation mechanism, that is, when the main queue machine is abnormal, the slave queue machine will quickly switch to the singleton mode to continue to provide queue service.

[0035] In the second step, in response to determining that the first operating state indicates that the master queuing machine is operating normally and the second operating state indicates that the slave queuing machine is operating normally, it is determined whether there is a historical file segmentation message.

[0036] The above historical file segmentation message is a fence message corresponding to the historical stream file generated before the above file segmentation message. In practice, in order to control the file length of the stream file, it is necessary to generate file segmentation messages irregularly according to the time sequence for segmenting the stream file. Specifically, since the file segmentation message is only generated by the main queue machine, the main queue machine can search whether there is a generated historical file segmentation message.

[0037] The third step is to determine whether the historical snapshot message flow and the historical service request message flow have been written into the above historical flow file in response to the existence of the historical file segmentation message.

[0038] In practice, the execution subject can determine whether there is a writing process and process status corresponding to the historical snapshot message flow and the historical business request message flow, so as to determine whether the historical snapshot message flow and the historical business request message flow have been written into the historical flow file.

[0039] The fourth step is to generate the above-mentioned file segmentation message through the above-mentioned main queue machine in response to the completion of writing.

[0040] In practice, the execution entity may generate the file segmentation message through the main queue machine.

[0041] In some optional implementations of some embodiments, the main queuing machine generates a file segmentation message, which may include the following steps:

[0042] The first step is to obtain the queue machine operating parameters corresponding to the main queue machine and the slave queue machine respectively, and obtain the first queue machine operating parameters and the second queue machine operating parameters.

[0043] The queue machine operation parameters are used to describe the working status of the queue machine. In practice, the queue machine operation parameters may include but are not limited to: the running status of the queue machine, the number of service messages in the main queue memory that have not been confirmed by the backup queue machine, the size of the main stream file and the latest split time, the queue machine throughput, and the queue machine load pressure.

[0044] The second step is to obtain the target history main flow file and the target history sub flow file.

[0045] The target historical main flow file is the historical main flow file closest to the current time. The target historical secondary flow file is the historical secondary flow file closest to the current time.

[0046] The third step is to generate the above-mentioned file segmentation message through the operating parameters of the first queue machine, the operating parameters of the second queue machine, the target historical mainstream flow file, the target historical secondary flow file and the pre-trained file segmentation message generation model.

[0047] Among them, the file segmentation message generation model is a binary classification model. Among them, the file segmentation message generation model includes: a first decision tree, a second decision tree, a business message feature extraction model and a classifier. Among them, the first decision tree corresponds to the first queue machine operation parameter. The second decision tree corresponds to the second queue machine operation parameter. In practice, since the queue machine operation parameters are pre-set, the decision tree can be used to map the complex "queue machine operation parameters" to the "queue machine operation state vector". The business message feature extraction model includes: 2 business message feature extraction sub-models, business message feature extraction sub-model A and business message feature extraction sub-model B. Among them, the business message feature extraction sub-model A and the business message feature extraction sub-model B both adopt a recurrent neural network model. The business message feature extraction sub-model A and the business message feature extraction sub-model B are set in parallel. The parameters between the business message feature extraction sub-model A and the business message feature extraction sub-model B are shared. The input of the business message feature extraction sub-model A is the target historical mainstream flow file. The input of the business message feature extraction sub-model B is the target historical secondary flow file. Since the main queue machine and the slave queue machine should ensure data consistency between each other in the running state, the business message feature extraction can be performed in parallel through two business message feature extraction sub-models that are set in parallel and share parameters. The classifier includes: a feature splicing layer and at least one fully connected layer. Among them, the feature splicing layer is used to splice the queue machine operation state vector corresponding to the main queue machine, the queue machine operation state vector corresponding to the slave queue machine, the business message feature corresponding to the main queue machine, and the business message feature corresponding to the slave queue machine to obtain the spliced ​​feature. Then the size of the last fully connected layer in at least one fully connected layer is 1×2. At least one fully connected layer is used to map high-dimensional features to low dimensions, and finally obtain a binary classification mark of whether to generate a file segmentation message. In this way, compared with the method of manually specifying the file size of the pipeline file, the main queue machine can automatically combine the working status and message processing pressure of the main queue machine and the slave queue machine to generate a file segmentation message. In this way, more effective file segmentation of pipeline files is achieved.

[0048] Step 102: confirm the master-slave synchronization according to the file segmentation message.

[0049] In some embodiments, the execution subject can perform master-slave synchronization confirmation according to the file segmentation message. In practice, the master queue machine can send the file segmentation message to the slave queue machine to confirm the consistency between the master queue machine and the slave queue machine to achieve master-slave synchronization confirmation.

[0050] In some optional implementations of some embodiments, the execution subject performs master-slave synchronization confirmation according to the file segmentation message, which may include the following steps:

[0051] The first step is to verify the message type by segmenting the above file from the queue machine.

[0052] Among them, the slave queue machine and the master queue machine adopt a dual-machine hot standby operation mechanism. In practice, the master queue machine and the slave queue machine are hot-backed up to each other, that is, the data consistency between the master queue machine and the slave queue machine is maintained, and when either the master or the slave queue machine fails to work normally due to an abnormality, the other queue machine can immediately switch to singleton mode to continue to provide queuing services. In addition, the master queue machine will send each received business message to the standby queue machine in sequence for confirmation, and the standby queue machine will send a response to the confirmation request back to the master queue machine. Therefore, the slave queue machine can determine whether the message type of the currently processed message corresponds to the file segmentation message when receiving the message confirmation request from the master queue, thereby achieving the purpose of message type verification.

[0053] The second step is to generate a pipeline file in response to the verification being successful.

[0054] The slave water flow file is a newly generated water flow file corresponding to the slave queue machine, and the slave water flow file is asynchronously consistent with the main water flow file. In practice, when the verification is passed, the slave queue machine can generate a new water flow file corresponding to the slave queue machine.

[0055] The third step is to write the above file segmentation messages into the secondary stream file.

[0056] In practice, the file segmentation message can be written into the stream file as a file header from the queue machine.

[0057] The fourth step is to write the above business message into the historical flow file in response to the verification failing.

[0058] Among them, the above-mentioned historical flow files are flow files that have not been segmented.

[0059] As an example, see Figure 2The relationship diagram of the historical slave water flow files, the historical mainstream water flow files, the slave water flow files and the mainstream water flow files shown in the figure, wherein the slave queue machine can generate the historical slave water flow files A, the historical slave water flow files B, the historical slave water flow files C and the slave water flow files in sequence according to the message confirmation request sent in order by the master queue machine. Among them, the files between the historical slave water flow files A, the historical slave water flow files B, the historical slave water flow files C and the slave water flow files are continuous. The master queue machine can generate the historical mainstream water flow files A, the historical mainstream water flow files B, the historical mainstream water flow files C and the mainstream water flow files in sequence according to the order of the messages confirmed by the slave queue machine. Among them, the files between the historical mainstream water flow files A, the historical mainstream water flow files B, the historical mainstream water flow files C and the mainstream water flow files are continuous. Among them, in order to ensure the data consistency between the master queue machine and the slave queue machine, the files between the historical mainstream water flow files A and the historical slave water flow files A are consistent; the files between the historical mainstream water flow files B and the historical slave water flow files B are consistent; the files between the historical mainstream water flow files C and the historical slave water flow files C are consistent; the files between the mainstream water flow files and the slave water flow files are consistent.

[0060] Step 5: In response to the successful writing, a master-slave synchronization confirmation response is sent to the master queuing machine.

[0061] The master-slave synchronization confirmation response indicates that the master-slave synchronization confirmation between the master queuing machine and the slave queuing machine is successful. In practice, the synchronization confirmation response may be a response to a synchronization request sent by the master queuing machine.

[0062] As an example, see Figure 3 The schematic diagram of the updated mainstream water flow file generation process shown in the figure, wherein, first, the master queue machine generates a file segmentation message 301. Then, the master queue machine sends the file segmentation message 301 to the slave queue machine for master-slave synchronization confirmation. Next, when the slave queue machine processes the file segmentation message 301, the slave queue machine will perform message type verification on the above-mentioned file segmentation message 301. Further, when the message type verification passes, the slave queue machine will generate a slave water flow file 302, and write the file segmentation message 301 as a file header into the slave water flow file 302. Next, when the message type verification fails, that is, the business message is not a file segmentation message, so the business message is written into the historical slave water flow file 303. Then, when the writing is successful, a master-slave synchronization confirmation response 304 is sent to the master queue machine. Further, the master queue machine will generate a mainstream water flow file 305. Finally, the file segmentation message 301 is written into the mainstream water flow file 305 to obtain the updated mainstream water flow file 306. At the same time, when the message type verification fails, the business message is not a file segmentation message. Therefore, for the slave queuing machine, the business message will be written into the historical slave flow file 303. At the same time, the business message will also be written into the historical mainstream flow file corresponding to the master queuing machine.

[0063] Step 103: In response to the master-slave synchronization being confirmed successfully, a mainstream water file is generated.

[0064] In some embodiments, in response to the master-slave synchronization confirmation being successful, the master queue machine generates a main stream water file, wherein the main stream water file is a newly generated water file corresponding to the above-mentioned master queue machine.

[0065] Step 104, write the file segmentation message into the mainstream water file as a business message to obtain an updated mainstream water file.

[0066] In some embodiments, the main queuing machine may write the file segmentation message into the file header of the main stream water file as a service message to obtain an updated main stream water file.

[0067] Step 105: in response to receiving the snapshot message flow or the service request message flow, performing master-slave synchronization confirmation on the snapshot message flow and the service request message flow.

[0068] In some embodiments, in response to receiving a snapshot message stream or a service request message stream, the master queue machine and the slave queue machine will perform master-slave synchronization confirmation on the snapshot message stream and the service request message stream. The snapshot message stream is a message stream including snapshot messages copied by the memory database mirror. The service request message is a message stream including service request messages sent by the service gateway.

[0069] As an example, see Figure 4 The schematic diagram of the interaction process between the service gateway, the main queue machine, the slave queue machine and the information processing engine shown, wherein the service gateway can receive a service request message 403 initiated by a client 402. Then, the service gateway can publish the service request message 403 to the main queue machine to add the service request message 403 to the service request message flow 404.

[0070] As yet another example, see Figure 5 The message flow structure diagram of the original service message flow of the system shown in the figure, wherein each unit in the original service message flow of the system corresponds to a service message object and is numbered incrementally starting from 0. For example, Figure 5 It may include: business message object A0 corresponding to number 0, business message object A1 corresponding to number 1, business message object A2 corresponding to number 2, business message object A3 corresponding to number 3055 3055 , Business message object A corresponding to number 3056 3056 , Business message object A corresponding to number 3057 3057. Wherein, the original business message stream of the system corresponds to a stream file stored in at least one disk, and each stream file stored in the disk in the specific original business message stream of the system includes a fixed header and a variable part. Wherein, the fixed header includes: a communication code and a business message first object number. Wherein, the communication code is a fixed 2-byte code, representing the communication subject code of the corresponding stream. In practice, the communication code is stored in an unsigned integer format. The business message first object number represents the object number of the business object corresponding to the first business message in the stream file. Wherein, the business message first object number is a fixed 8-byte code. In practice, the business message first object number is stored in an unsigned integer format. The variable part includes: K business messages. For each business message in the K business messages, the business message includes in the variable part: the text length value of the business message body and the business message body content. Wherein, the text length value of the business message body is used to record the content length of the business message body content, and is stored in a fixed 4-byte format. In practice, the text length value of the business message body is stored in an unsigned integer format. The business message body is used to record the specific message content of the business message. Figure 5 In it, there are 4 business messages, namely business message A, business message B, business message C and business message D. Among them, the text length value of the business message body corresponding to business message A is 40, and the corresponding business message body content is: "User A's order request". The text length value of the business message body corresponding to business message B is 20, and the corresponding business message body content is: "User B's order cancellation request". The text length value of the business message body corresponding to business message C is 10, and the corresponding business message body content is: "File switch". The text length value of the business message body corresponding to business message D is 40, and the corresponding business message body content is: "User C's order modification request".

[0071] In some optional implementations of some embodiments, the above snapshot message flow is generated by the following steps:

[0072] The first step is to subscribe to the mainstream water message of the above main queuing machine.

[0073] In practice, the information processing engine may adopt a subscription mechanism to subscribe to the business messages (eg, business request messages, file segmentation messages) published by the main queue machine.

[0074] In the second step, in response to the subscription being successful and receiving the file segmentation message, a mirror copy is performed on the memory database to obtain a memory data snapshot.

[0075] In practice, a memory database can be a database that stores data in memory. Compared with a disk-based database, a memory database has the advantage of fast data reading speed, but at the same time, a memory database needs to ensure continuous power supply. When the power supply fails, data in the memory database may be lost. Specifically, when computing resources are sufficient, the data stored in the memory database can be directly copied in memory as a memory data snapshot. In addition, the data saved in the transaction log of the memory database can also be used as a memory data snapshot based on a log. In addition, an incremental snapshot method can also be used to mirror the changed data based on the last memory data snapshot, and combine it with the last memory data snapshot to obtain a memory data snapshot.

[0076] In the third step, the snapshot start message, the memory data snapshot and the snapshot end message are sequentially written into the initial message stream through an asynchronous thread to obtain the snapshot message stream.

[0077] As an example, see Figure 4 The schematic diagram of the interaction process between the service gateway, the main queue machine, the slave queue machine and the information processing engine is shown, wherein when the information processing engine receives the file segmentation message 301, the information processing engine will mirror the memory database 408 to obtain the memory data snapshot 410. Then, the information processing engine can write the snapshot start message 409, the memory data snapshot 410 and the snapshot end message 411 into the initial message stream 412 in sequence through an asynchronous thread to obtain the snapshot message stream 406. Then, the above-mentioned main queue machine can obtain the above-mentioned snapshot message stream 406 from the information processing engine in a subscription manner. Among them, the initial message stream 412 must be a linear data stream.

[0078] In some optional implementations of some embodiments, before the execution subject writes the snapshot message included in the snapshot message flow or the service request message included in the service request message flow into the updated mainstream water file in response to the master-slave synchronization confirmation success, the method further includes:

[0079] The first step is to subscribe to the mainstream water message of the above main queuing machine.

[0080] In practice, new business messages published by the main queuing machine are obtained through message subscription.

[0081] In the second step, in response to the subscription being successful and receiving the message to be processed, the message type of the message to be processed is determined.

[0082] The above-mentioned to-be-processed message is a service request message sent by the service gateway and published by the above-mentioned main queuing machine.

[0083] In the third step, in response to the message type of the message to be processed being a non-memory data snapshot type, message processing is performed according to the service request message to generate a request processing result.

[0084] In practice, the above-mentioned information processing engine can trigger the business logic matching the business request message to generate a request processing result for the message to be processed.

[0085] The fourth step is to send the request processing result to the business gateway, so that the request processing result is forwarded to the client corresponding to the message to be processed through the business gateway.

[0086] Step 106 : In response to the master-slave synchronization confirmation being successful, the snapshot message included in the snapshot message flow or the service request message included in the service request message flow is written into the updated main flow file.

[0087] In some embodiments, in response to the master-slave synchronization confirmation being successful, the master queuing machine will write the snapshot message included in the snapshot message flow or the service request message included in the service request message flow into the updated main stream file.

[0088] As an example, see Figure 4 The schematic diagram of the interaction process between the service gateway, the master queue machine, the slave queue machine and the information processing engine shown, wherein, for the master queue machine, when receiving the service request message stream 404 composed of the service request message sent by the service gateway or the snapshot message stream 406 sent by the information processing engine, the master queue machine and the slave queue machine will perform master-slave synchronization confirmation, that is, to ensure the consistency of the service messages between the master queue machine and the slave queue machine. When the master queue machine receives the synchronization confirmation response 405 sent by the slave queue machine, the master queue machine will write the service request message 404 or the snapshot message stream 405 into the updated mainstream file 307.

[0089] In some optional implementations of some embodiments, the above method further includes:

[0090] The first step is to, in response to receiving a replay request for the updated mainstream water file, sequentially read the updated mainstream water file to obtain a service message sequence.

[0091] For details, see Figure 6 The information processing engine shown in the figure performs a replay process on a stream file, wherein the information processing engine, upon receiving a replay request for the updated mainstream stream file, can first load the updated mainstream stream file, and then sequentially read the updated mainstream stream file to obtain a service message sequence.

[0092] In the second step, for each service message in the above service message sequence, the following processing steps are performed:

[0093] The first sub-step is, in response to determining that the service message is a snapshot start message, initializing the above-mentioned memory database.

[0094] In practice, you can perform a self-check on the operating environment of the in-memory database, and start the in-memory database after the self-check passes.

[0095] Optionally, in response to the service message being a file segmentation message, the system consisting of the main queuing machine, the slave queuing machine, the service gateway and the information processing engine may be initialized.

[0096] The second sub-step is, in response to determining that the service message is a memory data snapshot and the memory database is initialized, saving the data in the memory data snapshot into the memory database.

[0097] In practice, the above execution entity can use a sequential writing method to save the data in the memory data snapshot into the memory database.

[0098] The third sub-step is, in response to the service message being a service request message and the in-memory database of the database initialization has not been loaded completely, placing the service message into a first-in-first-out message queue.

[0099] Optionally, in response to the service message being a service request message and the in-memory database for database initialization has been loaded, the service message is directly processed.

[0100] The fourth sub-step is, in response to determining that the business message is a snapshot end message, marking the memory database state as completed data loading, and sequentially reading the business messages from the first-in-first-out message queue, and processing the business messages in the order in which the messages are read.

[0101] In addition, when the service message is not a service request message, the second step may be performed again.

[0102] The above-mentioned content of "some optional implementations in some embodiments" as one of the inventions of the present disclosure realizes the segmentation and replay of the stream file without stopping the service, improves the replay speed, and greatly improves the efficiency of fault location, especially when there is an abnormality. Specifically, the reason for the inefficiency of fault location is that for a single stream file, as the data flow increases, the file size of the stream file increases accordingly, which is not conducive to the migration, backup and other operations of the stream file. If the stream file is simply physically segmented, when the stream file is replayed, it is still necessary to splice the stream file after physical segmentation. Especially when the fault point is located at the end of the stream file, the stream file with a larger file size greatly increases the time consumption of the stream replay. Based on this, the present disclosure first determines whether the business message is a file segmentation message, and judges whether it is the starting position of the stream file. By segmenting a single large-size stream file, and importing the memory snapshot into the header of the segmented stream file in the form of the original business message, it is possible to start replaying at the header of each segmented stream file. In addition, the present disclosure relies on snapshot start messages, memory data snapshots, and snapshot end messages to achieve fast data initialization for the memory database. In this way, the consistency between the memory state of the information processing engine and the normal operating program is guaranteed. In addition, considering that there may be new business messages (such as business request messages) without stopping the service, the replayed business messages are not directly processed during the replay process, but the order of business messages is controlled through a first-in-first-out message queue. After the memory database is initialized, the business messages are processed in the order in which they are in the first-in-first-out message queue. In this way, the replay speed is greatly improved, and the efficiency of fault location when anomalies occur is improved.

[0103] The above-mentioned various embodiments of the present disclosure have the following beneficial effects: through the file processing method applied to the running water file of some embodiments of the present disclosure, the flexible segmentation of the running water file is realized, and the failure risk of the disk storage space being exhausted due to the excessive file size of a single running water file is avoided. Specifically, the reason for the failure risk is that in order to ensure the business continuity of the system (such as maintenance and update without stopping the service), the information processing system often runs in an uninterrupted manner and continuously saves the original business message to the running water file effectively, which will cause the file size of a single running water file to be too large, so there may be a failure risk of the disk storage space being exhausted. Based on this, the file processing method applied to the running water file of some embodiments of the present disclosure, first, generates a file segmentation message, wherein the above-mentioned file segmentation message is a fence message for the running water file generated by the main queue machine. By generating the file segmentation message, the segmentation point for the running water file is determined. Secondly, the master-slave synchronization confirmation is performed according to the above-mentioned file segmentation message. Through the master-slave synchronization confirmation, the consistency of the data is guaranteed. Then, in response to the successful confirmation of master-slave synchronization, a mainstream water file is generated, wherein the mainstream water file is a newly generated water file corresponding to the above-mentioned main queue machine. In this way, the purpose of truncating and segmenting the water file is achieved. Further, the above-mentioned file segmentation message is written into the above-mentioned mainstream water file as a business message to obtain the updated mainstream water file. By writing the file segmentation message into the mainstream water file as a file header, subsequent file replay is facilitated. In addition, in response to receiving a snapshot message stream or a business request message stream, the above-mentioned snapshot message stream and the business request message stream are subjected to master-slave synchronization confirmation, wherein the above-mentioned snapshot message stream is a message stream including snapshot messages copied by the memory database mirror, and the above-mentioned business request message is a message stream including business request messages sent by the business gateway. In this way, data consistency is ensured. Finally, in response to the successful confirmation of master-slave synchronization, the snapshot message included in the above-mentioned snapshot message stream or the business request message included in the above-mentioned business request message stream is written into the above-mentioned updated mainstream water file. In this way, the segmentation of the stream file is achieved, that is, a single large stream file is divided into multiple small stream files, which can reduce the risk of failure due to the excessive file size of a single stream file and the exhaustion of disk storage space, and facilitate the subsequent replay of the stream file.

[0104] Further references Figure 7 As an implementation of the methods shown in the above figures, the present disclosure provides some embodiments of a file processing device for a running file. These device embodiments are similar to Figure 1 Corresponding to the method embodiments shown, the file processing device applied to pipeline files can be specifically applied to various electronic devices.

[0105] like Figure 7As shown, some embodiments of the file processing device 700 applied to the stream file include: a first generation unit 701, a first master-slave synchronization confirmation unit 702, a second generation unit 703, a first writing unit 704, a second master-slave synchronization confirmation unit 705, and a second writing unit 706. The first generation unit 701 is configured to generate a file segmentation message, wherein the file segmentation message is a fence message generated by the master queue machine for the stream file; the first master-slave synchronization confirmation unit 702 is configured to perform master-slave synchronization confirmation according to the file segmentation message; the second generation unit 703 is configured to generate a mainstream stream file in response to the success of the master-slave synchronization confirmation, wherein the mainstream stream file is a newly generated stream file corresponding to the master queue machine; the first writing unit 704 is configured to write the file segmentation message as a service message into the mainstream stream file to obtain an updated The main flow file after update; the second master-slave synchronization confirmation unit 705 is configured to perform master-slave synchronization confirmation on the snapshot message flow and the business request message flow in response to receiving the snapshot message flow or the business request message flow, wherein the snapshot message flow is a message flow including snapshot messages copied by the memory database mirror, and the business request message is a message flow including business request messages sent by the business gateway; the second writing unit 706 is configured to write the snapshot message included in the snapshot message flow or the business request message included in the business request message flow into the main flow file after update in response to the successful master-slave synchronization confirmation.

[0106] It can be understood that the units recorded in the file processing device 700 for the stream file are similar to the reference Figure 1 Therefore, the operations, features and beneficial effects described above for the method are also applicable to the file processing device 700 applied to the journal file and the units included therein, and will not be described in detail here.

[0107] Reference below Figure 8 , which shows a structural schematic diagram of an electronic device (eg, a computing device) 800 suitable for implementing some embodiments of the present disclosure. Figure 8 The electronic device shown is only an example and should not bring any limitation to the functions and scope of use of the embodiments of the present disclosure.

[0108] like Figure 8As shown, the electronic device 800 may include a processing device (e.g., a central processing unit, a graphics processing unit, etc.) 801, which can perform various appropriate actions and processes according to a program stored in a read-only memory 802 or a program loaded from a storage device 808 into a random access memory 803. Various programs and data required for the operation of the electronic device 800 are also stored in the random access memory 803. The processing device 801, the read-only memory 802, and the random access memory 803 are connected to each other via a bus 804. An input / output interface 805 is also connected to the bus 804.

[0109] Typically, the following devices may be connected to the input / output interface 805: an input device 806 including, for example, a touch screen, a touch pad, a keyboard, a mouse, a camera, a microphone, an accelerometer, a gyroscope, etc.; an output device 807 including, for example, a liquid crystal display (LCD), a speaker, a vibrator, etc.; a storage device 808 including, for example, a magnetic tape, a hard disk, etc.; and a communication device 809. The communication device 809 may allow the electronic device 800 to communicate with other devices wirelessly or by wire to exchange data. Although Figure 8 The electronic device 800 is shown with various devices, but it should be understood that it is not required to implement or possess all the devices shown. More or fewer devices may be implemented or possessed instead. Figure 8 Each block shown in the figure may represent one device, or may represent multiple devices as required.

[0110] In particular, according to some embodiments of the present disclosure, the process described above with reference to the flowchart can be implemented as a computer software program. For example, some embodiments of the present disclosure include a computer program product, which includes a computer program carried on a computer-readable medium, and the computer program includes a program code for executing the method shown in the flowchart. In some such embodiments, the computer program can be downloaded and installed from a network through a communication device 809, or installed from a storage device 808, or installed from a read-only memory 802. When the computer program is executed by the processing device 801, the above-mentioned functions defined in the method of some embodiments of the present disclosure are executed.

[0111] It should be noted that the computer-readable medium recorded in some embodiments of the present disclosure may be a computer-readable signal medium or a computer-readable storage medium or any combination of the above two. The computer-readable storage medium may be, for example, but not limited to, an electrical, magnetic, optical, electromagnetic, infrared, or semiconductor system, device or device, or any combination of the above. More specific examples of computer-readable storage media may include, but are not limited to: an electrical connection with one or more wires, a portable computer disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the above. In some embodiments of the present disclosure, a computer-readable storage medium may be any tangible medium containing or storing a program that can be used by or in combination with an instruction execution system, device or device. In some embodiments of the present disclosure, a computer-readable signal medium may include a data signal propagated in a baseband or as part of a carrier wave, which carries a computer-readable program code. This propagated data signal may take a variety of forms, including but not limited to electromagnetic signals, optical signals, or any suitable combination of the above. The computer readable signal medium may also be any computer readable medium other than a computer readable storage medium, which may send, propagate or transmit a program for use by or in conjunction with an instruction execution system, apparatus or device. The program code contained on the computer readable medium may be transmitted using any suitable medium, including but not limited to: wires, optical cables, RF (radio frequency), etc., or any suitable combination of the above.

[0112] In some embodiments, the client and the server may communicate using any currently known or future developed network protocol such as HTTP (Hyper Text Transfer Protocol), and may be interconnected with any form or medium of digital data communication (e.g., a communication network). Examples of communication networks include a local area network ("LAN"), a wide area network ("WAN"), an internet (e.g., the Internet), and a peer-to-peer network (e.g., an ad hoc peer-to-peer network), as well as any currently known or future developed network.

[0113] The computer-readable medium may be included in the electronic device; or it may exist independently without being assembled into the electronic device. The computer-readable medium carries one or more programs. When the one or more programs are executed by the electronic device, the electronic device: generates a file segmentation message, wherein the file segmentation message is a fence message for a running water file generated by the master queue machine; performs master-slave synchronization confirmation according to the file segmentation message; generates a main running water file in response to the master-slave synchronization confirmation success, wherein the main running water file is a newly generated running water file corresponding to the master queue machine; writes the file segmentation message as a service message into the main running water file to obtain an updated main running water file; performs master-slave synchronization confirmation on the snapshot message flow and the service request message flow in response to receiving a snapshot message flow or a service request message flow, wherein the snapshot message flow is a message flow including snapshot messages copied by a memory database mirror, and the service request message is a message flow including service request messages sent by a service gateway; writes the snapshot message included in the snapshot message flow or the service request message included in the service request message flow into the updated main running water file in response to the master-slave synchronization confirmation success.

[0114] Computer program code for performing the operations of some embodiments of the present disclosure may be written in one or more programming languages ​​or a combination thereof, including object-oriented programming languages ​​such as Java, Smalltalk, C++, and conventional procedural programming languages ​​such as "C" or similar programming languages. The program code may be executed entirely on the user's computer, partially on the user's computer, as a separate software package, partially on the user's computer and partially on a remote computer, or entirely on a remote computer or server. In cases involving a remote computer, the remote computer may be connected to the user's computer via any type of network, including a local area network (LAN) or a wide area network (WAN), or may be connected to an external computer (e.g., via the Internet using an Internet service provider).

[0115] The flow chart and block diagram in the accompanying drawings illustrate the possible architecture, function and operation of the system, method and computer program product according to various embodiments of the present disclosure. In this regard, each square box in the flow chart or block diagram can represent a module, a program segment or a part of a code, and the module, the program segment or a part of the code contains one or more executable instructions for realizing the specified logical function. It should also be noted that in some implementations as replacements, the functions marked in the square box can also occur in a sequence different from that marked in the accompanying drawings. For example, two square boxes represented in succession can actually be executed substantially in parallel, and they can sometimes be executed in the opposite order, depending on the functions involved. It should also be noted that each square box in the block diagram and / or flow chart, and the combination of the square boxes in the block diagram and / or flow chart can be implemented with a dedicated hardware-based system that performs a specified function or operation, or can be implemented with a combination of dedicated hardware and computer instructions.

[0116] The units described in some embodiments of the present disclosure may be implemented by software or by hardware. The described units may also be provided in a processor, for example, may be described as: a processor comprising a first generation unit, a first master-slave synchronization confirmation unit, a second generation unit, a first writing unit, a second master-slave synchronization confirmation unit, and a second writing unit. Among them, the names of these units do not constitute a limitation on the unit itself under certain circumstances, for example, the second writing unit may also be described as "a unit that writes the snapshot message included in the above-mentioned snapshot message flow or the service request message included in the above-mentioned service request message flow into the above-mentioned updated mainstream water file in response to successful master-slave synchronization confirmation."

[0117] The functions described above herein may be performed at least in part by one or more hardware logic components. For example, without limitation, exemplary types of hardware logic components that may be used include: field programmable gate arrays (FPGAs), application specific integrated circuits (ASICs), application specific standard products (ASSPs), systems on chips (SOCs), complex programmable logic devices (CPLDs), and the like.

[0118] The above descriptions are only some preferred embodiments of the present disclosure and an explanation of the technical principles used. Those skilled in the art should understand that the scope of the invention involved in the embodiments of the present disclosure is not limited to the technical solutions formed by a specific combination of the above-mentioned technical features, but should also cover other technical solutions formed by any combination of the above-mentioned technical features or their equivalent features without departing from the above-mentioned inventive concept. For example, the above-mentioned features are replaced with the technical features with similar functions disclosed in the embodiments of the present disclosure (but not limited to) and the technical solutions formed.

Claims

1. A file processing method applied to a journal file, comprising: Generate a file segmentation message, wherein the file segmentation message is a fence message for the pipeline file generated by the main queuing machine; Perform master-slave synchronization confirmation according to the file segmentation message; In response to the master-slave synchronization confirmation being successful, generating a main stream water file, wherein the main stream water file is a newly generated water file corresponding to the main queuing machine; Writing the file segmentation message into the main stream file as a service message to obtain an updated main stream file; In response to receiving a snapshot message flow or a service request message flow, performing master-slave synchronization confirmation on the snapshot message flow and the service request message flow, wherein the snapshot message flow is a message flow including snapshot messages copied by the memory database mirror, and the service request message is a message flow including service request messages sent by the service gateway; In response to the master-slave synchronization being confirmed successfully, the snapshot message included in the snapshot message flow or the service request message included in the service request message flow is written into the updated main flow file.

2. The method according to claim 1, wherein: The step of performing master-slave synchronization confirmation according to the file segmentation message includes: Verifying the message type of the file segmentation message by using a slave queuing machine, wherein the slave queuing machine and the master queuing machine adopt a dual-machine hot standby operation mechanism; In response to the verification being passed, a slave water flow file is generated, wherein the slave water flow file is a newly generated water flow file corresponding to the slave queuing machine, and the slave water flow file is asynchronously consistent with the main water flow file; Writing the file segmentation message into a secondary stream file; In response to the writing success, a master-slave synchronization confirmation response is sent to the master queuing machine, wherein the master-slave synchronization confirmation response indicates that the master-slave synchronization confirmation between the master queuing machine and the slave queuing machine is successful.

3. The method according to claim 2, wherein: The snapshot message flow is generated by the following steps: Subscribe to the main stream water message of the main queuing machine; In response to the subscription being successful and the file segmentation message being received, performing a mirror copy on the memory database to obtain a memory data snapshot; The snapshot start message, the memory data snapshot and the snapshot end message are sequentially written into the initial message stream through an asynchronous thread to obtain the snapshot message stream.

4. The method according to claim 3, wherein: Before writing the snapshot message included in the snapshot message flow or the service request message included in the service request message flow into the updated main flow file in response to the master-slave synchronization confirmation being successful, the method further includes: Subscribe to the main stream water message of the main queuing machine; In response to the subscription being successful and receiving a message to be processed, determining a message type of the message to be processed, wherein the message to be processed is a service request message sent by a service gateway and published by the main queuing machine; In response to the message type of the to-be-processed message being a non-memory data snapshot type, performing message processing according to the service request message to generate a request processing result; The request processing result is sent to the service gateway, so that the request processing result is forwarded to the client corresponding to the message to be processed through the service gateway.

5. The method according to claim 4, wherein: The generating of the file segmentation message comprises: Determine the operation status of the master queuing machine and the slave queuing machine respectively to generate a first operation status and a second operation status; In response to determining that the first operation state indicates that the master queue machine is operating normally and the second operation state indicates that the slave queue machine is operating normally, determining whether there is a historical file segmentation message, wherein the historical file segmentation message is a fence message corresponding to a historical pipeline file generated before the file segmentation message; In response to the existence of the historical file segmentation message, determining whether the historical snapshot message flow and the historical service request message flow have been written into the historical pipeline file; In response to the completion of writing, the file segmentation message is generated by the main queue machine.

6. The method according to claim 5, wherein: The method further comprises: In response to receiving a replay request for the updated mainstream water file, sequentially reading the updated mainstream water file to obtain a service message sequence; For each service message in the service message sequence, the following processing steps are performed: In response to determining that the service message is a snapshot start message, initializing the memory database; In response to determining that the service message is a memory data snapshot and the memory database is initialized, saving the data in the memory data snapshot into the memory database; In response to the business message being a business request message and the in-memory database of the database initialization not being loaded completely, placing the business message into a first-in-first-out message queue; In response to determining that the business message is a snapshot end message, the memory database state is marked as data loading completed, and the business messages are read sequentially from the first-in first-out message queue and processed in the order in which the messages are read.

7. A file processing device applied to a journal file, comprising: A first generating unit is configured to generate a file segmentation message, wherein the file segmentation message is a fence message for a pipeline file generated by a main queueing machine; A first master-slave synchronization confirmation unit is configured to perform master-slave synchronization confirmation according to the file segmentation message; A second generating unit is configured to generate a main stream file in response to the master-slave synchronization confirmation success, wherein the main stream file is a newly generated stream file corresponding to the main queue machine; A first writing unit is configured to write the file segmentation message as a service message into the main stream water file to obtain an updated main stream water file; a second master-slave synchronization confirmation unit, configured to, in response to receiving a snapshot message flow or a service request message flow, perform master-slave synchronization confirmation on the snapshot message flow and the service request message flow, wherein the snapshot message flow is a message flow including snapshot messages copied by the memory database mirror, and the service request message is a message flow including service request messages sent by the service gateway; The second writing unit is configured to write the snapshot message included in the snapshot message flow or the service request message included in the service request message flow into the updated main flow file in response to the master-slave synchronization confirmation success.

8. An electronic device comprising: one or more processors; a storage device having one or more programs stored thereon; When the one or more programs are executed by the one or more processors, the one or more processors implement the method according to any one of claims 1 to 6.

9. A computer readable medium having a computer program stored thereon, wherein: When the computer program is executed by a processor, the method according to any one of claims 1 to 6 is implemented.

Images