Data processing method and device, equipment, storage medium and computer program product

Abstract

The application discloses a data processing method, device and equipment, a storage medium and a computer program product. The method comprises the following steps: performing object conversion on each original data message obtained; and sending each original data message which is successfully converted to a preset terminal. Through the application, a large number of ticket message information generated by a ticket system provider (GDS) can be processed in time, and the data accuracy and stability are relatively high, thereby solving the technical problems that massive real-time data cannot be processed and the data accuracy and stability are relatively low in the related art.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of data processing, and in particular to a data processing method, device, equipment, storage medium and computer program product. BACKGROUND

[0002] Currently, a ticket system provider (GDS) is required to send data generated by agents when booking, canceling, changing or invalidating tickets in real time to the International Air Transport Association (IATA) through an XML message. The main content of the data includes booking agents, ticket numbers, transaction codes, IATA numbers, booking times, payment methods, payment amounts, and currency names. The IATA uses the data to monitor the real-time sales risk of agents and requires all GDSs around the world to send real-time risk control data to the IATA.

[0003] According to the requirements of the IATA, the data message type generated by the GDS for BP needs to be processed in real time, and the data needs to be accurately and correctly sent to the IATA (International Air Transport Association), and the accuracy and stability of the data need to reach 99.999%. However, the GDS generates a large amount of ticket message information every moment, and the existing technology uses a traditional non-distributed application program, which cannot process massive real-time data and meet the requirements of the IATA. SUMMARY

[0004] The present application provides a data processing method, device, equipment, storage medium and computer program product, which can solve the technical problems of being unable to process massive real-time data and having low data accuracy and stability in the prior art.

[0005] To achieve the above object, the present application provides the following technical solutions.

[0006] In a first aspect, the present application provides a data processing method, which comprises the following steps.

[0007] Converting each obtained original data message into an object;

[0008] Sending each original data message that is successfully converted to a preset terminal.

[0009] In a second aspect, the present application provides a data processing device, which comprises the following modules.

[0010] A data processing module configured to convert each obtained original data message into an object;

[0011] A data sending module configured to send each original data message that is successfully converted to a preset terminal.

[0012] In a third aspect, an embodiment of the present application further provides an electronic device, comprising a memory and a processor; the processor is configured to read and execute a computer program stored in the memory, so as to realize the steps of the data processing method.

[0013] In a fourth aspect, an embodiment of the present application further provides a computer storage medium, wherein the computer storage medium stores computer executable instructions, and the computer executable instructions realize the steps of the data processing method when executed.

[0014] In a fifth aspect, an embodiment of the present application further provides a computer program product, comprising computer program / instructions, and the computer program / instructions realize the steps of the data processing method when executed by a processor.

[0015] The technical scheme provided by the embodiment of the present application has the following beneficial effects:

[0016] The object conversion is performed on each obtained original data message, and each original data message after successful conversion is sent to a preset terminal. Through the present application, a large number of ticket message information generated by a ticket system provider (GDS) can be processed in time, and the data accuracy and stability are high, thereby solving the technical problem of inability to process massive real-time data and low data accuracy and stability in the related art. BRIEF DESCRIPTION OF DRAWINGS

[0017] In order to more clearly illustrate the technical scheme in the embodiments of the present application, the drawings needed in the embodiment description will be briefly introduced. Obviously, the drawings in the following description are only some embodiments of the present application, and other drawings can be obtained by those skilled in the art without creative effort.

[0018] Figure 1 The figure is a flowchart of the first embodiment of the data processing method of the present application.

[0019] Figure 2 The figure is a topology diagram of the overall architecture of the data processing method of the present application.

[0020] Figure 3 The figure is a flowchart of the overall processing flow of an original data message of the data processing method of the present application.

[0021] Figure 4 The figure is a flowchart of the second embodiment of the data processing method of the present application.

[0022] Figure 5 The figure is a functional module diagram of an embodiment of the data processing device of the present application.

[0023] Figure 6Fig. 1 is a structural schematic diagram of an electronic device according to an embodiment of the present application. DETAILED DESCRIPTION

[0024] The technical solutions in the embodiments of the present application will be clearly and completely described below with reference to the drawings in the embodiments of the present application. Obviously, the described embodiments are only part of the embodiments of the present application, rather than all the embodiments of the present application. Based on the embodiments in the present application, all other embodiments obtained by a person of ordinary skill in the art without creative effort belong to the protection scope of the present application.

[0025] In order to make the objects, technical solutions and advantages of the present application clearer, the embodiments of the present application will be described in detail below with reference to the drawings.

[0026] In a first aspect, the embodiments of the present application provide a data processing method.

[0027] In an embodiment, with reference to Figure 1 , Figure 1 Fig. 1 is a flow schematic diagram of a data processing method according to a first embodiment of the present application. As shown in the figure, the data processing method comprises: Figure 1

[0028] Step S10, converting each original data message obtained into an object;

[0029] In the embodiment, the object conversion is a process of converting an object from one type to another type. With reference to Figure 2 , Figure 2 Fig. 2 is a whole architecture topology diagram of the data processing method according to the present application. As shown in the figure, the data stream is obtained from REDIS SPOUT in real time and is handed over to the downstream STORM BOLT (ConverBolt), and each original data message obtained is converted from BP xml to IATA xml through the ConverBolt. Figure 2

[0030] Figure 2 In the figure, topology represents a topology diagram; STORM: a technology for real-time data processing in the field of big data; REDIS SPOUT: Spout is a data source in Storm, which represents a REDIS data source here; ConverBolt: Bolt is a data processing unit in Storm, which is responsible for processing and converting the received data, and ConverBolt mainly represents a data file processing and conversion service here; DB SPOUT: SPOUT is a data source in Storm, which represents a database data source here; PushtBolt: Bolt is a data processing unit in Storm, which is responsible for processing and converting the received data, and PushtBolt mainly represents a data sending service here; and tuple represents a data tuple.​​

[0031] Specifically, in an embodiment, the preset terminal is taken as the International Air Transport Association (IATA) for example, and nine data protocol scenarios of the IATA correspond to five data message types of the China Aviation Service BP ticket. The five different BP ticket data message types are automatically assembled into corresponding five data processors, and each processor processes each data protocol of the IATA.

[0032] The five data message types of the China Aviation Service BP ticket are shown in Table 1.

[0033] Table 1

[0034]

[0035] BP: a kind of ticket data message type in the field of civil aviation. The original BP data message is XML, which is converted into a JAVA object through JAVA, and the message content is processed according to the requirements of the IATA and is sent to the interface of the other party through HTTPS. The mapping relationship between the nine scenes of risk control implemented by the IATA for BP risk control and the five data message types of the China Aviation Service BP ticket is shown in Table 2.

[0036] Table 2

[0037]

[0038] According to the mapping relationship between the nine scenes of risk control implemented by the IATA for BP risk control and the five data message types of the China Aviation Service BP ticket in Table 2, corresponding logical processing is performed on each data message type, and the JAVA object meeting the requirements of the IATA message is converted into XML data results. The data message field requirements of the IATA are as follows:

[0039] ticketAmount: total ticket price (including tax);

[0040] Payment: payment method, including CC credit card payment, CASH cash payment, and mixed payment method;

[0041] operationCode: two-character code of the country where the ticket is issued;

[0042] ticketId: ticket number;

[0043] agnetNumericCode: IATA settlement number;

[0044] issueDate Operation timestamp, for example, 2019-09-05T12:30:00;

[0045] issuingCurrency Currency code (not required for this field when the ticket is voided).

[0046] According to the requirements of IATA for the result message, the data conversion model of the above-mentioned 7 fields is shown in Table 3:

[0047] Table 3

[0048]

[0049]

[0050] Step S20, send each of the original data messages converted successfully to a preset terminal.

[0051] In this embodiment, continuing to refer to Figure 2 , after the object conversion of each original data message obtained by ConverBolt, before data transmission, the JAVA object meeting the requirements of the message of IATA is converted into XML data result as shown in Table 3, and then each of the original data messages converted successfully is sent to PushtBolt, referring to Figure 3 , Figure 3 is a schematic diagram of the overall processing flow of the original data message of the data processing method of the present application. As shown in Figure 3 , in PushtBolt, through API, each of the original data messages converted successfully is sent to IATA by HTTPS. HTTPS (full name: Hypertext Transfer Protocol Secure) is a secure HTTP channel, which guarantees the security of the transmission process through transmission encryption and identity authentication on the basis of HTTP.

[0052] Furthermore, DB SPOUT is used to store the original data packets of each conversion failure and each successful conversion sent, so as to record the results and facilitate subsequent review. It should be noted that each original data packet sent for successful conversion, regardless of whether it is sent successfully or failed, will be stored in DB SPOUT, so that each original data packet of each successful conversion that failed to be sent can be resent from DB SPOUT to IATA later, until the number of transmissions reaches the preset number. If the transmission fails, the subsequent program will rescan and send again to avoid the situation where network problems cause data failure and the data cannot be correctly sent to IATA. The Ack mechanism in Storm is a confirmation mechanism used to ensure the reliable transmission of data in the topology. The STORM ACK mechanism, combined with the tracking of each link in data processing and the data retransmission mechanism, greatly improves the accuracy of the results of massive data processing. The cluster composed of multiple machines avoids single point failure protection and greatly improves the stability of massive data processing results.

[0053] Figure 3 In the figure, REDIS SPOUT: Spout is the data source in Storm, which here refers to the REDIS data source; ConverBolt: Bolt is the data processing unit in Storm, responsible for processing and converting the received data, which mainly refers to the data file processing and conversion service; PushtBolt: Bolt is the data processing unit in Storm, responsible for processing and converting the received data, which mainly refers to the data sending service; DB SPOUT: Spout is the data source in Storm, which here refers to the database data source.

[0054] In this embodiment, object conversion is performed on each acquired raw data message; each successfully converted raw data message is sent to IATA. This embodiment enables timely processing of large amounts of passenger ticket message information generated by the general data system (GDS), with high data accuracy and stability. This addresses the technical issues of related technologies such as the inability to process massive amounts of real-time data and the low accuracy and stability of the data.

[0055] Optionally, in one embodiment, referring to Figure 4 , Figure 4 FIG. 1 is a flow chart of the second embodiment of the data processing method of the present invention. Figure 4 As shown, before step S10, the following steps are included:

[0056] Step S001: generating an identifier corresponding to each original data message obtained;

[0057] Step S002, store each of the original data messages to the REDIS MSGIDXML queue, and set the survival time length of each of the original data messages in the REDIS MSGIDXML queue.

[0058] In the embodiment, the REDIS cluster is installed and the message queue is designed to receive the real-time data sent by the upstream, the REDIS can play a role of peak shaving and valley filling, and the data is decoupled from the source of the upstream data. With reference to the foregoing description, the three message queues designed by the REDIS include ORGXML, MSGXML and FAILXML. The ORGXML is used to receive the upstream original XML data; the MSGXML is used to store the original XML data currently being processed by the system, and the FAILXML is used to store the original XML data failed in the logical processing of the system. Figure 3

[0059] For each of the original XML data obtained, the identifier MSGID corresponding to each of the original data messages is generated, and each of the original XML data is stored to the REDIS MSGIDXML queue, and the survival time length of each of the original data messages in the REDIS MSGIDXML queue is set. The original XML data exceeding the survival time length in the REDIS MSGIDXML queue is discarded.

[0060] Optionally, in an embodiment, after step S10, the following steps are included.

[0061] For each of the original data messages converted successfully, the identifier corresponding to each of the original data messages is sent to the ACK mechanism.

[0062] After the ACK mechanism receives each of the identifiers, the original data message corresponding to each of the identifiers is deleted from the REDIS MSGIDXML queue.

[0063] In the embodiment, with reference to the foregoing description, Figure 3 The reliability guarantee of the data processing is performed by the STORM ack / fail mechanism, and the message processing reliability starts from the Spout. In order to ensure that the data can be correctly processed, the STORM can track each of the original data messages generated by the Spout and process the original data messages by using the STORM ack / fail mechanism. STORM: a technology for real-time data processing in the field of big data.

[0064] ​For each original data packet that is successfully converted, the identifier corresponding to each original data packet that is successfully converted is sent to the ACK mechanism, and the ACK mechanism is triggered by callback. After the ACK mechanism receives the identifier corresponding to each original data packet that is successfully converted, the original data packet corresponding to each identifier that is successfully converted is deleted from the REDIS MSGIDXML queue, preventing a large amount of useless data from being retained. The ACK (Acknowledge character) is a transmission class control character used in data communication. In the TCP / IP protocol, the ACK mechanism is used to confirm the successful reception of data, ensuring the integrity and reliability of the data.

[0065] Optionally, in an embodiment, after step S10, the following steps are included:

[0066] For each original data packet that fails to be converted, the identifier corresponding to each original data packet is sent to the FAIL mechanism.

[0067] After the FAIL mechanism receives each identifier, the original data packet corresponding to each identifier is deleted from the REDIS MSGIDXML queue, and the original data packet corresponding to each identifier is stored in the REDIS FAILXML queue.

[0068] In this embodiment, the foregoing description continues to refer to Figure 3 Since FAILXML is used for original XML data that fails logical processing in the system, for each original data packet that fails to be converted, the original data packet that fails to be converted is stored in the convetxml_error table of the DB (POSTGRESQL) database, and the identifier corresponding to each original data packet that fails to be converted is sent to the FAIL mechanism, and the FAIL mechanism is triggered by callback. After the FAIL mechanism receives the identifier corresponding to each original data packet that fails to be converted, the original data packet that fails to be converted corresponding to each identifier is deleted from the REDIS MSGIDXML queue, and the original data packet that fails to be converted corresponding to each identifier is stored in the REDIS FAILXML queue, facilitating subsequent manual intervention processing. The FAIL mechanism refers to providing a backup solution or error handling strategy when the system has an exception or error, to ensure the reliability and stability of the system.

[0069] Optionally, in an embodiment, after step S10, the following steps are included:

[0070] Detecting whether conversion is successful;

[0071] If the original data packet is converted into a preset terminal required message type, it is determined that the conversion is successful;

[0072] If the message content required by the preset terminal is missing in the original data message or the program is abnormal during the conversion, it is determined that the conversion fails.

[0073] In this embodiment, it is determined whether the object conversion of each original data message obtained is successful, mainly identifying the message type corresponding to the original data message, as shown in Table 1 and Table 2. If the message type corresponding to the original data message is correctly converted into the required message format corresponding to IATA, it is determined that the conversion is successful.

[0074] If the data content required by the preset terminal is missing in the original data message or the program is abnormal during the conversion, it is determined that the conversion of the original message fails.

[0075] Optionally, in an embodiment, continuing to refer to Figure 4 After step S20, it includes:

[0076] Step S30, taking each target original data message as each original data message obtained, returning to execute the step of converting the object of each original data message obtained until there is no target original data message in each original data message converted successfully.

[0077] The target original data message is data that fails to be sent and the number of sending times is less than the preset number of times in each original data message converted successfully.

[0078] In this embodiment, continuing to refer to Figure 3 After each original data message converted successfully is sent to IATA, the sent each original data message converted successfully is saved in the sendxml_record table of the DB (POSTGRESQL) database, and the sending state (success or failure) and the number of sending times are recorded. It is easily thought that the number of sending times of each original data message converted successfully is 1 in the first cycle.

[0079] Due to network or interface response problems, during the process of sending each original data message converted successfully to IATA, sending failure may occur. Taking 3 times as the preset number of times as an example, data with a sending state of failure and a number of sending times less than 3 times is selected from each original data message converted successfully, and the selected data is taken as a target original data message.

[0080] With each target original data message as each obtained original data message, the step of performing object conversion on each obtained original data message is returned, each original data message converted successfully is sent to the IATA, each time the loop is updated once the sending state (success or failure) of the target original data message and the sending times (the sending times are added by 1), until there is no target original data message in each original data message converted successfully, the loop is ended, so as to guarantee that each original data message converted successfully can be accurately and correctly sent to the IATA. Further, for each original data message converted successfully with the sending times reaching 3 times, manual intervention is performed for processing.

[0081] In a second aspect, the embodiment of the present application further provides a data processing device.

[0082] In an embodiment, the data processing device comprises Figure 5 , Figure 5 The figure is a schematic diagram of the functional modules of the data processing device of an embodiment of the present application. As shown in the figure, the data processing device comprises: Figure 5

[0083] The data processing module 10 is configured to perform object conversion on each obtained original data message;

[0084] The data sending module 20 is configured to send each original data message converted successfully to a preset terminal.

[0085] Optionally, in an embodiment, the data processing device further comprises a setting module configured to:

[0086] For each obtained original data message, an identifier corresponding to each original data message is generated;

[0087] Each original data message is stored in a REDIS MSGIDXML queue, and a survival time length of each original data message in the REDIS MSGIDXML queue is set.

[0088] Optionally, in an embodiment, the data processing module 10 is further configured to:

[0089] For each original data message converted successfully, an identifier corresponding to each original data message is sent to an ACK mechanism;

[0090] After the ACK mechanism receives each identifier, the original data message corresponding to each identifier is deleted from the REDIS MSGIDXML queue.

[0091] Optionally, in an embodiment, the data processing module 10 is further configured to:

[0092] ​For each of the original data packets that fail to be converted, the identifier corresponding to each of the original data packets is sent to the FAIL mechanism;

[0093] After receiving each of the identifiers, the FAIL mechanism deletes the original data packet corresponding to each of the identifiers from the REDIS MSGID XML queue and stores the original data packet corresponding to each of the identifiers to the REDIS FAIL XML queue.

[0094] Optionally, in an embodiment, the data processing apparatus further comprises a detection module configured to:

[0095] detect whether the conversion is successful;

[0096] if the original data packet is converted into the message type required by the preset terminal, it is determined that the conversion is successful;

[0097] if the message content required by the preset terminal is missing in the original data packet during the conversion process or there is an internal exception in the program during the conversion process, it is determined that the conversion fails.

[0098] Optionally, in an embodiment, the data sending module 20 is further configured to:

[0099] return to performing the step of object conversion on each of the obtained original data packets with each of the target original data packets as the obtained original data packet until there is no target original data packet in each of the original data packets that successfully converts.

[0100] wherein the target original data packet is data that fails to be sent and has a sending number less than a preset number in each of the original data packets that successfully converts.

[0101] Wherein, the functions of each module in the above data processing apparatus correspond to each step in the above data processing method embodiment, and the functions and implementation processes will not be repeated here.

[0102] In a third aspect, the embodiments of the present application also provide an electronic device, as shown in Figure 6 The electronic device comprises a memory and a processor, and the processor is configured to read and execute a computer program stored in the memory to implement the above data processing method.

[0103] In a fourth aspect, the embodiments of the present application also provide a computer storage medium, which stores computer executable instructions, and the computer executable instructions implement the above data processing method when executed.

[0104] In a fifth aspect, an embodiment of the present application provides a computer program product stored in a storage medium, which is executed by at least one processor to implement each process of the above-mentioned data processing method embodiment and achieve the same technical effects. To avoid repetition, details are not described herein.

[0105] Finally, it should be noted that in some of the processes described in the embodiments of the present application, a plurality of operations or steps appear in a particular order, but it should be understood that these operations or steps can be performed in an order different from that in which they appear in the embodiments of the present application or in parallel, and the serial number of the operations is only used to distinguish different operations, and the serial number itself does not represent any execution order. In addition, these processes can include more or fewer operations, and these operations or steps can be performed in sequence or in parallel, and these operations or steps can be combined.

[0106] The above only describes the preferred embodiments of the present application and is not intended to limit the present application. Although the present application has been described in detail with reference to the foregoing embodiments, those skilled in the art can modify the technical solutions described in the foregoing embodiments or make equivalent replacements to some of the technical features, and any modification, equivalent replacement, improvement, etc. made within the spirit and principles of the present application shall be included in the protection scope of the present application.

Claims

1. A data processing method, characterized in that: The method is implemented based on Storm computing and includes: Performing object conversion on each acquired original data message; the original data message is a passenger ticket data message type in the civil aviation field; For each original data message that is successfully converted, sending an identifier corresponding to each original data message to an ACK mechanism; After receiving each of the identifiers, the ACK mechanism deletes the original data message corresponding to each of the identifiers from the REDISMSGIDXML queue; For each original data message that fails to be converted, sending an identifier corresponding to each original data message to the FAIL mechanism; After receiving each of the identifiers, the FAIL mechanism deletes the original data message corresponding to each of the identifiers from the REDISMSGIDXML queue, and stores the original data message corresponding to each of the identifiers into the REDIS FAILXML queue; Sending each successfully converted original data message to a preset terminal; Taking each target original data message as each obtained original data message, returning to execute the step of object conversion on each obtained original data message until the target original data message does not exist in each of the successfully converted original data messages; The target original data message is the data that failed to be sent and whose number of sending times is less than a preset number in each of the successfully converted original data messages.

2. The data processing method according to claim 1, wherein: Before the step of performing object conversion on each acquired original data message, the method includes: For each acquired original data message, generate an identifier corresponding to each original data message; Each of the original data messages is stored in the REDIS MSGIDXML queue, and the survival time of each of the original data messages in the REDIS MSGIDXML queue is set.

3. The data processing method according to claim 1, wherein: After the step of performing object conversion on each acquired original data message, the method includes: Check whether the conversion is successful; If the original data message is converted into the message type required by the preset terminal, it is determined that the conversion is successful; If the message content required by the preset terminal is missing in the original data message during the conversion process or there is an internal abnormality in the program during the conversion process, it is determined that the conversion has failed.

4. A data processing device, characterized in that: The device is implemented based on Storm computing and includes: A data processing module is configured to perform object conversion on each acquired original data message; the original data message is a passenger ticket data message type in the civil aviation field; For each original data message that is successfully converted, an identifier corresponding to each original data message is sent to the ACK mechanism; after the ACK mechanism receives each identifier, the original data message corresponding to each identifier is deleted from the REDIS MSGIDXML queue; For each original data message that fails to be converted, an identifier corresponding to each original data message is sent to the FAIL mechanism; after receiving each identifier, the FAIL mechanism deletes the original data message corresponding to each identifier from the REDIS MSGIDXML queue and stores the original data message corresponding to each identifier in the REDISFAILXML queue; A data sending module, configured to send each successfully converted original data message to a preset terminal; Taking each target original data message as each original data message obtained, return to execute the step of object conversion for each original data message obtained until the target original data message does not exist in each of the original data messages that are successfully converted; wherein, the target original data message is the data in each of the original data messages that are successfully converted that fails to be sent and the number of times it is sent is less than the preset number of times.

5. An electronic device, characterized in that: include: memory and processor; The processor is configured to read and execute the computer program stored in the memory to implement the steps of the data processing method according to any one of claims 1 to 3.

6. A computer-readable storage medium, characterized in that The computer-readable storage medium stores computer-executable instructions, and when the computer-executable instructions are executed, the steps of the data processing method according to any one of claims 1 to 3 are implemented.

7. A computer program product comprising a computer program / instructions, characterized in that When the computer program / instruction is executed by a processor, the steps of the data processing method according to any one of claims 1 to 3 are implemented.

Images