Data processing method and device, equipment, storage medium and computer program product

Abstract

The invention discloses a data processing method and device, equipment, a storage medium and a computer program product. The method comprises the following steps: performing object conversion on each obtained original data message; and sending each successfully converted original data message to a preset terminal. According to the invention, a large amount of passenger ticket message information generated by an air ticket system supplier (GDS) can be processed in time, the data accuracy and stability are high, and the technical problems that massive real-time data cannot be processed and the data accuracy and stability are low in the prior art are solved.

Description

Technical Field

[0001] The present invention relates to the field of data processing technology, and in particular to a data processing method, device, equipment, storage medium and computer program product. Background Art

[0002] Currently, the Global Distribution System (GDS) is required to send the data generated by agents when issuing, refunding, changing and canceling tickets to IATA in real time via XML messages. The data mainly includes: issuing agent, ticket number, transaction code, IATA number, ticket issuance time, payment method, payment amount, currency name, etc. IATA uses this data to monitor the sales risks of agents in real time and requires GDSs around the world to send real-time risk control data to IATA.

[0003] According to IATA requirements, among the ticket data message types generated by GDS, the BP data message type needs to be processed in real time and accurately sent to IATA (International Air Transport Association), and the data accuracy and stability must reach 99.999%. However, GDS generates a large amount of ticket message information every moment, and the existing technology uses traditional non-distributed applications, which cannot process massive real-time data and meet IATA requirements. Summary of the invention

[0004] The present invention provides a data processing method, device, equipment, storage medium and computer program product, which can solve the technical problems existing in the prior art that massive real-time data cannot be processed and the data accuracy and stability are low.

[0005] To achieve the above object, the present invention provides the following technical solutions:

[0006] In a first aspect, an embodiment of the present invention provides a data processing method, the method comprising:

[0007] Perform object conversion on each acquired raw data message;

[0008] Each of the original data messages that is successfully converted is sent to a preset terminal.

[0009] In a second aspect, an embodiment of the present invention provides a data processing device, the device comprising:

[0010] A data processing module is configured to perform object conversion on each acquired original data message;

[0011] The data sending module is configured to send each of the successfully converted original data messages to a preset terminal.

[0012] In a third aspect, an embodiment of the present invention further provides an electronic device, comprising: a memory, a processor; the processor is used to read and execute a computer program stored in the memory to implement the steps of the aforementioned data processing method.

[0013] In a fourth aspect, an embodiment of the present invention further provides a computer storage medium, wherein the computer storage medium stores computer executable instructions, and the computer executable instructions implement the steps of the aforementioned data processing method when executed.

[0014] In a fifth aspect, an embodiment of the present invention further provides a computer program product, including a computer program / instruction, which, when executed by a processor, implements the steps of the aforementioned data processing method.

[0015] The beneficial effects brought by the technical solution provided by the embodiment of the present invention include:

[0016] Each acquired original data message is subjected to object conversion; each successfully converted original data message is sent to a preset terminal. Through the present invention, a large amount of passenger ticket message information generated by a ticket system supplier (GDS) can be processed in a timely manner, and the data accuracy and stability are high, solving the technical problem that the related technology cannot process massive real-time data and the data accuracy and stability are low. BRIEF DESCRIPTION OF THE DRAWINGS

[0017] In order to more clearly illustrate the technical solutions in the embodiments of the present invention, the following briefly introduces the drawings required for use in the description of the embodiments. Obviously, the drawings described below are only some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without creative work.

[0018] Figure 1 It is a flowchart of a first embodiment of a data processing method of the present invention;

[0019] Figure 2 It is the overall architecture topology diagram of the data processing method of the present invention;

[0020] Figure 3 This is a schematic diagram of the overall processing flow of the original data message of the data processing method of the present invention;

[0021] Figure 4 It is a flowchart of a second embodiment of a data processing method of the present invention;

[0022] Figure 5 A schematic diagram of functional modules of an embodiment of a data processing device of the present invention;

[0023] Figure 6The figure is a schematic diagram of the structure of an electronic device according to an embodiment of the present invention. DETAILED DESCRIPTION

[0024] The following will be combined with the drawings in the embodiments of the present invention to clearly and completely describe the technical solutions in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0025] To make the objectives, technical solutions and advantages of the present invention more clear, the embodiments of the present invention will be described in detail below with reference to the accompanying drawings.

[0026] In a first aspect, an embodiment of the present invention provides a data processing method.

[0027] In one embodiment, referring to Figure 1 , Figure 1 FIG. 1 is a flow chart of the first embodiment of the data processing method of the present invention. Figure 1 As shown, the data processing method includes:

[0028] Step S10, performing object conversion on each acquired original data message;

[0029] In this embodiment, object conversion is the process of converting an object from one type to another type. Figure 2 , Figure 2 This is the overall architecture topology diagram of the data processing method of the present invention. Figure 2 As shown, the data stream is obtained from REDIS SPOUT in real time and handed over to the downstream STORM BOLT (ConverBolt), and each obtained original data message is converted from BP xml to IATA xml through ConverBolt.

[0030] Figure 2 In the figure, topology means topology graph; STORM: technology for real-time data processing in the field of big data; REDISSPOUT: Spout is the data source in Storm, which means REDIS data source here; ConverBolt: Bolt is the data processing unit in Storm, responsible for processing and converting the received data, here ConverBolt mainly means data file processing and conversion service; DB SPOUT: SPOUT is the data source in Storm, which means database data source here; PushtBolt: Bolt is the data processing unit in Storm, responsible for processing and converting the received data, here PushtBolt mainly means data sending service; tuple: means data tuple.

[0031] Specifically, in one embodiment, taking the preset terminal as IATA as an example, the 9 data protocol scenarios of IATA correspond to the 5 data message types of BP tickets of TravelSky. The data message types of the 5 different BP tickets are automatically assembled to the corresponding 5 data processors, and each processor processes each data protocol of IATA.

[0032] The five data message types of TravelSky BP tickets are shown in Table 1:

[0033] Table 1

[0034] EVENT describe BPISSUESUCCESS BP ticket issued successfully BPREISSUESUCCESS BP exchange successful BPVOIDSUCCESS BP successfully abolished the vote BPREFUNDSUCCESS BP refund successful BPREFUNDCANCEL BP Refund Cancellation Event Type

[0035] BP: A type of ticket data message in the civil aviation field. The original BP data message is XML. XML is converted into a JAVA object through JAVA. The message content is processed according to the requirements of IATA and sent to the other party's interface through HTTPS. The mapping relationship between the 9 risk control scenarios of BP risk control implemented by the International Air Transport Association (IATA) and the 5 data message types of China Aviation Information BP tickets is shown in Table 2:

[0036] Table 2

[0037]

[0038]

[0039]

[0040] According to the mapping relationship between the 9 scenarios of IATA's implementation of BP risk control and the 5 data message types of BP tickets of TravelSky in Table 2, each data message type is processed logically accordingly, and the JAVA objects that meet the IATA message requirements are converted into XML data results. IATA's requirements for data message fields are as follows:

[0041] ticketAmount: total ticket price (tax included);

[0042] Payment methods include CC credit card payment, CASH cash payment, and mixed payment methods;

[0043] operationCode: The two-letter country code of the ticket issuing place;

[0044] ticketId ticket number;

[0045] agnetNumericCode Airline Association settlement number;

[0046] issueDate operation timestamp, for example, 2019-09-05T12:30:00;

[0047] issuingCurrency Currency code (this field is not required for voided bills).

[0048] According to IATA's requirements for result messages, the data conversion model of the above 7 fields is shown in Table 3:

[0049] Table 3

[0050]

[0051]

[0052] Step S20, sending each of the successfully converted original data messages to a preset terminal.

[0053] In this embodiment, continue to refer to Figure 2 After each original data message is converted by ConverBolt, the JAVA object that meets the requirements of the IATA message is converted into an XML data result before the data is sent, as shown in Table 3, and then each successfully converted original data message is sent to PushtBolt. Figure 3 , Figure 3 FIG. 1 is a schematic diagram of the overall processing flow of the data processing method of the present invention - original data message. Figure 3 As shown, in PushtBolt, each successfully converted raw data message is sent to IATA via API using HTTPS. HTTPS (full name: Hypertext Transfer Protocol Secure) is an HTTP channel with security as its goal. It ensures the security of the transmission process through transmission encryption and identity authentication based on HTTP.

[0054] Furthermore, DB SPOUT is used to store the original data packets of each conversion failure and each successful conversion sent, so as to realize result recording and facilitate subsequent viewing. It should be noted that each successfully converted original data packet sent, regardless of whether it is sent successfully or failed, will be stored in DB SPOUT, so that each successfully converted original data packet that failed to be sent can be resent from DB SPOUT to IATA later, until the number of transmissions reaches the preset number. If the transmission fails, the subsequent program will rescan and send again to avoid the situation where the data fails due to network problems and the data cannot be correctly sent to IATA. And the Ack mechanism in Storm is a confirmation mechanism to ensure the reliable transmission of data in the topology. Through the STORMACK mechanism, the tracking of each link of data processing and the guarantee of data retransmission mechanism, the accuracy of the results of massive data processing is greatly improved. The cluster composed of multiple machines avoids single point failure protection and greatly improves the stability of massive data processing results.

[0055] Figure 3 In the figure, REDIS SPOUT: Spout is the data source in Storm, which refers to the REDIS data source here; ConverBolt: Bolt is the data processing unit in Storm, responsible for processing and converting the received data, which mainly refers to the data file processing and conversion service here; PushtBolt: Bolt is the data processing unit in Storm, responsible for processing and converting the received data, which mainly refers to the data sending service here; DB SPOUT: Spout is the data source in Storm, which refers to the database data source here.

[0056] In this embodiment, object conversion is performed on each acquired original data message; each successfully converted original data message is sent to IATA. Through this embodiment, a large amount of passenger ticket message information generated by the ticket system supplier (GDS) can be processed in a timely manner, and the data accuracy and stability are high, which solves the technical problem that the related technology cannot process massive real-time data and the data accuracy and stability are low.

[0057] Optionally, in one embodiment, referring to Figure 4 , Figure 4 FIG. 2 is a flow chart of the second embodiment of the data processing method of the present invention. Figure 4 As shown, before step S10, it includes:

[0058] Step S001, for each acquired original data message, generating an identifier corresponding to each original data message;

[0059] Step S002: store each of the original data messages in the REDIS MSGIDXML queue, and set the survival time of each of the original data messages in the REDIS MSGIDXML queue.

[0060] In this embodiment, a REDIS cluster is installed and a message queue is designed to receive real-time data sent from upstream. The use of REDIS can play a role in eliminating peaks and filling valleys, making it easier to decouple data from the source of upstream data. Figure 3 ,The three message queues designed by REDIS include: ORGXML, MSGXML and FAILXML. ORGXML is used to receive the original XML data from upstream; MSGXML is used to store the original XML data currently being processed by the system, and FAILXML is used for the original XML data that failed logical processing in the system.

[0061] For each original XML data obtained, an identifier MSGID corresponding to each original data message is generated. At the same time, each original XML data is stored in the REDIS MSGIDXML queue, and the survival time of each original data message in the REDIS MSGIDXML queue is set. The original XML data in the REDIS MSGIDXML queue that exceeds the survival time is discarded.

[0062] Optionally, in one embodiment, after step S10, the following steps are included:

[0063] For each original data message that is successfully converted, sending an identifier corresponding to each original data message to an ACK mechanism;

[0064] After receiving each of the identifiers, the ACK mechanism deletes the original data message corresponding to each of the identifiers from the REDIS MSGIDXML queue.

[0065] In this embodiment, continue to refer to Figure 3 The reliability of data processing is guaranteed by the STORM ack / fail mechanism. The reliability of message processing starts from Spout. To ensure that data can be processed correctly, STORM can track each raw data message generated by Spout and use the STORM ack / fail mechanism to process it. STORM: A technology used for real-time data processing in the field of big data.

[0066] For each successfully converted original data message, the identifier corresponding to each successfully converted original data message is sent to the ACK mechanism, and the ACK mechanism is triggered by callback. After the ACK mechanism receives the identifier corresponding to each successfully converted original data message, it deletes the successfully converted original data message corresponding to each identifier from the REDIS MSGIDXML queue to prevent a large amount of useless data from being retained. Among them, ACK (Acknowledge character) is a confirmation character, which is a transmission control character used in data communication. In the TCP / IP protocol, the ACK mechanism is used to confirm the successful reception of data and ensure the integrity and reliability of the data.

[0067] Optionally, in one embodiment, after step S10, the following steps are included:

[0068] For each of the original data packets that fail to be converted, sending an identifier corresponding to each of the original data packets to the FALL mechanism;

[0069] After receiving each of the identifiers, the FALL mechanism deletes the original data message corresponding to each of the identifiers from the REDIS MSGIDXML queue, and stores the original data message corresponding to each of the identifiers into the REDISFALLXML queue.

[0070] In this embodiment, continue to refer to Figure 3 , since FAILXML is used for the original XML data that fails logical processing in the system, for each original data message that fails to be converted, each original data message that fails to be converted is stored in the convetxml_error table of the DB (POSTGRESQL) database, and the identifier corresponding to each original data message that fails to be converted is sent to the FALL mechanism, and the FALL mechanism is triggered by callback. After the FALL mechanism receives the identifier corresponding to each original data message that fails to be converted, the original data message that fails to be converted corresponding to each identifier is deleted from the REDIS MSGIDXML queue, and the original data message that fails to be converted corresponding to each identifier is stored in the REDIS FALLXML queue to facilitate subsequent manual intervention. Among them, the FALL mechanism refers to providing a backup plan or error handling strategy when an exception or error occurs in the system to ensure the reliability and stability of the system.

[0071] Optionally, in one embodiment, after step S10, the following steps are included:

[0072] Check whether the conversion is successful;

[0073] If the original data message is converted into the message type required by the preset terminal, it is determined that the conversion is successful;

[0074] If the message content required by the preset terminal is missing in the original data message during the conversion process or there is an internal abnormality in the program during the conversion process, it is determined that the conversion has failed.

[0075] In this embodiment, it is detected whether the object conversion of each acquired original data message is successful, mainly to identify the message type corresponding to the original data message, as shown in Table 1 and Table 2, and the message type corresponding to the original data message is correctly converted into the required message format corresponding to IATA, then it is determined that the conversion is successful.

[0076] If the data content required by the preset terminal is missing in the original data message during the conversion process, or there is an internal abnormality in the program during the conversion process, it is determined that the conversion of the original message has failed.

[0077] Optionally, in one embodiment, continue to refer to Figure 4 , after step S20, comprising:

[0078] Step S30, taking each target original data message as each original data message obtained, returning to execute the step of object conversion for each original data message obtained, until the target original data message does not exist in each of the original data messages that have been successfully converted;

[0079] The target original data message is the data that fails to be sent and has a number of transmissions less than a preset number of times in each of the original data messages that have been successfully converted.

[0080] In this embodiment, continue to refer to Figure 3 After each successfully converted raw data message is sent to IATA, each successfully converted raw data message is saved in the sendxml_record table of the DB (POSTGRESQL) database, and the sending status (successful or failed) and the number of times it is sent are recorded. It is easy to imagine that in the first cycle, the number of times each successfully converted raw data message is sent is 1.

[0081] Due to network or interface response problems, each successfully converted original data message may fail to be sent to IATA. Taking the preset number of times as 3 as an example, data with a failed sending status and a sending number of less than 3 times is selected from each successfully converted original data message, and the selected data is used as the target original data message.

[0082] Each target original data message is used as each original data message obtained, and the step of performing object conversion on each original data message obtained is returned, and each successfully converted original data message is sent to IATA. Each time the cycle is repeated, the sending status (success or failure) and the number of sending times (the number of sending times plus 1) of the target original data message are updated, until the target original data message does not exist in each successfully converted original data message, and the cycle ends, so as to ensure that each successfully converted original data message can be accurately sent to IATA. Furthermore, for each successfully converted original data message with a sending time of 3 times, manual processing is intervened.

[0083] In a second aspect, an embodiment of the present invention further provides a data processing device.

[0084] In one embodiment, referring to Figure 5 , Figure 5 FIG. 1 is a schematic diagram of the functional modules of an embodiment of a data processing device of the present invention. Figure 5 As shown, the data processing device includes:

[0085] The data processing module 10 is configured to perform object conversion on each acquired original data message;

[0086] The data sending module 20 is configured to send each successfully converted original data message to a preset terminal.

[0087] Optionally, in one embodiment, the data processing device further includes a setting module configured to:

[0088] For each acquired original data message, generate an identifier corresponding to each of the original data messages;

[0089] Each of the original data messages is stored in the REDIS MSGIDXML queue, and the survival time of each of the original data messages in the REDIS MSGIDXML queue is set.

[0090] Optionally, in one embodiment, the data processing module 10 is further configured to:

[0091] For each original data message that is successfully converted, sending an identifier corresponding to each original data message to an ACK mechanism;

[0092] After receiving each of the identifiers, the ACK mechanism deletes the original data message corresponding to each of the identifiers from the REDIS MSGIDXML queue.

[0093] Optionally, in one embodiment, the data processing module 10 is further configured to:

[0094] For each of the original data packets that fail to be converted, sending an identifier corresponding to each of the original data packets to the FALL mechanism;

[0095] After receiving each of the identifiers, the FALL mechanism deletes the original data message corresponding to each of the identifiers from the REDIS MSGIDXML queue, and stores the original data message corresponding to each of the identifiers into the REDISFALLXML queue.

[0096] Optionally, in one embodiment, the data processing device further includes a detection module configured to:

[0097] Check whether the conversion is successful;

[0098] If the original data message is converted into the message type required by the preset terminal, it is determined that the conversion is successful;

[0099] If the message content required by the preset terminal is missing in the original data message during the conversion process or there is an internal abnormality in the program during the conversion process, it is determined that the conversion has failed.

[0100] Optionally, in one embodiment, the data sending module 20 is further configured to:

[0101] Taking each target original data message as each original data message obtained, returning to execute the step of object conversion for each original data message obtained, until the target original data message does not exist in each of the original data messages that are successfully converted;

[0102] The target original data message is the data that failed to be sent and whose number of sending times is less than the preset number of times in each of the original data messages that were successfully converted.

[0103] Among them, the functional implementation of each module in the above-mentioned data processing device corresponds to each step in the above-mentioned data processing method embodiment, and its functions and implementation processes are no longer repeated here one by one.

[0104] In a third aspect, an embodiment of the present invention further provides an electronic device, whose structure is as follows: Figure 6 As shown, it includes: a memory and a processor, and the processor is used to read and execute the computer program stored in the memory to implement the aforementioned data processing method.

[0105] In a fourth aspect, an embodiment of the present invention further provides a computer storage medium, wherein the computer storage medium stores computer executable instructions, and the computer executable instructions implement the aforementioned data processing method when executed.

[0106] In the fifth aspect, an embodiment of the present invention provides a computer program product, which is stored in a storage medium and is executed by at least one processor to implement the various processes of the above-mentioned data processing method embodiment and can achieve the same technical effect. To avoid repetition, it will not be repeated here.

[0107] Finally, it should be noted that: in some processes described in the embodiments of the present invention, multiple operations or steps appearing in a specific order are included, but it should be understood that these operations or steps may not be executed in the order in which they appear in the embodiments of the present invention or may be executed in parallel, and the sequence number of the operation is only used to distinguish between different operations, and the sequence number itself does not represent any execution order. In addition, these processes may include more or fewer operations, and these operations or steps may be executed in sequence or in parallel, and these operations or steps may be combined.

[0108] The above description is only a preferred embodiment of the present invention and is not intended to limit the present invention. Although the present invention has been described in detail with reference to the aforementioned embodiments, it is still possible for those skilled in the art to modify the technical solutions described in the aforementioned embodiments, or to make equivalent substitutions for some of the technical features therein. Any modifications, equivalent substitutions, improvements, etc. made within the spirit and principles of the present invention should be included in the protection scope of the present invention.

Claims

1. A data processing method, characterized in that: The method comprises: Perform object conversion on each acquired raw data message; Each of the original data messages that is successfully converted is sent to a preset terminal.

2. The data processing method according to claim 1, characterized in that: Before the step of performing object conversion on each acquired original data message, the method includes: For each acquired original data message, generate an identifier corresponding to each of the original data messages; Each of the original data messages is stored in the REDIS MSGIDXML queue, and the survival time of each of the original data messages in the REDIS MSGIDXML queue is set.

3. The data processing method according to claim 2, characterized in that: After the step of performing object conversion on each acquired original data message, the method further comprises: For each original data message that is successfully converted, sending an identifier corresponding to each original data message to an ACK mechanism; After receiving each of the identifiers, the ACK mechanism deletes the original data message corresponding to each of the identifiers from the REDISMSGIDXML queue.

4. The data processing method according to claim 2, characterized in that: After the step of performing object conversion on each acquired original data message, the method further comprises: For each of the original data packets that fail to be converted, sending an identifier corresponding to each of the original data packets to the FALL mechanism; After receiving each of the identifiers, the FALL mechanism deletes the original data message corresponding to each of the identifiers from the REDISMSGIDXML queue, and stores the original data message corresponding to each of the identifiers into the REDIS FALLXML queue.

5. The data processing method according to claim 1, characterized in that: After the step of performing object conversion on each acquired original data message, the method further comprises: Check whether the conversion is successful; If the original data message is converted into the message type required by the preset terminal, it is determined that the conversion is successful; If the message content required by the preset terminal is missing in the original data message during the conversion process or there is an internal abnormality in the program during the conversion process, it is determined that the conversion has failed.

6. The data processing method according to claim 1, characterized in that: After the step of sending each successfully converted original data message to a preset terminal, the method further comprises: Taking each target original data message as each original data message obtained, returning to execute the step of object conversion for each original data message obtained, until the target original data message does not exist in each of the original data messages that are successfully converted; The target original data message is the data that failed to be sent and whose number of sending times is less than the preset number of times in each of the original data messages that were successfully converted.

7. A data processing device, characterized in that: The device comprises: A data processing module is configured to perform object conversion on each acquired original data message; The data sending module is configured to send each of the successfully converted original data messages to a preset terminal.

8. An electronic device, characterized in that: include: Memory and processor; The processor is used to read and execute the computer program stored in the memory to implement the steps of the data processing method as described in any one of claims 1-6.

9. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores computer-executable instructions, and when the computer-executable instructions are executed, the steps of the data processing method according to any one of claims 1 to 6 are implemented.

10. A computer program product comprising a computer program / instructions, characterized in that When the computer program / instructions are executed by a processor, the steps of the data processing method as described in any one of claims 1 to 6 are implemented.

Images