Data processing method and processing system
By prioritizing passenger data based on flight information and determining the processing node scope with a consistent hashing algorithm, the problem of data backlog in passenger data processing is solved, efficient concurrent processing is achieved, and the smooth progress of departure business is ensured.
Patent Information
- Application Number
- CN202411954890.X
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2024-12-27
- Publication Date
- 2025-04-04
AI Technical Summary
The prior art can easily lead to a data backlog when processing passenger data, affecting the smooth progress of departure business, and lacking effective solutions.
By dividing passenger data into high-priority and low-priority data based on flight information, and determining the range of nodes that the application can process data in the high-priority and low-priority queues, the data can be processed separately.
It realizes concurrent processing on the basis of ensuring the order of data, and can prioritize emergency data according to the urgency of data, thereby avoiding data backlog and ensuring the smooth progress of departure business.
Smart Images
Figure CN119759537A_ABST
Abstract
Description
Technical Field
[0001] The present invention relates to the technical field of data processing, and in particular to a data processing method and a data processing system. Background Art
[0002] With the rapid development of the civil aviation industry, the increase in passenger volume will inevitably lead to a substantial increase in passenger data. In order to meet the data processing needs brought about by this growth, it is crucial to improve the processing performance of passenger data.
[0003] At present, in order to ensure the orderly status of passengers, it is necessary to parse the messages in a serial manner. As the number of passengers increases, the number of messages also increases. If the messages of an airport are parsed serially, it will cause backlogs and affect departure business.
[0004] To address the above-mentioned problems, no effective solution has been proposed yet. Summary of the invention
[0005] The embodiments of the present invention provide a data processing method and a data processing system, which solves the technical problem that the way of processing passenger data in at least the prior art may cause data backlog and thus affect departure business.
[0006] According to one aspect of an embodiment of the present invention, a data processing method is provided, including: dividing the passenger data into high-priority passenger data and low-priority passenger data according to the flight information of the passenger data, wherein the flight information includes: a three-letter airport code, a flight number and a flight date; determining the node range in which each available application can process the passenger data in combination with a consistent hashing algorithm; determining first data to be processed by the application from the high-priority passenger data according to the node range in which the application can process the passenger data, storing the first data to be processed in a high-priority queue to be parsed and performing data processing; determining second data to be processed by the application from the low-priority passenger data according to the node range in which the application can process the passenger data, storing the second data to be processed in a low-priority queue to be parsed and performing data processing.
[0007] Furthermore, in combination with the consistent hashing algorithm, the node range of each available application that can process passenger data is determined, including: obtaining the available number of applications; according to the consistent hashing algorithm, allocating multiple processing nodes to the application, and mapping each processing node to the hash ring; determining the step size of each available application based on the available number of applications and the number of processing nodes; and allocating a node range to each available application based on the step size.
[0008] Furthermore, obtaining the available number of applications includes: monitoring the heartbeat data of the application in the heartbeat table of the database and the timestamp of the heartbeat data, wherein the heartbeat data includes the IP address of the server where the application is located; determining the current state of the application based on whether the time interval for receiving the heartbeat data exceeds a time threshold; and determining the available number of applications based on the current state of the application.
[0009] Further, based on the flight information of the passenger data, the passenger data is divided into high-priority passenger data and low-priority passenger data, including: determining the grouping mark of each passenger data based on the flight information of the passenger data, wherein the grouping mark includes: an airport three-letter code mark, a flight number mark and a flight date mark; determining the grading mark of each passenger data based on the grouping mark and the flight date, wherein the grading mark includes: a same-day mark and a non-same-day mark; storing the passenger data with the grading mark as the same-day mark in a high-priority passenger table, wherein the passenger data in the high-priority passenger table is high-priority passenger data; storing the passenger data with the grading mark as the non-same-day mark in a low-priority passenger table, wherein the passenger data in the low-priority passenger table is low-priority passenger data.
[0010] Furthermore, the classification mark of each passenger data is determined according to the grouping mark and the flight date, including: before determining the classification mark of the current passenger data, comparing the grouping mark and the classification mark of the current passenger data with the historical passenger data; when the grouping marks of the current passenger data and the historical passenger data are the same but the classification marks are different, the classification mark of the current passenger data is not set.
[0011] Furthermore, based on the node range in which the application can process passenger data, the first data to be processed of the application is determined from the corresponding high-priority passenger data to be processed, and the first data to be processed is stored in the high-priority queue to be parsed of the application and data processing is performed, including: obtaining the first passenger data from the high-priority queue to be parsed; determining whether the grouping tag of the first passenger data is stored in the high-priority cache object; if so, not processing the first passenger data, and continuing to obtain the next passenger data.
[0012] Furthermore, based on the node range in which the application can process passenger data, the first data to be processed of the application is determined from the corresponding high-priority passenger data to be processed, the first data to be processed is stored in the high-priority queue to be parsed of the application and data processing is performed, and it also includes: if the grouping mark of the first passenger data is not stored in the high-priority cache object, the first passenger data is processed; after the processing of the first passenger data is completed, the grouping mark of the first passenger data is removed from the high-priority cache object.
[0013] According to another aspect of an embodiment of the present invention, there is also provided a data processing system, including: a preprocessing module, the preprocessing module is used to divide the passenger data into high-priority passenger data and low-priority passenger data according to the flight information of the passenger data, wherein the flight information includes: the three-letter airport code, the flight number and the flight date; a calculation module, the calculation module is used to determine the node range of each available application that can process the passenger data in combination with the consistent hashing algorithm; a first data processing module, the first data processing module is used to determine the first to-be-processed data of the application from the high-priority passenger data according to the node range of the application that can process the passenger data, store the first to-be-processed data into the high-priority to-be-parsed queue of the application and perform data processing; a second data processing module, the second data processing module is used to determine the second to-be-processed data of the application from the low-priority passenger data according to the node range of the application that can process the passenger data, store the second to-be-processed data into the low-priority to-be-parsed queue of the application and perform data processing.
[0014] According to another aspect of an embodiment of the present invention, a computer storage medium is provided. The computer storage medium includes a stored program, wherein when the program is executed, the device where the computer storage medium is located is controlled to execute the above processing method.
[0015] According to another aspect of an embodiment of the present invention, a computer program product is provided, including a computer program, and when the computer program is executed by a processor, the above-mentioned processing method is implemented.
[0016] In an embodiment of the present invention, passenger data is divided into high-priority passenger data and low-priority passenger data according to flight information, and the node ranges of each available application that can process passenger data are divided according to a consistent hashing algorithm. According to the node ranges, the first data to be processed of the application is determined from the high-priority passenger data, and the second data to be processed of the application is determined from the low-priority passenger data. The first data to be processed is stored in a high-priority queue to be parsed, and the second data to be processed is stored in a low-priority queue to be parsed. In the above scheme, multiple applications are used to synchronously process passenger data of different flight information, and each application processes passenger data of different priorities separately, that is, concurrent processing is achieved on the basis of ensuring the orderliness of passenger data, and emergency data can be processed preferentially according to the urgency of the data, thereby ensuring the smooth progress of departure business. BRIEF DESCRIPTION OF THE DRAWINGS
[0017] The drawings described herein are used to provide a further understanding of the present invention and constitute a part of this application. The exemplary embodiments of the present invention and their descriptions are used to explain the present invention and do not constitute an improper limitation of the present invention. In the drawings:
[0018] Figure 1is a hardware structure block diagram of a computer terminal (or mobile device) for implementing a data processing method according to an embodiment of the present invention;
[0019] Figure 2 is a flowchart of a method for processing data according to an embodiment of the present invention;
[0020] Figure 3 is a structural block diagram of a data processing system according to an embodiment of the present invention. DETAILED DESCRIPTION
[0021] In order to enable those skilled in the art to better understand the scheme of the present invention, the technical scheme in the embodiments of the present invention will be clearly and completely described below in conjunction with the drawings in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work should fall within the scope of protection of the present invention.
[0022] It should be noted that the terms "first", "second", etc. in the specification and claims of the present invention and the above-mentioned drawings are used to distinguish similar objects, and are not necessarily used to describe a specific order or sequence. It should be understood that the data used in this way can be interchanged where appropriate, so that the embodiments of the present invention described herein can be implemented in an order other than those illustrated or described herein. In addition, the terms "including" and "having" and any variations thereof are intended to cover non-exclusive inclusions, for example, a process, method, system, product or device that includes a series of steps or units is not necessarily limited to those steps or units that are clearly listed, but may include other steps or units that are not clearly listed or inherent to these processes, methods, products or devices.
[0023] Example 1
[0024] According to an embodiment of the present invention, an embodiment of a data processing method is provided. It should be noted that the steps shown in the flowchart of the accompanying drawings can be executed in a computer system such as a set of computer executable instructions, and although a logical order is shown in the flowchart, in some cases, the steps shown or described can be executed in an order different from that shown here.
[0025] Figure 1 1 is a hardware structure block diagram of a computer terminal (or mobile device) for implementing a data processing method according to an embodiment of the present invention. Figure 1As shown, the computer terminal (or mobile device) may include one or more processors 102 (the processor may include but is not limited to a central processing unit (CPU), a graphics processing unit (GPU), a digital signal processing (DSP) chip, a microprocessor (MCU), a field programmable logic device (FPGA), a neural network processor (NPU), a tensor processing unit (TPU), an artificial intelligence (AI) type processor, etc.) and a memory 104 for storing data. In addition, a transmission device 106 for communication functions, an input and output device 108, and a display 110 may also be included. It can be understood by those skilled in the art that Figure 1 The structure shown is for illustration only and does not limit the structure of the above-mentioned computer terminal (or mobile device). For example, the computer terminal may also include more or fewer components than the above-mentioned structure description, or have a configuration different from the above-mentioned structure description.
[0026] The memory 104 can be used to store computer programs, for example, software programs and modules of application software, such as the computer program corresponding to the data processing method in the embodiment of the present invention. The processor 102 executes various functional applications and data processing by running the computer program stored in the memory 104, that is, realizes the above-mentioned data processing method. The memory 104 may include a high-speed random access memory, and may also include a non-volatile memory, such as one or more magnetic storage devices, flash memory, or other non-volatile solid-state memory. In some examples, the memory 104 may further include a memory remotely arranged relative to the processor 102, and these remote memories may be connected to the mobile terminal via a network. Examples of the above-mentioned network include, but are not limited to, the Internet, an intranet, a local area network, a mobile communication network, and combinations thereof.
[0027] The transmission device 106 is used to receive or send data via a network. The specific example of the above network may include a wireless network provided by a communication provider of the mobile terminal. In one example, the transmission device 106 includes a network adapter (Network Interface Controller, referred to as NIC), which can be connected to other network devices through a base station so as to communicate with the Internet. In one example, the transmission device 106 can be a radio frequency (RF) module, which is used to communicate with the Internet wirelessly.
[0028] The display 110 may be a touch screen type liquid crystal display (LCD). The liquid crystal display may enable a user to interact with a user interface of the mobile terminal. In some embodiments, the mobile terminal has a graphical user interface (GUI), and a user may interact with the GUI by finger contact and / or gestures on a touch-sensitive surface, wherein the human-computer interaction functions here may optionally include the following interactions: creating web pages, drawing, word processing, making electronic documents, games, video conferencing, instant messaging, sending and receiving emails, call interfaces, playing digital videos, playing digital music and / or web browsing, etc. Executable instructions for performing the above human-computer interaction functions are configured / stored in a computer program product or readable storage medium executable by one or more processors.
[0029] Figure 2 is a flow chart of a method for processing data according to one embodiment of the present invention. Figure 2 As shown, the process includes the following steps:
[0030] Step S1: Divide the passenger data into high-priority passenger data and low-priority passenger data according to the flight information of the passenger data, wherein the flight information includes: three-letter airport code, flight number and flight date.
[0031] It should be noted that the passenger data is stored in the passenger data table, the pre-processing module obtains the passenger data from the passenger data table, and parses the corresponding flight information from the obtained passenger data; the pre-processing module prioritizes the passenger data according to the flight information.
[0032] Step S2: Combined with the consistent hashing algorithm, determine the node range of each available application that can process passenger data.
[0033] It should be noted that the consistent hashing algorithm relies on a hash function to map a key (passenger data) to an integer (hash value), and maps the hash value to a hash ring. The hash ring has multiple processing nodes, each of which corresponds to a piece of passenger data. Each available application can process multiple pieces of passenger data, so it is necessary to determine the node range of the processing nodes of each available application.
[0034] Step S3: According to the node range of the passenger data that the application can process, the first data to be processed of the application is determined from the high-priority passenger data, and the first data to be processed is stored in the high-priority queue to be parsed of the application and processed. Step S4: According to the node range of the passenger data that the application can process, the second data to be processed of the application is determined from the low-priority passenger data, and the second data to be processed is stored in the low-priority queue to be parsed of the application and processed.
[0035] It should be noted that each application has a high-priority queue to be parsed and a low-priority queue to be parsed. Before processing the data, the data to be processed needs to be stored in the corresponding queue to be parsed so that passenger data of different priorities can be processed separately to avoid data of different priorities affecting each other during the data processing process.
[0036] In an embodiment of the present application, passenger data is divided into high-priority passenger data and low-priority passenger data according to flight information, and the node ranges that can process passenger data for each available application are divided according to a consistent hashing algorithm. According to the node ranges, the first data to be processed of the application is determined from the high-priority passenger data, and the second data to be processed of the application is determined from the low-priority passenger data. The first data to be processed is stored in a high-priority queue to be parsed, and the second data to be processed is stored in a low-priority queue to be parsed. In the above scheme, multiple applications are used to synchronously process passenger data of different flight information, and each application processes passenger data of different priorities separately, that is, concurrent processing is achieved on the basis of ensuring the orderliness of passenger data, and urgent data can be processed preferentially according to the urgency of the data, thereby ensuring the smooth progress of departure business.
[0037] In an exemplary embodiment of the present application, in step S2, the consistent hashing algorithm is combined to determine the node range of each available application that can process passenger data, including the following steps:
[0038] Step S21: Obtain the available number of applications.
[0039] Step S22: According to the consistent hashing algorithm, multiple processing nodes are allocated to the application program, and each processing node is mapped to a hash ring.
[0040] For example, the passenger data is of int type, and the data range of the passenger data is (Integer.min_value, Integer.max_value). The hash ring is defined as an integer ring, and both the processing nodes and the passenger data are mapped to the hash ring. The processing nodes and passenger data on the hash ring need to be set accordingly. Therefore, the number of processing nodes is set to N1, N1 = Integer.max_value-Integer.min_value.
[0041] Step S23: Determine the step length of each available application program according to the available number of application programs and the number of processing nodes.
[0042] Exemplarily, the available number of applications is N2, the step length of each application is step, and step = (Integer.max_value-Integer.min_value) / N2.
[0043] Step S24: Allocate a node range to each available application based on the step size.
[0044] Exemplarily, the minimum value of the processing range of each application is defined as floor, the maximum value of the processing range of each application is defined as ceiling, and the serial number of the available application is set to n, n=1, 2, 3, . . . , n.
[0045] floor(1)=Integer.min_value, ceiling(1)=Integer.min_value+step.
[0046] floor(2)=ceiling(1)+1, ceiling(2)=floor(2)+step.
[0047] …
[0048] floor(n-1)=ceiling(n-2)+1, ceiling(n-1)=floor(n-1)+step.
[0049] floor(n)=ceiling(n-1)+1, ceiling(n)=Integer.max_value.
[0050] The hash value of the current passenger data is calculated, and when the hash value of the current passenger data is within the processing range of the application, the current passenger data is processed by the application.
[0051] Furthermore, obtaining the available number of applications in step S21 includes the following steps:
[0052] Step S211: monitoring the heartbeat data of the application program in the heartbeat table of the database and the timestamp of the heartbeat data, wherein the heartbeat data includes the IP address of the server where the application program is located.
[0053] Specifically, a heartbeat detection thread is started to monitor the heartbeat of the application and write it into the database heartbeat table, which records the IP address of the server where the application is located and the timestamp of the heartbeat data. Among them, each application corresponds to a server, that is, each application is installed on a server.
[0054] Step S212: Determine the current state of the application program according to whether the time interval for receiving the heartbeat data exceeds a time threshold.
[0055] Specifically, a time threshold for considering the heartbeat alive is set, and the time interval for receiving the heartbeat data is calculated based on adjacent timestamps of the heartbeat data. When the time interval exceeds the time threshold, the heartbeat is considered abnormal and the application is unavailable.
[0056] Step S213: Determine the available number of applications based on the current status of the application.
[0057] Specifically, the number of IPs whose time interval is less than or equal to the time threshold is queried from the heartbeat table, and the number of IPs is also the available number of applications.
[0058] In an exemplary embodiment of the present application, in step S1, the passenger data is divided into high-priority passenger data and low-priority passenger data according to the flight information of the passenger data, including the following steps:
[0059] Step S11: determining the grouping mark of each passenger data according to the flight information of the passenger data, wherein the grouping mark includes: an airport three-letter code mark, a flight number mark and a flight date mark.
[0060] Step S12: Determine the classification mark of each passenger data according to the group mark and the flight date, wherein the classification mark includes: a same-day mark and a non-same-day mark.
[0061] Step S13: storing the passenger data graded and marked as the day mark into a high priority passenger table, wherein the passenger data in the high priority passenger table is high priority passenger data.
[0062] It should be noted that the passenger data is stored in the passenger data table before the classification process. After the high-priority passenger data is stored in the high-priority passenger table, the transfer status is marked as transferred in the passenger data table. Step S14: Store the passenger data with the classification mark as non-same-day identification in the low-priority passenger table, wherein the passenger data in the low-priority passenger table is low-priority passenger data.
[0063] It should be noted that the passenger data are stored in the passenger data table before the hierarchical processing, and after the low-priority passenger data are stored in the low-priority passenger table, the transfer status is also marked as transferred in the passenger data table.
[0064] Through steps S11 to S14, the passenger data is grouped and marked so that the passenger data of different flights can be processed concurrently; the passenger data is further graded and marked on the basis of the grouping marks so that the passenger data of the day and the passenger data of other days can be processed separately to avoid the passenger data of the day departing that cannot be processed in time.
[0065] Furthermore, in step S12, the classification mark of each passenger data is determined according to the group mark and the flight date, including the following steps:
[0066] Step S121: before determining the classification mark of the current passenger data, compare the grouping mark and classification mark of the current passenger data with that of the historical passenger data.
[0067] It should be noted that the data in the passenger data table with the transfer status of transferred has been marked for grouping and classification, and has been transferred to the high-priority passenger table and the low-priority passenger table. At this time, the non-transferred passenger data in the passenger data table is being graded and confirmed. The historical passenger data refers to the passenger data in the passenger data table that has been graded but has a transfer status of non-transferred.
[0068] Step S122: When the grouping marks of the current passenger data and the historical passenger data are the same but the classification marks are different, the classification mark of the current passenger data is not set.
[0069] It should be noted that the current passenger data and historical passenger data refer to passenger data in the passenger data table whose status is not transferred, that is, the object of the hierarchical preprocessing is the passenger data in the passenger data table whose transfer status is not transferred. It should be noted that each passenger will generate multiple data in the departure business. When setting the priority of passenger data, the priority of multiple data of the same passenger cannot be inconsistent. Through steps S121 to S122, inconsistent priorities of multiple data of the same passenger can be avoided.
[0070] In an exemplary embodiment of the present application, in step S3, according to the node range of the passenger data that can be processed by the application, the first data to be processed by the application is determined from the corresponding high-priority passenger data to be processed, the first data to be processed is stored in the high-priority queue to be parsed of the application, and data processing is performed, including the following steps:
[0071] It should be noted that the grouping marks of the passenger data to be parsed in the high-priority queue to be parsed are different from those in the low-priority queue to be parsed, that is, multiple data of the same passenger cannot exist in both the high-priority queue to be parsed and the low-priority queue to be parsed.
[0072] Step S31: Acquire the first passenger data from the high priority queue to be analyzed.
[0073] Step S32: Determine whether the grouping mark of the first passenger data is stored in a high-priority cache object.
[0074] Step S33: If yes, the first passenger data is not processed, and the next passenger data is acquired, that is, the first passenger data should be processed after the data with the same grouping mark is processed.
[0075] Step S34: If the grouping mark of the first passenger data is not stored in the high-priority cache object, the first passenger data is processed, that is, a new thread is started to start processing the first passenger data.
[0076] Step S35: After the first passenger data is processed, the grouping mark of the first passenger data is removed from the high-priority cache object.
[0077] Among them, a first time threshold for processing passenger data is set. When passenger data in the high-priority cache is not processed within the first time threshold, the grouping mark of the passenger data will be cleared from the high-priority cache. The passenger data will be released to re-enter the high-priority queue to be parsed and the parsing process will be repeated.
[0078] It should be noted that after the passenger data processing is completed, the message status will be updated, that is, the processing will be marked as successful or failed.
[0079] In an exemplary embodiment of the present application, in step S4, according to the node range of the application program that can process the passenger data, the second data to be processed of the application program is determined from the low-priority passenger data, the second data to be processed is stored in the low-priority queue to be parsed of the application program, and data processing is performed, including the following steps:
[0080] It should be noted that the grouping marks of the passenger data to be parsed in the high-priority queue to be parsed are different from those in the low-priority queue to be parsed, that is, multiple data of the same passenger cannot exist in both the high-priority queue to be parsed and the low-priority queue to be parsed.
[0081] Step S41: Acquire the second passenger data from the low priority queue to be analyzed.
[0082] Step S42: Determine whether the grouping mark of the second passenger data is stored in a low-priority cache object.
[0083] Step S43: If yes, the second passenger data is not processed, and the next passenger data is acquired, that is, the second passenger data should be processed after the processing of the data with the same grouping mark is completed.
[0084] Step S44: If the grouping mark of the second passenger data is not stored in the low-priority cache object, the second passenger data is processed, that is, a new thread is started to start processing the second passenger data.
[0085] Step S45: After the second passenger data is processed, the grouping mark of the second passenger data is removed from the low-priority cache object.
[0086] The second time threshold for processing passenger data is set. When the passenger data in the low-priority cache is not processed within the second time threshold, the grouping mark of the passenger data will be cleared from the low-priority cache. The passenger data will be released to re-enter the low-priority queue for parsing and the parsing process will be repeated.
[0087] According to another aspect of the embodiment of the present invention, a data processing system is also provided. Figure 3 As shown, the processing system includes: a preprocessing module, a calculation module, a first data processing module and a second data processing module.
[0088] The pre-processing module is used to divide the passenger data into high-priority passenger data and low-priority passenger data according to the flight information of the passenger data, wherein the flight information includes: three-letter airport code, flight number and flight date.
[0089] The computing module is used to combine the consistent hashing algorithm to determine the node range of each available application that can process passenger data.
[0090] The first data processing module is used to determine the first data to be processed of the application from the high-priority passenger data according to the node range of the application that can process passenger data, store the first data to be processed into the high-priority queue to be parsed of the application and perform data processing.
[0091] The second data processing module is used to determine the second data to be processed of the application from the low-priority passenger data according to the node range that the application can process passenger data, store the second data to be processed into the low-priority queue to be parsed of the application and perform data processing.
[0092] In an embodiment of the present application, passenger data is divided into high-priority passenger data and low-priority passenger data according to flight information, and the node ranges that can process passenger data for each available application are divided according to a consistent hashing algorithm. According to the node ranges, the first data to be processed of the application is determined from the high-priority passenger data, and the second data to be processed of the application is determined from the low-priority passenger data. The first data to be processed is stored in a high-priority queue to be parsed, and the second data to be processed is stored in a low-priority queue to be parsed. In the above scheme, multiple applications are used to synchronously process passenger data of different flight information, and each application processes passenger data of different priorities separately, that is, concurrent processing is achieved on the basis of ensuring the orderliness of passenger data, and urgent data can be processed preferentially according to the urgency of the data, thereby ensuring the smooth progress of departure business.
[0093] According to another embodiment of the present application, a computer storage medium is provided. The computer storage medium includes a stored program, wherein when the program is executed, the device where the computer storage medium is located is controlled to execute the above-mentioned processing method.
[0094] In an embodiment of the present application, the above storage medium may be configured to store a computer program for performing the following steps:
[0095] Step S1: Divide the passenger data into high-priority passenger data and low-priority passenger data according to the flight information of the passenger data, wherein the flight information includes: three-letter airport code, flight number and flight date.
[0096] Step S2: Combined with the consistent hashing algorithm, determine the node range of each available application that can process passenger data.
[0097] Step S3: According to the node range in which the application can process passenger data, determine the first data to be processed by the application from the high-priority passenger data, store the first data to be processed in the high-priority queue to be parsed of the application, and perform data processing.
[0098] Step S4: according to the node range of the passenger data that can be processed by the application, determine the second data to be processed by the application from the low-priority passenger data, store the second data to be processed in the low-priority queue to be parsed of the application and perform data processing.
[0099] In an embodiment of the present application, passenger data is divided into high-priority passenger data and low-priority passenger data according to flight information, and the node ranges that can process passenger data for each available application are divided according to a consistent hashing algorithm. According to the node ranges, the first data to be processed of the application is determined from the high-priority passenger data, and the second data to be processed of the application is determined from the low-priority passenger data. The first data to be processed is stored in a high-priority queue to be parsed, and the second data to be processed is stored in a low-priority queue to be parsed. In the above scheme, multiple applications are used to synchronously process passenger data of different flight information, and each application processes passenger data of different priorities separately, that is, concurrent processing is achieved on the basis of ensuring the orderliness of passenger data, and urgent data can be processed preferentially according to the urgency of the data, thereby ensuring the smooth progress of departure business.
[0100] According to another embodiment of the present application, a computer program product is provided, including a computer program, and the computer program implements the above processing method when executed by a processor.
[0101] In an embodiment of the present application, the processor may be configured to perform the following steps through a computer program:
[0102] Step S1: Divide the passenger data into high-priority passenger data and low-priority passenger data according to the flight information of the passenger data, wherein the flight information includes: three-letter airport code, flight number and flight date.
[0103] Step S2: Combined with the consistent hashing algorithm, determine the node range of each available application that can process passenger data.
[0104] Step S3: According to the node range in which the application can process passenger data, determine the first data to be processed by the application from the high-priority passenger data, store the first data to be processed in the high-priority queue to be parsed of the application, and perform data processing.
[0105] Step S4: according to the node range of the passenger data that can be processed by the application, determine the second data to be processed by the application from the low-priority passenger data, store the second data to be processed in the low-priority queue to be parsed of the application and perform data processing.
[0106] In an embodiment of the present application, passenger data is divided into high-priority passenger data and low-priority passenger data according to flight information, and the node ranges that can process passenger data for each available application are divided according to a consistent hashing algorithm. According to the node ranges, the first data to be processed of the application is determined from the high-priority passenger data, and the second data to be processed of the application is determined from the low-priority passenger data. The first data to be processed is stored in a high-priority queue to be parsed, and the second data to be processed is stored in a low-priority queue to be parsed. In the above scheme, multiple applications are used to synchronously process passenger data of different flight information, and each application processes passenger data of different priorities separately, that is, concurrent processing is achieved on the basis of ensuring the orderliness of passenger data, and urgent data can be processed preferentially according to the urgency of the data, thereby ensuring the smooth progress of departure business.
[0107] The serial numbers of the above embodiments of the present invention are only for description and do not represent the advantages or disadvantages of the embodiments.
[0108] In the above embodiments of the present invention, the description of each embodiment has its own emphasis. For parts that are not described in detail in a certain embodiment, reference can be made to the relevant descriptions of other embodiments.
[0109] In the several embodiments provided in this application, it should be understood that the disclosed technical content can be implemented in other ways. Among them, the device embodiments described above are only schematic. For example, the division of the units can be a logical function division. There may be other division methods in actual implementation. For example, multiple units or components can be combined or integrated into another system, or some features can be ignored or not executed. Another point is that the mutual coupling or direct coupling or communication connection shown or discussed can be through some interfaces, indirect coupling or communication connection of units or modules, which can be electrical or other forms.
[0110] The units described as separate components may or may not be physically separated, and the components shown as units may or may not be physical units, that is, they may be located in one place or distributed on multiple units. Some or all of the units may be selected according to actual needs to achieve the purpose of the present embodiment.
[0111] In addition, each functional unit in each embodiment of the present invention may be integrated into one processing unit, or each unit may exist physically separately, or two or more units may be integrated into one unit. The above-mentioned integrated unit may be implemented in the form of hardware or in the form of software functional units.
[0112] If the integrated unit is implemented in the form of a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present invention, in essence, or the part that contributes to the prior art, or all or part of the technical solution can be embodied in the form of a software product, and the computer software product is stored in a storage medium, including a number of instructions for a computer device (which can be a personal computer, a server or a network device, etc.) to perform all or part of the steps of the method described in each embodiment of the present invention. The aforementioned storage medium includes: U disk, read-only memory (ROM, Read-Only Memory), random access memory (RAM, Random Access Memory), mobile hard disk, magnetic disk or optical disk and other media that can store program codes.
[0113] The above is only a preferred embodiment of the present invention. It should be pointed out that for ordinary technicians in this technical field, several improvements and modifications can be made without departing from the principle of the present invention. These improvements and modifications should also be regarded as the scope of protection of the present invention.
Claims
1. A data processing method, characterized in that: include: Dividing the passenger data into high-priority passenger data and low-priority passenger data according to flight information of the passenger data, wherein the flight information includes: an airport three-letter code, a flight number, and a flight date; In combination with a consistent hashing algorithm, determining a range of nodes that each available application can process the passenger data; According to the node range in which the application can process the passenger data, determining the first data to be processed by the application from the high-priority passenger data, storing the first data to be processed in the high-priority queue to be parsed of the application and performing data processing; According to the node range in which the application can process the passenger data, second data to be processed by the application is determined from the low-priority passenger data, and the second data to be processed is stored in the low-priority queue to be parsed of the application and data processing is performed.
2. The processing method according to claim 1, characterized in that: Combined with the consistent hashing algorithm, determine the node range of each available application that can process the passenger data, including: Obtaining the available number of the application; According to the consistent hashing algorithm, a plurality of processing nodes are allocated to the application, and each of the processing nodes is mapped to a hash ring respectively; Determining the step length of each available application according to the available number of the application and the number of the processing nodes; Based on the step size, the node range is allocated to each available application.
3. The processing method according to claim 2, characterized in that: Get the available number of the application, including: Monitoring the heartbeat data of the application in the heartbeat table of the database and the timestamp of the heartbeat data, wherein the heartbeat data includes the IP address of the server where the application is located; Determining the current state of the application program according to whether the time interval for receiving the heartbeat data exceeds a time threshold; Based on the current status of the application, the available number of the application is determined.
4. The processing method according to claim 1, characterized in that: According to the flight information of the passenger data, the passenger data is divided into high-priority passenger data and low-priority passenger data, including: Determine a grouping mark of each passenger data according to the flight information of the passenger data, wherein the grouping mark includes: an airport three-letter code mark, a flight number mark and a flight date mark; Determine a classification mark of each passenger data according to the grouping mark and the flight date, wherein the classification mark includes: a same-day mark and a non-same-day mark; storing the passenger data with the classification mark as the day mark into a high priority passenger table, wherein the passenger data in the high priority passenger table is the high priority passenger data; The passenger data with the classification mark as non-same-day identification is stored in a low-priority passenger table, wherein the passenger data in the low-priority passenger table is the low-priority passenger data.
5. The processing method according to claim 4, characterized in that: Determining the classification mark of each passenger data according to the grouping mark and the flight date includes: Before determining the classification mark of the current passenger data, comparing the grouping mark and the classification mark of the current passenger data with the historical passenger data; In the case where the grouping marks of the current passenger data and the historical passenger data are the same and the classification marks are different, the classification mark of the current passenger data is not set.
6. The processing method according to claim 4, characterized in that: According to the node range in which the application can process the passenger data, determining first data to be processed by the application from the high-priority passenger data, storing the first data to be processed in a high-priority queue to be parsed of the application and performing data processing, including: Acquire first passenger data from the high priority queue to be parsed; Determining whether the grouping mark of the first passenger data is stored in a high-priority cache object; If yes, the first passenger data is not processed, and the next passenger data is obtained.
7. The processing method according to claim 6, characterized in that: According to the node range in which the application can process the passenger data, determining first data to be processed by the application from the high-priority passenger data, storing the first data to be processed in a high-priority queue to be parsed of the application and performing data processing, further comprising: If the grouping tag of the first passenger data is not stored in the high-priority cache object, processing the first passenger data; After the first passenger data is processed, the grouping mark of the first passenger data is removed from the high priority cache object.
8. A data processing system, characterized in that: include: A preprocessing module, the preprocessing module is used to divide the passenger data into high-priority passenger data and low-priority passenger data according to flight information of the passenger data, wherein the flight information includes: an airport three-letter code, a flight number and a flight date; A calculation module, the calculation module is used to determine the node range of each available application program that can process the passenger data in combination with a consistent hashing algorithm; A first data processing module, the first data processing module is used to determine first data to be processed of the application from the high-priority passenger data according to a node range in which the application can process the passenger data, store the first data to be processed in a high-priority queue to be parsed of the application, and perform data processing; The second data processing module is used to determine the second data to be processed of the application from the low-priority passenger data according to the node range that the application can process the passenger data, store the second data to be processed into the low-priority queue to be parsed of the application and perform data processing.
9. A computer storage medium, characterized in that The computer storage medium includes a stored program, wherein when the program is executed, the device where the computer storage medium is located is controlled to execute the processing method according to any one of claims 1 to 7.
10. A computer program product, comprising a computer program, characterized in that When the computer program is executed by a processor, the processing method according to any one of claims 1 to 7 is implemented.