Data processing method and device, storage medium and electronic equipment

Abstract

The invention discloses a data processing method and device, a storage medium and electronic equipment. Relates to the technical field of data processing and other related technical fields, and the method comprises the following steps: when a data processing request is received through a historical data processing system, analyzing the data processing request through the historical data processing system to obtain initial request information; intercepting and packaging the initial request information through an intercepting and packaging component to obtain target request information; and under the condition that the target request information meets a preset condition, forwarding the target request information to the target data processing system so as to process the data processing request through the target data processing system to obtain a target processing result. Through the data processing method and device, the problem that the upgraded data processing system cannot process the data processing request of the user in the historical data processing system after the data processing system is upgraded in the prior art is solved.

Description

Technical Field

[0001] The present application relates to the field of data processing technology and other related technical fields, and in particular, to a data processing method and device, a storage medium and an electronic device. Background Art

[0002] As competition in the aviation market intensifies, the traditional ticket sales model can no longer meet customer needs. Therefore, airlines have begun to improve user satisfaction by adding additional services. In order to make passengers more aware of these additional services, airlines need to publish clear fare information and efficiently manage a large amount of fare data. However, the traditional underlying data design and storage method fails to consider more security and scalability from the data life cycle. It is mainly reflected in the incomplete storage of some data, which brings risks and hidden dangers to sales and subsequent refund and change services; the lack of data in extreme scenarios causes airlines to bear claims risks; some tables have redundant designs, resulting in high data update costs; data entry and verification methods are inconsistent, and there are multiple sets of verification implementations for the same business, which not only has a poor user experience, but also leads to a long development cycle. Therefore, it is necessary to upgrade the historical data processing system, but during the upgrade process, it is necessary to achieve seamless transfer and processing of user needs.

[0003] With regard to the problem in the related art that after a data processing system is upgraded, the upgraded data processing system cannot process data processing requests of users in the historical data processing system, no effective solution has been proposed yet. Summary of the invention

[0004] The main purpose of the present application is to provide a data processing method and device, a storage medium and an electronic device to solve the problem in the related art that after the data processing system is upgraded, the upgraded data processing system cannot process the data processing requests of users in the historical data processing system.

[0005] In order to achieve the above-mentioned purpose, according to one aspect of the present application, a data processing method is provided. The method comprises: when a data processing request is received through a historical data processing system, the data processing request is parsed through the historical data processing system to obtain initial request information; the initial request information is intercepted and encapsulated through an interception and encapsulation component to obtain target request information; when the target request information meets a preset condition, the target request information is forwarded to a target data processing system, so that the data processing request is processed through the target data processing system to obtain a target processing result.

[0006] Furthermore, the initial request information is intercepted and encapsulated by an interception and encapsulation component to obtain target request information, including: intercepting the initial request information by the interception and encapsulation component, and performing data conversion on the initial request information according to a target data structure to obtain converted initial request information; calculating the delayed processing time of the converted initial request information according to an estimated processing time and business priority corresponding to the data processing request to obtain delayed processing time information; and encapsulating the delayed processing time information and the converted initial request information by the interception and encapsulation component to obtain the target request information.

[0007] Furthermore, after intercepting and encapsulating the initial request information through the interception and encapsulation component to obtain the target request information, the method also includes: determining the position information of the target request information in the delay queue based on the delayed processing time information; based on the position information, storing the target request information in the delay queue.

[0008] Furthermore, after storing the target request information in a delay queue based on the location information, the method also includes: judging through a listener in the delay queue whether the delayed processing time information of the target request information is greater than or equal to the current time, and obtaining a first judgment result; if the first judgment result indicates that the delayed processing time information of the target request information is greater than or equal to the current time, then determining that the target request information meets the preset condition; if the first judgment result indicates that the delayed processing time information of the target request information is less than the current time, then determining that the target request information does not meet the preset condition.

[0009] Furthermore, after determining that the target request information meets the preset conditions, the method also includes: storing the target request information in the delay queue in the target execution queue; determining whether the waiting time information of the target request information in the target execution queue is greater than or equal to a preset threshold, and obtaining a second judgment result; if the second judgment result indicates that the waiting time information of the target request information in the target execution queue is greater than or equal to the preset threshold, then forwarding the target request information to the target data processing system through the request transfer processing component, so that the data processing request is processed through the target data processing system to obtain the target processing result.

[0010] Furthermore, after determining whether the waiting time information of the target request information in the target execution queue is greater than or equal to a preset threshold and obtaining a second judgment result, the method also includes: if the second judgment result indicates that the waiting time information of the target request information in the target execution queue is less than the preset threshold, then determining the execution order of the target request information in the target execution queue based on the waiting time information; forwarding the target request information to the target data processing system based on the execution order through the request transfer processing component, so that the data processing request is processed through the target data processing system to obtain the target processing result.

[0011] Furthermore, the data processing request is processed by the target data processing system to obtain a target processing result, including: processing the data processing request by the target data processing system to obtain a processing log file and an initial processing result; detecting whether there is abnormal information in the processing log file to obtain a detection result; if the detection result indicates that the abnormal information does not exist in the processing log file, then determining the initial processing result as the target processing result.

[0012] In order to achieve the above-mentioned purpose, according to another aspect of the present application, a data processing device is provided. The device includes: a parsing unit, which is used to parse the data processing request through the historical data processing system when receiving the data processing request through the historical data processing system to obtain the initial request information; an interception and encapsulation unit, which is used to intercept and encapsulate the initial request information through the interception and encapsulation component to obtain the target request information; a first forwarding unit, which is used to forward the target request information to the target data processing system when the target request information meets the preset conditions, so that the data processing request is processed by the target data processing system to obtain the target processing result.

[0013] Furthermore, the interception and encapsulation unit includes: a conversion module, which is used to intercept the initial request information through the interception and encapsulation component, and perform data conversion on the initial request information according to the target data structure to obtain the converted initial request information; a calculation module, which is used to calculate the delayed processing time of the converted initial request information according to the estimated processing time and business priority corresponding to the data processing request to obtain the delayed processing time information; and an encapsulation module, which is used to encapsulate the delayed processing time information and the converted initial request information through the interception and encapsulation component to obtain the target request information.

[0014] Furthermore, the device also includes: a first determination unit, which is used to intercept and encapsulate the initial request information through the interception and encapsulation component to obtain the target request information, and then determine the position information of the target request information in the delay queue according to the delayed processing time information; a first storage unit, which is used to store the target request information in the delay queue according to the position information.

[0015] Furthermore, the device also includes: a first judgment unit, which is used to determine whether the delayed processing time information of the target request information is greater than or equal to the current time through a listener in the delay queue after storing the target request information in the delay queue based on the location information, to obtain a first judgment result; a second determination unit, which is used to determine that the target request information meets the preset condition if the first judgment result indicates that the delayed processing time information of the target request information is greater than or equal to the current time; and a third determination unit, which is used to determine that the target request information does not meet the preset condition if the first judgment result indicates that the delayed processing time information of the target request information is less than the current time.

[0016] Furthermore, the device also includes: a second storage unit, used to store the target request information in the delay queue into the target execution queue after determining that the target request information meets the preset conditions; a second judgment unit, used to judge whether the waiting time information of the target request information in the target execution queue is greater than or equal to a preset threshold, and obtain a second judgment result; a second forwarding unit, used to forward the target request information to the target data processing system through a request transfer processing component if the second judgment result indicates that the waiting time information of the target request information in the target execution queue is greater than or equal to the preset threshold, so as to process the data processing request through the target data processing system and obtain the target processing result.

[0017] Furthermore, the device also includes: a fourth determination unit, which is used to determine whether the waiting time information of the target request information in the target execution queue is greater than or equal to a preset threshold, and after obtaining a second judgment result, if the second judgment result indicates that the waiting time information of the target request information in the target execution queue is less than the preset threshold, then determine the execution order of the target request information in the target execution queue based on the waiting time information; and a third forwarding unit, which is used to forward the target request information to the target data processing system based on the execution order through the request transfer processing component, so that the data processing request is processed through the target data processing system to obtain the target processing result.

[0018] Furthermore, the first forwarding unit includes: a processing module, which is used to process the data processing request through the target data processing system to obtain a processing log file and an initial processing result; a detection module, which is used to detect whether there is abnormal information in the processing log file to obtain a detection result; and a determination module, which is used to determine the initial processing result as the target processing result if the detection result indicates that the abnormal information does not exist in the processing log file.

[0019] In order to achieve the above-mentioned purpose, according to one aspect of the present application, a computer-readable storage medium is provided, wherein the storage medium stores a program, wherein when the program is running, the device where the storage medium is located is controlled to execute any one of the data processing methods described above.

[0020] In order to achieve the above-mentioned purpose, according to another aspect of the present application, an electronic device is further provided, the electronic device includes one or more processors and a memory, and the memory is used to store the data processing method implemented by one or more processors as described above.

[0021] Through this application, the following steps are adopted: when a data processing request is received through a historical data processing system, the data processing request is parsed through the historical data processing system to obtain initial request information; the initial request information is intercepted and encapsulated through an interception and encapsulation component to obtain target request information; when the target request information meets the preset conditions, the target request information is forwarded to the target data processing system, so that the data processing request is processed by the target data processing system to obtain the target processing result. Through this application, the problem of the related technology that after the data processing system is upgraded, the upgraded data processing system cannot process the data processing request of the user in the historical data processing system. In this scheme, when the historical data processing system receives a data processing request, it is intercepted by an interception and encapsulation component, and the initial request information obtained by parsing the data processing request is encapsulated and processed to obtain the target request information. After the target request information meets the preset conditions, the target request information is forwarded to the target data processing system for processing to obtain the target processing result. By intercepting, encapsulating and forwarding the initial request information obtained from the analysis of the data processing request, the data processing request can be replicated and smoothly transferred from the historical data processing system to the target data processing system, thereby improving the load processing capacity, fault tolerance and system stability of the target data processing system. BRIEF DESCRIPTION OF THE DRAWINGS

[0022] The drawings constituting a part of the present application are used to provide a further understanding of the present application. The illustrative embodiments and descriptions of the present application are used to explain the present application and do not constitute an improper limitation on the present application. In the drawings:

[0023] Figure 1 is a flow chart of a data processing method provided according to an embodiment of the present application;

[0024] Figure 2 is a schematic diagram of a delay queue structure component provided according to an embodiment of the present application;

[0025] Figure 3 is a schematic diagram of a data structure of target request information provided according to an embodiment of the present application;

[0026] Figure 4 is a schematic diagram of message state transition provided according to an embodiment of the present application;

[0027] Figure 5 is a schematic diagram of a data processing method provided according to an embodiment of the present application;

[0028] Figure 6 is a schematic diagram of a data processing device provided according to an embodiment of the present application;

[0029] Figure 7 is a schematic diagram of an electronic device provided according to an embodiment of the present application. DETAILED DESCRIPTION

[0030] It should be noted that, in the absence of conflict, the embodiments and features in the embodiments of the present application can be combined with each other. The present application will be described in detail below with reference to the accompanying drawings and in combination with the embodiments.

[0031] In order to enable those skilled in the art to better understand the solution of the present application, the technical solution in the embodiments of the present application will be clearly and completely described below in conjunction with the drawings in the embodiments of the present application. Obviously, the described embodiments are only part of the embodiments of the present application, not all of the embodiments. Based on the embodiments in the present application, all other embodiments obtained by ordinary technicians in this field without creative work should fall within the scope of protection of the present application.

[0032] It should be noted that the terms "first", "second", etc. in the specification and claims of the present application and the above-mentioned drawings are used to distinguish similar objects, and are not necessarily used to describe a specific order or sequential order. It should be understood that the data used in this way can be interchanged where appropriate, so that the embodiments of the present application described here. In addition, the terms "including" and "having" and any of their variations are intended to cover non-exclusive inclusions, for example, a process, method, system, product or device comprising a series of steps or units is not necessarily limited to those steps or units clearly listed, but may include other steps or units that are not clearly listed or inherent to these processes, methods, products or devices.

[0033] It should be noted that the relevant information (including but not limited to user device information, user personal information, etc.) and data (including but not limited to data for display, data for analysis, etc.) involved in this disclosure are all information and data authorized by the user or fully authorized by all parties. For example, an interface is set between this system and the relevant user or organization. Before obtaining relevant information, it is necessary to send an acquisition request to the aforementioned user or organization through the interface, and obtain relevant information after receiving the consent information fed back by the aforementioned user or organization.

[0034] The present invention is described below in conjunction with preferred implementation steps. Figure 1 is a flow chart of a data processing method provided according to an embodiment of the present application, such as Figure 1 As shown, the method comprises the following steps:

[0035] Step S101: when a data processing request is received through a historical data processing system, the historical data processing system parses the data processing request to obtain initial request information.

[0036] Optionally, when the original additional service fare publishing system (ie, historical data processing system) receives a data processing request, it parses it to obtain initial request information, which includes but is not limited to request content, request status, request context and other information.

[0037] Step S102, intercepting and encapsulating the initial request information through the interception and encapsulation component to obtain the target request information.

[0038] Optionally, after the initial request information is intercepted by the interception and encapsulation component, a delay time is set for it, and the delay time and the initial request information are encapsulated to obtain the target request information.

[0039] It should be noted that the delay time can be set according to the specific needs of the data processing request. For example, it can be dynamically adjusted according to the expected processing time, business priority, etc.

[0040] Step S103, when the target request information meets the preset conditions, the target request information is forwarded to the target data processing system, so that the data processing request is processed by the target data processing system to obtain the target processing result.

[0041] Optionally, after the target request information meets preset conditions, it is forwarded to the additional service fare publishing system (i.e., the target data processing system), and the additional service fare publishing system responds to the data processing request in the original additional service fare publishing system to obtain the target processing result.

[0042] In summary, when the historical data processing system receives a data processing request, it is intercepted by the interception and encapsulation component, and the initial request information obtained by parsing the data processing request is encapsulated to obtain the target request information. After the target request information meets the preset conditions, the target request information is forwarded to the target data processing system for processing to obtain the target processing result. By intercepting, encapsulating and forwarding the initial request information obtained by parsing the data processing request, the data processing request is replicated and smoothly transferred from the historical data processing system to the target data processing system, thereby improving the load processing capacity, fault tolerance and system stability of the target data processing system.

[0043] Optionally, in the data processing method provided in the embodiment of the present application, the initial request information is intercepted and encapsulated by an interception and encapsulation component to obtain the target request information, including: intercepting the initial request information by the interception and encapsulation component, and performing data conversion on the initial request information according to the target data structure to obtain the converted initial request information; calculating the delayed processing time of the converted initial request information according to the estimated processing time and business priority corresponding to the data processing request to obtain the delayed processing time information; encapsulating the delayed processing time information and the converted initial request information by the interception and encapsulation component to obtain the target request information.

[0044] In an optional embodiment, an interception and encapsulation component is used to make a section in the original additional service fare publishing system, intercepting the initial request information (including but not limited to: request content, request status and request context information) obtained by parsing all data processing requests. According to the Job meta-information type data structure (i.e., the target data structure), the delay queue structure component (including the Job data structure) is designed, and the initial request information is converted according to the target data structure to obtain the converted initial request information. The delay time (i.e., delayed processing time information) of the converted initial request information is dynamically adjusted according to the expected processing time and business priority information corresponding to the data processing request. Finally, the delay time and the converted initial request information are encapsulated into a Job (i.e., the target request information) by the interception and encapsulation component, and stored in the Job Pool of the delay queue structure component. The Job Pool is used to store the meta-information of all Jobs, and its structure is in the form of Key / Value, with the Key being the Job Id and the Value being the Job struct.

[0045] It should be noted that Figure 2 Schematic diagram of a delay queue structure component provided according to an embodiment of the present application. Figure 2As shown, the delay queue structure component is a message delay queue data structure used to maintain continuity during the request transfer process. This component stores the data processing requests to be transferred and sorts them according to the specified delay time. The delay queue structure components include but are not limited to: Job Pool, Delay Bucket and ReadyQueue. Among them, the structure of Delay Bucket is an ordered queue, and the data structure of bucket is the zset structure of the Redis protocol; the structure of Ready Queue is an array or queue.

[0046] It should be noted that Figure 3 is a schematic diagram of the data structure of the target request information provided according to an embodiment of the present application, such as Figure 3 As shown in the figure, the structure of each target request information (i.e., Job struct) includes multiple attributes, namely: Topic (the type of job, which can be understood as a specific business name); Job Id (the unique identifier of the job, used to retrieve and delete the specified job information); Delay (the time the job needs to be delayed, in seconds, which the server will convert into absolute time in order to ensure the reliability of message transmission); Time-To-Run (also expressed as TTR, the job execution timeout, in seconds); Body (the content of the job, for consumers to perform specific business processing, stored in json format).

[0047] By intercepting and encapsulating the component, the data processing request of the original additional service fare publishing system can be quickly intercepted, and the delayed processing time information can be encapsulated with it, thereby accurately obtaining the target request information.

[0048] Optionally, in the data processing method provided in an embodiment of the present application, after the initial request information is intercepted and encapsulated by the interception and encapsulation component to obtain the target request information, the method also includes: determining the position information of the target request information in the delay queue based on the delayed processing time information; based on the position information, storing the target request information in the delay queue.

[0049] In an optional embodiment, the position information of the target request information in the delay queue is determined by delaying the processing time information, and the target request information is stored in the delay queue based on the position information using the Redission framework.

[0050] It should be noted that the Delay Bucket is a set of ordered queues based on time, which is used to store all jobs that need to be delayed or are in the reserve state (only the Job Id is stored here).

[0051] It should be noted that the Redission framework is an open source distributed object storage and caching framework that implements the Redis protocol and provides multiple functions and API interfaces such as distributed collections, distributed locks, distributed queues, distributed publish and subscribe, and distributed current limiting, which simplifies the complexity of distributed development and improves development efficiency.

[0052] The delayed queue request transfer is implemented through the Redission distributed object storage framework, ensuring the high availability, reliability and practicality of message transmission.

[0053] Optionally, in the data processing method provided in the embodiment of the present application, after storing the target request information in a delay queue based on the location information, the method also includes: judging through a listener in the delay queue whether the delayed processing time information of the target request information is greater than or equal to the current time, and obtaining a first judgment result; if the first judgment result indicates that the delayed processing time information of the target request information is greater than or equal to the current time, it is determined that the target request information meets the preset conditions; if the first judgment result indicates that the delayed processing time information of the target request information is less than the current time, it is determined that the target request information does not meet the preset conditions.

[0054] In an optional embodiment, a listener (Timer) is set in the delay queue to scan and monitor the target request information in the delay queue in real time, and determine whether the delay time of the target request information is greater than or equal to the current system time: if the delayed processing time information of the target request information is greater than or equal to the current system time, it indicates that the target request time information meets the preset conditions, and the target request information is transferred to the target processing node; if the delayed processing time information of the target request information is less than the current system time, it indicates that the target request information does not meet the preset conditions, and monitoring continues through the Timer.

[0055] Through the listener, it is possible to quickly and accurately determine whether the target request information meets the preset conditions, and then improve the processing efficiency of the target data processing system based on the judgment results.

[0056] Optionally, in the data processing method provided in the embodiment of the present application, after determining that the target request information meets the preset conditions, the method also includes: storing the target request information in the delay queue in the target execution queue; determining whether the waiting time information of the target request information in the target execution queue is greater than or equal to a preset threshold, and obtaining a second judgment result; if the second judgment result indicates that the waiting time information of the target request information in the target execution queue is greater than or equal to the preset threshold, then forwarding the target request information to the target data processing system through the request transfer processing component, so that the data processing request is processed through the target data processing system to obtain the target processing result.

[0057] In an optional embodiment, if the delayed processing time information of the target request information is greater than or equal to the current system time, the target request information in the delayed queue is placed in the corresponding target execution queue (Ready Queue), and it is determined whether the waiting time of the target request information in the target execution queue is greater than or equal to the threshold time (i.e., the preset threshold): if the waiting time of the target request information in the target execution queue is greater than or equal to the threshold time, it indicates that the processing priority of the target request information is higher, and the request information is forwarded to the target data processing system through the request transfer processing component.

[0058] It should be noted that the Ready Queue stores jobs in the Ready state and only stores the Job ID for consumption by consumer programs. When the timeout of the Ready Queue is greater than the set threshold, the target request information is put back into the Delay Bucket for queuing.

[0059] It should be noted that Figure 4 Schematic diagram of message state conversion according to an embodiment of the present application. Figure 4 As shown, there are four states for a job: ready (executable state, waiting for consumption); delay (unexecutable state, waiting for clock cycles); reserved (read by the consumer, but no response from the consumer, i.e. delete, finish); deleted (consumed or deleted).

[0060] The priority level corresponding to the request information can be accurately determined by the waiting time of the target request information in the target execution queue, thereby enabling the target data processing system to quickly process the request information.

[0061] Optionally, in the data processing method provided in the embodiment of the present application, after determining whether the waiting time information of the target request information in the target execution queue is greater than or equal to a preset threshold and obtaining a second judgment result, the method also includes: if the second judgment result indicates that the waiting time information of the target request information in the target execution queue is less than the preset threshold, then determining the execution order of the target request information in the target execution queue based on the waiting time information; forwarding the target request information to the target data processing system based on the execution order through the request transfer processing component, so that the data processing request is processed through the target data processing system to obtain the target processing result.

[0062] In an optional embodiment, if the waiting time of the target request information in the target execution queue is less than the threshold time, it indicates that the processing priority of the target request information is low, and the execution order of the target request information in the target execution queue can be recalculated according to the waiting time. The request transfer processing component takes out the task in the target execution queue according to the execution order and forwards it to the target data processing system.

[0063] It should be noted that the target request information includes an interval array, and the request transfer processing component can determine the specific time interval information and the number of forwarding times according to the array. For example: the interval array is Array{0, 0, 2 minutes, 10 minutes, 10 minutes, 1 hour, 2 hours, 6 hours, 15 hours}, and the number of forwarding times is N. When the request transfer processing component takes out the task from the target execution queue, it determines that the corresponding time interval is 0 according to N, so it is forwarded immediately. If the first forwarding fails, N is 2 at this time, and the interval is determined to be 2 minutes from the Array, so a task with an interval of 2m is added to the transfer queue, and the task content still contains Array and N. If the second forwarding fails, N is 3 at this time, and the interval is determined to be 10 minutes from the Array, so a task with an interval of 10 minutes is added to the transfer queue. Similarly, if the seventh forwarding fails, N is 8 at this time, and the interval is determined to be 15 hours from the Array, so a task with an interval of 15 hours is added to the transfer queue. Similarly, if the 8th forwarding fails, N is 9 at this time, and the interval time cannot be determined in the Array, and the forwarding ends.

[0064] The execution order of the target request information in the target execution queue can be determined quickly and accurately through the waiting time of the target request information.

[0065] Optionally, in the data processing method provided in the embodiment of the present application, the data processing request is processed by the target data processing system to obtain the target processing result, including: processing the data processing request by the target data processing system to obtain a processing log file and an initial processing result; detecting whether there is abnormal information in the processing log file to obtain a detection result; if the detection result indicates that there is no abnormal information in the processing log file, determining the initial processing result as the target processing result.

[0066] In an optional embodiment, the target data processing system uses built-in algorithms, models or tools to process the received data processing requests, and the processing process includes but is not limited to: data cleaning, conversion, analysis or machine learning operations. During the data processing process, the system needs to record all key steps and decision points for subsequent review or debugging to generate a processing log file, and the processing log file includes but is not limited to: processing time, error information, warnings or important status changes. After completing the processing operation of the data processing request, the system will generate an initial result. According to the processing log file, it is determined whether there is any abnormal information in the initial processing result. If no abnormal information is found in the processing log file, the system determines the initial processing result as the target processing result; if there is abnormal information in the processing log file, the target data processing system reprocesses the data processing request or notifies the user to check.

[0067] It should be noted that the target processing result processing can also be used to compare the processing time and accuracy of the processing results of the target data processing system and the historical data processing system for the data processing request.

[0068] By processing log files, it is possible to accurately and quickly determine whether there are any abnormalities in the processing process of the target data system, thereby improving the processing efficiency and accuracy of the target data processing system for data processing requests.

[0069] It should be noted that Figure 5 Schematic diagram of a data processing method according to an embodiment of the present application. Figure 5 As shown, the data processing method realizes the automatic transfer and distribution of data processing requests through multiple components, including but not limited to: Delay queue structure component: used to maintain the continuity of the connection during the request transfer process. This component stores the requests to be transferred and sorts them according to the specified delay time; interception and encapsulation component: when the original additional service fare publishing system receives the request, it intercepts it, and encapsulates the request according to the required information and writes it into the Redis protocol. At the same time, set the delay time for the intercepted request. The delay time can be set according to specific needs, for example: dynamically adjusted according to processing time, business priority, etc.; listener: responsible for monitoring the requests in the delay queue, and transferring the requests to the target execution queue after the delay time is reached; request transfer processing component: when the listener or background task detects that a request in the delay queue has reached the delay time, the request is taken out of the delay queue and placed in the Ready Queue.

[0070] The data processing method provided in the embodiment of the present application, when receiving a data processing request through a historical data processing system, parses the data processing request through the historical data processing system to obtain initial request information; intercepts and encapsulates the initial request information through an interception and encapsulation component to obtain target request information; when the target request information meets the preset conditions, the target request information is forwarded to the target data processing system, so that the data processing request is processed by the target data processing system to obtain the target processing result, which solves the problem of the related technology that after the data processing system is upgraded, the upgraded data processing system cannot process the data processing request of the user in the historical data processing system. In summary, in this scheme, when the historical data processing system receives a data processing request, it is intercepted by an interception and encapsulation component, and the initial request information obtained by parsing the data processing request is encapsulated and processed to obtain the target request information. After the target request information meets the preset conditions, the target request information is forwarded to the target data processing system for processing to obtain the target processing result. By intercepting, encapsulating and forwarding the initial request information obtained from the analysis of the data processing request, the data processing request can be replicated and smoothly transferred from the historical data processing system to the target data processing system, thereby improving the load processing capacity, fault tolerance and system stability of the target data processing system.

[0071] It should be noted that the steps shown in the flowcharts of the accompanying drawings can be executed in a computer system such as a set of computer executable instructions, and that, although a logical order is shown in the flowcharts, in some cases, the steps shown or described can be executed in an order different from that shown here.

[0072] The embodiment of the present application also provides a data processing device. It should be noted that the data processing device of the embodiment of the present application can be used to execute the data processing method provided by the embodiment of the present application. The data processing device provided by the embodiment of the present application is introduced below.

[0073] Figure 6 is a schematic diagram of a data processing device according to an embodiment of the present application. Figure 6 As shown, the device includes: a parsing unit 601, an interception and encapsulation unit 602, and a first forwarding unit 603.

[0074] The parsing unit 601 is used to parse the data processing request through the historical data processing system to obtain initial request information when the data processing request is received through the historical data processing system;

[0075] The interception and encapsulation unit 602 is used to intercept and encapsulate the initial request information through the interception and encapsulation component to obtain the target request information;

[0076] The first forwarding unit 603 is used to forward the target request information to the target data processing system when the target request information meets the preset conditions, so that the target data processing system processes the data processing request to obtain the target processing result.

[0077] The data processing device provided by the embodiment of the present application, when the parsing unit 601 receives the data processing request through the historical data processing system, the data processing request is parsed by the historical data processing system to obtain the initial request information; the interception and encapsulation unit 602 intercepts and encapsulates the initial request information through the interception and encapsulation component to obtain the target request information; the first forwarding unit 603 forwards the target request information to the target data processing system when the target request information meets the preset conditions, so as to process the data processing request through the target data processing system to obtain the target processing result, which solves the problem in the related technology that after the data processing system is upgraded, the upgraded data processing system cannot process the data processing request of the user in the historical data processing system. In this scheme, when the historical data processing system receives the data processing request, it is intercepted by the interception and encapsulation component, and the initial request information obtained by parsing the data processing request is encapsulated and processed to obtain the target request information. After the target request information meets the preset conditions, the target request information is forwarded to the target data processing system for processing to obtain the target processing result. By intercepting, encapsulating and forwarding the initial request information obtained from the analysis of the data processing request, the data processing request can be replicated and smoothly transferred from the historical data processing system to the target data processing system, thereby improving the load processing capacity, fault tolerance and system stability of the target data processing system.

[0078] Optionally, in the data processing device provided in the embodiment of the present application, the interception and encapsulation unit includes: a conversion module, which is used to intercept the initial request information through the interception and encapsulation component, and perform data conversion on the initial request information according to the target data structure to obtain the converted initial request information; a calculation module, which is used to calculate the delayed processing time of the converted initial request information according to the estimated processing time and business priority corresponding to the data processing request, and obtain the delayed processing time information; an encapsulation module, which is used to encapsulate the delayed processing time information and the converted initial request information through the interception and encapsulation component to obtain the target request information.

[0079] Optionally, in the data processing device provided in the embodiment of the present application, the device also includes: a first determination unit, used to intercept and encapsulate the initial request information through the interception and encapsulation component to obtain the target request information, and then determine the position information of the target request information in the delay queue according to the delayed processing time information; a first storage unit, used to store the target request information in the delay queue based on the position information.

[0080] Optionally, in the data processing device provided in the embodiment of the present application, the device also includes: a first judgment unit, used to, after storing the target request information in the delay queue based on the location information, determine through the listener in the delay queue whether the delayed processing time information of the target request information is greater than or equal to the current time, to obtain a first judgment result; a second determination unit, used to determine that the target request information meets the preset condition if the first judgment result indicates that the delayed processing time information of the target request information is greater than or equal to the current time; and a third determination unit, used to determine that the target request information does not meet the preset condition if the first judgment result indicates that the delayed processing time information of the target request information is less than the current time.

[0081] Optionally, in the data processing device provided in the embodiment of the present application, the device also includes: a second storage unit, used to store the target request information in the delay queue in the target execution queue after determining that the target request information meets the preset conditions; a second judgment unit, used to judge whether the waiting time information of the target request information in the target execution queue is greater than or equal to a preset threshold, and obtain a second judgment result; a second forwarding unit, used to forward the target request information to the target data processing system through the request transfer processing component if the second judgment result indicates that the waiting time information of the target request information in the target execution queue is greater than or equal to the preset threshold, so as to process the data processing request through the target data processing system to obtain a target processing result.

[0082] Optionally, in the data processing device provided in the embodiment of the present application, the device also includes: a fourth determination unit, for determining whether the waiting time information of the target request information in the target execution queue is greater than or equal to a preset threshold, and after obtaining a second judgment result, if the second judgment result indicates that the waiting time information of the target request information in the target execution queue is less than the preset threshold, then determining the execution order of the target request information in the target execution queue based on the waiting time information; and a third forwarding unit, for forwarding the target request information to the target data processing system based on the execution order through the request transfer processing component, so that the data processing request is processed through the target data processing system to obtain a target processing result.

[0083] Optionally, in the data processing device provided in the embodiment of the present application, the first forwarding unit includes: a processing module, which is used to process the data processing request through the target data processing system to obtain a processing log file and an initial processing result; a detection module, which is used to detect whether there is abnormal information in the processing log file to obtain a detection result; and a determination module, which is used to determine the initial processing result as the target processing result if the detection result indicates that there is no abnormal information in the processing log file.

[0084] The data processing device includes a processor and a memory. The above-mentioned parsing unit 601, interception encapsulation unit 602, first forwarding unit 603, etc. are all stored in the memory as program units, and the processor executes the above-mentioned program units stored in the memory to realize efficient processing of data.

[0085] The processor contains a kernel, which calls the corresponding program unit from the memory. One or more kernels can be set, and efficient data processing can be achieved by adjusting kernel parameters.

[0086] The memory may include non-permanent memory in a computer-readable medium, random access memory (RAM) and / or non-volatile memory, such as read-only memory (ROM) or flash RAM, and the memory includes at least one memory chip.

[0087] An embodiment of the present invention provides a computer-readable storage medium on which a program is stored. When the program is executed by a processor, a data processing method is implemented.

[0088] An embodiment of the present invention provides a processor, which is used to run a program, wherein the program executes a data processing method when it is run.

[0089] like Figure 7 As shown, an embodiment of the present invention provides an electronic device, the device includes a processor, a memory and a program stored in the memory and executable on the processor, and the processor implements the following steps when executing the program: when a data processing request is received through a historical data processing system, the data processing request is parsed through the historical data processing system to obtain initial request information; the initial request information is intercepted and encapsulated through an interception and encapsulation component to obtain target request information; when the target request information meets preset conditions, the target request information is forwarded to a target data processing system so that the data processing request is processed through the target data processing system to obtain a target processing result.

[0090] Optionally, the initial request information is intercepted and encapsulated by an interception and encapsulation component to obtain the target request information, including: intercepting the initial request information by the interception and encapsulation component, and performing data conversion on the initial request information according to the target data structure to obtain the converted initial request information; calculating the delayed processing time of the converted initial request information according to the estimated processing time and business priority corresponding to the data processing request to obtain the delayed processing time information; encapsulating the delayed processing time information and the converted initial request information by the interception and encapsulation component to obtain the target request information.

[0091] Optionally, after intercepting and encapsulating the initial request information through the interception and encapsulation component to obtain the target request information, the method also includes: determining the position information of the target request information in the delay queue based on the delay processing time information; based on the position information, storing the target request information in the delay queue.

[0092] Optionally, after storing the target request information in a delay queue based on the location information, the method further includes: determining, through a listener in the delay queue, whether the delayed processing time information of the target request information is greater than or equal to the current time, to obtain a first judgment result; if the first judgment result indicates that the delayed processing time information of the target request information is greater than or equal to the current time, then determining that the target request information meets the preset conditions; if the first judgment result indicates that the delayed processing time information of the target request information is less than the current time, then determining that the target request information does not meet the preset conditions.

[0093] Optionally, after determining that the target request information meets the preset conditions, the method also includes: storing the target request information in the delay queue in the target execution queue; determining whether the waiting time information of the target request information in the target execution queue is greater than or equal to a preset threshold, and obtaining a second judgment result; if the second judgment result indicates that the waiting time information of the target request information in the target execution queue is greater than or equal to the preset threshold, forwarding the target request information to the target data processing system through the request transfer processing component, so as to process the data processing request through the target data processing system and obtain the target processing result.

[0094] Optionally, after determining whether the waiting time information of the target request information in the target execution queue is greater than or equal to a preset threshold and obtaining a second judgment result, the method also includes: if the second judgment result indicates that the waiting time information of the target request information in the target execution queue is less than the preset threshold, then determining the execution order of the target request information in the target execution queue based on the waiting time information; forwarding the target request information to the target data processing system based on the execution order through the request transfer processing component, so that the data processing request is processed through the target data processing system to obtain the target processing result.

[0095] Optionally, processing the data processing request through the target data processing system to obtain the target processing result includes: processing the data processing request through the target data processing system to obtain a processing log file and an initial processing result; detecting whether there is abnormal information in the processing log file to obtain a detection result; if the detection result indicates that there is no abnormal information in the processing log file, determining the initial processing result as the target processing result.

[0096] The devices in this article can be servers, PCs, PADs, mobile phones, etc.

[0097] The present application also provides a computer program product, which, when executed on a data processing device, is suitable for executing an initialization program having the following method steps: when a data processing request is received through a historical data processing system, the data processing request is parsed through the historical data processing system to obtain initial request information; the initial request information is intercepted and encapsulated through an interception and encapsulation component to obtain target request information; when the target request information meets preset conditions, the target request information is forwarded to a target data processing system so that the data processing request is processed through the target data processing system to obtain a target processing result.

[0098] Optionally, the initial request information is intercepted and encapsulated by an interception and encapsulation component to obtain the target request information, including: intercepting the initial request information by the interception and encapsulation component, and performing data conversion on the initial request information according to the target data structure to obtain the converted initial request information; calculating the delayed processing time of the converted initial request information according to the estimated processing time and business priority corresponding to the data processing request to obtain the delayed processing time information; encapsulating the delayed processing time information and the converted initial request information by the interception and encapsulation component to obtain the target request information.

[0099] Optionally, after intercepting and encapsulating the initial request information through the interception and encapsulation component to obtain the target request information, the method also includes: determining the position information of the target request information in the delay queue based on the delay processing time information; based on the position information, storing the target request information in the delay queue.

[0100] Optionally, after storing the target request information in a delay queue based on the location information, the method further includes: determining, through a listener in the delay queue, whether the delayed processing time information of the target request information is greater than or equal to the current time, to obtain a first judgment result; if the first judgment result indicates that the delayed processing time information of the target request information is greater than or equal to the current time, then determining that the target request information meets the preset conditions; if the first judgment result indicates that the delayed processing time information of the target request information is less than the current time, then determining that the target request information does not meet the preset conditions.

[0101] Optionally, after determining that the target request information meets the preset conditions, the method also includes: storing the target request information in the delay queue in the target execution queue; determining whether the waiting time information of the target request information in the target execution queue is greater than or equal to a preset threshold, and obtaining a second judgment result; if the second judgment result indicates that the waiting time information of the target request information in the target execution queue is greater than or equal to the preset threshold, forwarding the target request information to the target data processing system through the request transfer processing component, so as to process the data processing request through the target data processing system and obtain the target processing result.

[0102] Optionally, after determining whether the waiting time information of the target request information in the target execution queue is greater than or equal to a preset threshold and obtaining a second judgment result, the method also includes: if the second judgment result indicates that the waiting time information of the target request information in the target execution queue is less than the preset threshold, then determining the execution order of the target request information in the target execution queue based on the waiting time information; forwarding the target request information to the target data processing system based on the execution order through the request transfer processing component, so that the data processing request is processed through the target data processing system to obtain the target processing result.

[0103] Optionally, processing the data processing request through the target data processing system to obtain the target processing result includes: processing the data processing request through the target data processing system to obtain a processing log file and an initial processing result; detecting whether there is abnormal information in the processing log file to obtain a detection result; if the detection result indicates that there is no abnormal information in the processing log file, determining the initial processing result as the target processing result.

[0104] Those skilled in the art will appreciate that the embodiments of the present application may be provided as methods, systems, or computer program products. Therefore, the present application may adopt the form of a complete hardware embodiment, a complete software embodiment, or an embodiment in combination with software and hardware. Moreover, the present application may adopt the form of a computer program product implemented in one or more computer-usable storage media (including but not limited to disk storage, CD-ROM, optical storage, etc.) that include computer-usable program code.

[0105] The present application is described with reference to the flowcharts and / or block diagrams of the methods, devices (systems), and computer program products according to the embodiments of the present application. It should be understood that each process and / or box in the flowchart and / or block diagram, as well as the combination of the processes and / or boxes in the flowchart and / or block diagram, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, a special-purpose computer, an embedded processor, or other programmable data processing device to generate a machine, so that the instructions executed by the processor of the computer or other programmable data processing device generate instructions for implementing the processes in the flowchart and / or block diagram. Figure 1 A process or multiple processes and / or boxes Figure 1 A device that provides the functions specified in a block or multiple blocks.

[0106] These computer program instructions may also be stored in a computer-readable memory capable of directing a computer or other programmable data processing device to operate in a specific manner, so that the instructions stored in the computer-readable memory produce an article of manufacture comprising an instruction device, which implements the process Figure 1 A process or multiple processes and / or boxes Figure 1A function specified in one or more boxes.

[0107] These computer program instructions can also be loaded onto a computer or other programmable data processing device so that a series of operating steps are executed on the computer or other programmable device to produce a computer-implemented process, thereby providing instructions for implementing the process. Figure 1 A process or multiple processes and / or boxes Figure 1 The steps for the functions specified in one or more boxes.

[0108] In a typical configuration, a computing device includes one or more processors (CPU), input / output interfaces, network interfaces, and memory.

[0109] The memory may include non-permanent memory in a computer-readable medium, random access memory (RAM) and / or non-volatile memory in the form of read-only memory (ROM) or flash RAM. The memory is an example of a computer-readable medium.

[0110] Computer readable media include permanent and non-permanent, removable and non-removable media that can be implemented by any method or technology to store information. Information can be computer readable instructions, data structures, program modules or other data. Examples of computer storage media include, but are not limited to, phase change memory (PRAM), static random access memory (SRAM), dynamic random access memory (DRAM), other types of random access memory (RAM), read-only memory (ROM), electrically erasable programmable read-only memory (EEPROM), flash memory or other memory technology, compact disk read-only memory (CD-ROM), digital versatile disk (DVD) or other optical storage, magnetic cassettes, magnetic tape disk storage or other magnetic storage devices or any other non-transmission media that can be used to store information that can be accessed by a computing device. As defined herein, computer readable media does not include temporary computer readable media (transitory media), such as modulated data signals and carrier waves.

[0111] It should also be noted that the terms "include", "comprises" or any other variations thereof are intended to cover non-exclusive inclusion, so that a process, method, commodity or device including a series of elements includes not only those elements, but also other elements not explicitly listed, or also includes elements inherent to such process, method, commodity or device. In the absence of more restrictions, the elements defined by the sentence "comprises a ..." do not exclude the existence of other identical elements in the process, method, commodity or device including the elements.

[0112] Those skilled in the art will appreciate that the embodiments of the present application may be provided as methods, systems or computer program products. Therefore, the present application may adopt the form of a complete hardware embodiment, a complete software embodiment or an embodiment in combination with software and hardware. Moreover, the present application may adopt the form of a computer program product implemented on one or more computer-usable storage media (including but not limited to disk storage, CD-ROM, optical storage, etc.) that contain computer-usable program code.

[0113] The above are only embodiments of the present application and are not intended to limit the present application. For those skilled in the art, the present application may have various changes and variations. Any modification, equivalent replacement, improvement, etc. made within the spirit and principle of the present application should be included within the scope of the claims of the present application.

Claims

1. A data processing method, characterized in that: include: When a data processing request is received through the historical data processing system, the data processing request is parsed through the historical data processing system to obtain initial request information; The initial request information is intercepted and encapsulated by an interception and encapsulation component to obtain target request information; When the target request information meets a preset condition, the target request information is forwarded to a target data processing system, so that the target data processing system processes the data processing request to obtain a target processing result.

2. The method according to claim 1, characterized in that The initial request information is intercepted and encapsulated by the interception and encapsulation component, and the target request information obtained includes: The initial request information is intercepted by the interception encapsulation component, and data conversion is performed on the initial request information according to the target data structure to obtain the converted initial request information; Calculating the delayed processing time of the converted initial request information according to the estimated processing time and the service priority corresponding to the data processing request to obtain delayed processing time information; The delayed processing time information and the converted initial request information are encapsulated by the interception encapsulation component to obtain the target request information.

3. The method according to claim 1, characterized in that: After the initial request information is intercepted and encapsulated by the interception and encapsulation component to obtain the target request information, the method further includes: Determining the position information of the target request information in the delay queue according to the delay processing time information; According to the location information, the target request information is stored in a delay queue.

4. The method according to claim 3, characterized in that After storing the target request information in a delay queue according to the location information, the method further includes: Determine, by the listener in the delay queue, whether the delayed processing time information of the target request information is greater than or equal to the current time, and obtain a first determination result; If the first judgment result indicates that the delayed processing time information of the target request information is greater than or equal to the current time, it is determined that the target request information meets the preset condition; If the first judgment result indicates that the delayed processing time information of the target request information is less than the current time, it is determined that the target request information does not meet the preset condition.

5. The method according to claim 4, characterized in that After determining that the target request information meets the preset condition, the method further includes: Storing the target request information in the delay queue into the target execution queue; Determine whether the waiting time information of the target request information in the target execution queue is greater than or equal to a preset threshold, and obtain a second determination result; If the second judgment result indicates that the waiting time information of the target request information in the target execution queue is greater than or equal to the preset threshold, the target request information is forwarded to the target data processing system through the request transfer processing component, so that the data processing request is processed by the target data processing system to obtain the target processing result.

6. The method according to claim 5, characterized in that After determining whether the waiting time information of the target request information in the target execution queue is greater than or equal to a preset threshold and obtaining a second determination result, the method further includes: If the second judgment result indicates that the waiting time information of the target request information in the target execution queue is less than the preset threshold, determining the execution order of the target request information in the target execution queue according to the waiting time information; The target request information is forwarded to the target data processing system based on the execution order by the request transfer processing component, so that the data processing request is processed by the target data processing system to obtain the target processing result.

7. The method according to claim 1, characterized in that Processing the data processing request by the target data processing system to obtain a target processing result includes: Processing the data processing request through the target data processing system to obtain a processing log file and an initial processing result; Detecting whether there is abnormal information in the processing log file and obtaining a detection result; If the detection result indicates that the abnormal information does not exist in the processing log file, the initial processing result is determined as the target processing result.

8. A data processing device, characterized in that: include: A parsing unit, configured to parse the data processing request through the historical data processing system to obtain initial request information when the data processing request is received through the historical data processing system; An interception and encapsulation unit, configured to intercept and encapsulate the initial request information through an interception and encapsulation component to obtain target request information; The first forwarding unit is used to forward the target request information to a target data processing system when the target request information meets a preset condition, so that the target data processing system processes the data processing request to obtain a target processing result.

9. A computer-readable storage medium, characterized in that: The computer-readable storage medium includes a stored program, wherein when the program is executed, the storage medium is controlled to execute the data processing method according to any one of claims 1 to 7 on a device.

10. An electronic device, characterized in that: It includes one or more processors and a memory, wherein the memory is used to store one or more programs, wherein when the one or more programs are executed by the one or more processors, the one or more processors implement the data processing method described in any one of claims 1 to 7.

Images