Data processing method, device and server

Abstract

This specification provides a data processing method, device and server. Based on this method, when the first system and the second system running in parallel are applied to process the data processing request received, the first system and the second system can be used at the same time to process the data processing request received in parallel according to the preset processing rules; and by introducing and using the preset monitoring log, the processing time and processing results of each processing stage of the second system and the first system when processing the data processing request, as well as the target parameters such as the state characteristic parameters of the second system and the state characteristic parameters of the first system can be recorded; and then the operation status of the second system can be monitored according to the preset monitoring log; and when an abnormality is found in the second system, the first system can be switched to be used to process the data processing request alone in a timely manner, so that the operation of the newly constructed second system can be monitored for a long time and effectively, and the received data processing request can be processed stably.

Description

Technical Field

[0001] This specification belongs to the field of Internet technology, and in particular to data processing methods, devices and servers. Background Art

[0002] In many data processing scenarios (for example, transaction data processing scenarios in banks), due to business needs, a new processing system is often built in addition to the old processing system.

[0003] Since the new processing system has just been built, there are often some unstable factors, which can easily lead to errors when using it.

[0004] When using the above system to process data based on existing methods, there are often technical problems such as unstable data processing and inability to conduct long-term and effective monitoring of the new processing system.

[0005] Currently, no effective solution has been proposed for the above technical problems. Summary of the invention

[0006] This specification provides a data processing method, device and server, which can monitor the operation of the newly constructed second system in a long term and effectively, and stably process the received data processing requests.

[0007] This specification provides a data processing method, which is applied to a first system and a second system running in parallel; wherein the second system includes a data processing system independent of the first system that is constructed based on a preset construction rule, and the second system and the first system perform data synchronization according to a preset synchronization rule; the method includes:

[0008] Using the first system and the second system to process the received data processing request in parallel according to a preset processing rule; and recording target parameters through a preset monitoring log; wherein the target parameters include: the processing time and processing results of the second system and the first system at each processing stage of processing the data processing request, as well as the state characteristic parameters of the second system and the state characteristic parameters of the first system;

[0009] Monitoring the operating status of the second system according to the preset monitoring log;

[0010] When it is detected that the operating status of the second system is abnormal, the first system is switched to independently process the received data processing request.

[0011] In one embodiment, the data processing request carries a user's identity.

[0012] In one embodiment, using the first system and the second system to process the received data processing request according to a preset processing rule includes:

[0013] Extracting the user's identity from the data processing request;

[0014] Determine whether the initiator of the data processing request is a target user according to a preset list and the identity of the user; wherein the preset list stores the identity of multiple target users;

[0015] When it is determined that the initiator of the data processing request is a target user, the data processing request is sent to the second system for corresponding processing.

[0016] In one embodiment, the second system includes a second database independent of the first database in the first system, a second storage area independent of the first storage area in the first system, and a second cache independent of the first cache in the first system.

[0017] In one embodiment, the second database is used to store user data of the target user; the second storage area is used to store target index data; the second cache is used to store intermediate parameters whose usage frequency is greater than a preset frequency threshold; wherein the target index data is used to characterize the mapping relationship between the user data of the target user stored in the second database and the corresponding identity identifier of the target user.

[0018] In one embodiment, the second system synchronizes data with the first system according to a preset synchronization rule, including:

[0019] According to the first synchronization rule, at every first preset time interval, the user data of the target user stored in the second database and the first database are synchronized by using the data channel between the second database and the first database.

[0020] In one embodiment, the second system synchronizes data with the first system according to a preset synchronization rule, further comprising:

[0021] According to the second synchronization rule, at every second preset time interval, the intermediate parameters stored in the second cache and the first cache are synchronized by using the user data of the target user stored in the second database.

[0022] In one embodiment, the second system synchronizes data with the first system according to a preset synchronization rule, further comprising:

[0023] According to the third synchronization rule, at every third preset time interval, the target index data stored in the second storage area and the first storage area are synchronized by using Kafka messages.

[0024] In one embodiment, the state characteristic parameters of the second system include at least one of the following: a state parameter associated with the second database, a state parameter associated with the second storage area, and a state parameter associated with the second cache.

[0025] In one embodiment, monitoring the operating status of the second system according to the preset monitoring log includes:

[0026] Calculating the synchronization rate and accuracy of the second system and the first system when processing the data processing request according to the preset monitoring log;

[0027] An operating state of the second system is determined according to the synchronization rate and accuracy.

[0028] In one embodiment, when it is detected that the operating state of the second system is abnormal, after switching to use the first system to independently process the received data processing request, the method further includes:

[0029] According to the preset monitoring log, determining the abnormal reason causing the abnormal operation state of the second system; wherein the abnormal reason includes: communication abnormality between the first system and the second system, and / or internal abnormality of the second system;

[0030] The second system is repaired according to the abnormality cause.

[0031] In one embodiment, the method further comprises:

[0032] According to the preset monitoring log, the construction structure of the second system and / or the operation configuration parameters are adjusted.

[0033] This specification also provides a data processing device, which is applied to a first system and a second system running in parallel; wherein the second system includes a data processing system independent of the first system that is constructed based on a preset construction rule, and the second system and the first system perform data synchronization according to a preset synchronization rule; the device includes:

[0034] A processing module, used to use the first system and the second system to process the received data processing request in parallel according to a preset processing rule; and record target parameters through a preset monitoring log; wherein the target parameters include: the processing time and processing results of the second system and the first system at each processing stage of processing the data processing request, and the state characteristic parameters of the second system and the state characteristic parameters of the first system;

[0035] A monitoring module, used to monitor the operating status of the second system according to the preset monitoring log;

[0036] The switching module is used to switch to using the first system to independently process the received data processing request when it is detected that the operating status of the second system is abnormal.

[0037] The present specification also provides a server, comprising a processor and a memory for storing processor executable instructions, wherein when the processor executes the instructions, the first system and the second system are used to process the received data processing requests in parallel according to preset processing rules; and target parameters are recorded through a preset monitoring log; wherein the target parameters include: the processing time and processing results of the second system and the first system in each processing stage of processing the data processing request, as well as the state characteristic parameters of the second system and the state characteristic parameters of the first system; according to the preset monitoring log, the operating status of the second system is monitored; when it is detected that the operating status of the second system is abnormal, switching to use the first system to independently process the received data processing request.

[0038] The present specification also provides a computer-readable storage medium having computer instructions stored thereon, which, when executed, implement the use of the first system and the second system to process the received data processing requests in parallel according to preset processing rules; and record target parameters through a preset monitoring log; wherein the target parameters include: the processing time and processing results of the second system and the first system in each processing stage of processing the data processing request, as well as the state characteristic parameters of the second system and the state characteristic parameters of the first system; according to the preset monitoring log, monitor the operating status of the second system; and when it is detected that the operating status of the second system is abnormal, switch to using the first system to independently process the received data processing request.

[0039] The present specification provides a data processing method, device and server. When a first system and a second system running in parallel are used to process an incoming data processing request, the first system and the second system can be used to process the incoming data processing request in parallel according to preset processing rules; at the same time, by introducing and utilizing preset monitoring logs to record target parameters, for example, the processing time and processing results of the second system and the first system at each processing stage of processing the above-mentioned data processing request, as well as the state characteristic parameters of the second system and the state characteristic parameters of the first system, etc.; further, according to the preset monitoring log, the operating status of the second system can be monitored in quasi-real time; and when an abnormality is found in the second system, the first system can be switched to be used to process the data processing request alone in a timely manner, so that the operation of the newly constructed second system can be monitored for a long time and effectively, and the incoming data processing request can be processed stably. BRIEF DESCRIPTION OF THE DRAWINGS

[0040] In order to more clearly illustrate the embodiments of this specification, the drawings required for use in the embodiments will be briefly introduced below. The drawings described below are only some embodiments recorded in this specification. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative labor.

[0041] Figure 1 is a flowchart of a data processing method provided by an embodiment of this specification;

[0042] Figure 2 It is a schematic diagram of an embodiment of applying the data processing method provided by the embodiments of this specification in a scenario example;

[0043] Figure 3 It is a schematic diagram of the structure of a server provided by an embodiment of this specification;

[0044] Figure 4 is a schematic diagram of the structure of a data processing device provided by an embodiment of this specification;

[0045] Figure 5 It is a schematic diagram of an embodiment of applying the data processing method provided by the embodiments of this specification in a scenario example. DETAILED DESCRIPTION

[0046] In order to enable those skilled in the art to better understand the technical solutions in this specification, the technical solutions in the embodiments of this specification will be clearly and completely described below in conjunction with the drawings in the embodiments of this specification. Obviously, the described embodiments are only part of the embodiments of this specification, not all of the embodiments. Based on the embodiments in this specification, all other embodiments obtained by ordinary technicians in this field without creative work should fall within the scope of protection of this specification.

[0047] Considering that based on existing methods, when a new system is introduced and applied to process data, it is often difficult to monitor the operating status of the new system in a long-term and effective manner. At the same time, since the new system itself has many unstable factors, problems are prone to occur when using the new system to process data, affecting the stability of data processing.

[0048] In view of the root cause of the above problems, this specification considers that the original first system and the newly introduced second system can be used simultaneously to process the received data processing requests in parallel according to the preset processing rules. At the same time, by introducing and utilizing the preset monitoring log, the processing time and processing results of the second system and the first system in each processing stage of processing the above data processing request, as well as the target parameters such as the state characteristic parameters of the second system and the state characteristic parameters of the first system are recorded; and then the operation status of the second system can be accurately monitored according to the target parameters recorded in the preset monitoring log; and when the operation status of the second system is found to be abnormal, the first system is promptly switched to process the data processing request alone to normally process the received data processing request, so that the operation of the newly constructed second system can be monitored long-term and effectively, and the received data processing request can be stably processed.

[0049] Based on the above ideas, this manual provides a data processing method. Figure 1 As shown. The method can be specifically applied to a first system and a second system in which the method is applied to parallel operation. The second system includes a data processing system independent of the first system that is constructed based on a preset construction rule, and the second system and the first system perform data synchronization according to a preset synchronization rule. When the method is specifically implemented, it may include the following contents.

[0050] S101: Use the first system and the second system to process the received data processing request in parallel according to preset processing rules; and record target parameters through a preset monitoring log; wherein the target parameters include: the processing time and processing results of the second system and the first system in each processing stage of processing the data processing request, as well as the state characteristic parameters of the second system and the state characteristic parameters of the first system.

[0051] S102: Monitoring the operating status of the second system according to the preset monitoring log.

[0052] S103: When it is detected that the operating state of the second system is abnormal, switch to using the first system to independently process the received data processing request.

[0053] In one embodiment, the first system can be specifically understood as an old data processing system (or existing system architecture) constructed based on existing construction rules. According to different application scenarios, the first system can be responsible for different data processing. For example, in the transaction data processing scenario of a bank, the first system can be specifically a data processing system responsible for processing transaction data processing requests from the user side.

[0054] For details, see Figure 2 As shown, the first system may specifically include the following components: a server, an application server, an operating system, a database, an application core module, and an application development platform, etc.

[0055] Specifically, the application server used in the first system may include an X86 physical machine, the database used may include Oracle, the operating system used may include RedHat Linux, the application server used may include VM, VMWare, Intel X86 CPU, and the development platform used may include a microservice framework, WebLogic, Coherence, etc. Of course, the first system listed above is only a schematic illustration. In specific implementation, according to different application scenarios and processing requirements, the first system can also be constructed based on existing construction rules using other software and hardware resources.

[0056] In one embodiment, the second system can be specifically understood as a new data processing system (or target system architecture) that is independent of the first system and meets relevant requirements and is constructed based on preset construction rules. The data processing responsible for the second system can be exactly the same as that of the first system, or it can only be responsible for part of the data processing responsible for the first system. For example, in the transaction data processing scenario of a bank, the second system can also be a data processing system responsible for processing transaction data processing requests from the user side.

[0057] For details, see Figure 2As shown, the components included in the second system are the same as those of the first system, that is, the second system may specifically include: a server, an application server, an operating system, a database, an application core module, and an application development platform, etc. However, according to the preset construction rules, the software and hardware resources used when constructing the second system, as well as the relevant configuration parameters, may be different from those of the first system.

[0058] Specifically, the application server used in the second system may include a Taishan server equipped with a Kunpeng 920 chip, the database used may include a Gauss T database (Gauss DB), the operating system used may include the Kylin system, the application server used may include VM, Fusionsphere (virtualization), KunPeng920 ARM CPU, and the development platform used may be an independently developed component that has abandoned Weblogic, Coherence, etc.

[0059] For example, the development platform used may include a self-developed microservice framework, the message component uses kafka (message), the routing and configuration center uses nginx and zookeeper, the cache can use Redis, the NoSQL (non-relational database) solution uses Cassandra, the streaming computing uses Storm, and the log search component adopts the ELK solution. In addition, the application middleware uses tomcat, and the development language package uses OpenJDK. The application core layer can also be divided into several microservice modules according to business functions. Of course, the second system listed above is only a schematic illustration. In specific implementation, according to different application scenarios and processing requirements, the above-mentioned second system can also be constructed based on other suitable preset construction rules using other software and hardware resources.

[0060] In one embodiment, after the corresponding second system is constructed in the above manner, considering that the second system is a newly constructed system, there are still some uncertain and unstable factors. In order to ensure the stability of processing data processing requests, the first system and the second system can be run simultaneously to process the received data processing requests.

[0061] When the first system and the second system are run in parallel, the first system and the second system can be used to process the received data processing requests in parallel according to the preset processing rules.

[0062] In one embodiment, during specific implementation, a data processing request to be processed initiated by a user through a terminal device may be received.

[0063] In one embodiment, the data processing request may specifically carry a user's identity. The user may be specifically understood as a user object that initiates the data processing request. The identity may be specifically understood as identification information that can correspond one-to-one with a user object. Specifically, the identity may include: the user's name, the user's account name, the user's registered mobile phone number, and the like. Of course, the identity listed above is only a schematic illustration. In specific implementation, the identity may also include other types of identification information according to specific circumstances and processing requirements.

[0064] In one embodiment, the above-mentioned use of the first system and the second system to process the received data processing request according to the preset processing rules may include the following contents during specific implementation.

[0065] S1: extracting the user's identity from the data processing request;

[0066] S2: Determine whether the initiator of the data processing request is a target user according to a preset list and the identity of the user; wherein the preset list stores the identity of multiple target users;

[0067] S3: When it is determined that the initiator of the data processing request is a target user, the data processing request is sent to the second system for corresponding processing.

[0068] In one embodiment, the target users may include users with a high tolerance for errors in data processing. Specifically, the target users may be test users who are willing to participate in data processing tests based on the newly constructed second system, or internal users.

[0069] In one embodiment, the preset list (also referred to as a whitelist, etc.) may specifically store the identity of the target user.

[0070] In one embodiment, after receiving the data processing request, it can be determined whether the initiating user of the data processing request belongs to the target user based on the identity of the user carried in the data processing request according to the preset processing rules. In the case where it is determined that the initiating user of the data processing request belongs to the target user, the data processing request can be sent to the second system, and then the second system can be used to process the data processing request later. In the case where it is determined that the initiating user of the data processing request does not belong to the target user, the data processing request can be sent to the first system, and then the first system can be used to process the data processing request later.

[0071] Specifically, the data processing request can be sent to the corresponding first system or second system for specific processing through global routing according to the identity carried by the received data processing request and the preset list.

[0072] In one embodiment, in order to ensure that the first system and the second system can accurately and efficiently complete specific data processing in parallel in the above manner, data synchronization is also performed on the first system and the second system according to preset synchronization rules, so that the above two systems maintain data level uniformity in data processing and reduce errors.

[0073] In one embodiment, the first system may specifically include: a first database, a first storage area, a data storage structure independent of the first cache, and the like.

[0074] The first database may be used to store user data of all users (including target users and non-target users). The user data may be understood as associated data used when processing a data processing request initiated by a user, such as user account information, user identity verification information, user financial data, etc.

[0075] The first storage area can be specifically used to store index data of all users. The index data can be specifically understood as data for mapping the relationship between user data stored in the first database and the corresponding user identity. In specific implementation, by using the index data, the corresponding user data can be retrieved from the first database according to the identity carried in the data processing request. The index data can specifically include NoSQL-based index data.

[0076] The first buffer area may be used to store intermediate parameters that are used more frequently (for example, the frequency of use is greater than a preset frequency threshold). Specifically, the intermediate parameters may include a portion of user data that is used more frequently. In addition, the intermediate parameters may further include some parameter data that have high requirements for timeliness of use. For example, in a scenario where a payment order is confirmed within a limited time, the intermediate parameters may be identity authentication information of a user that is required to be used within a short period of time.

[0077] In one embodiment, correspondingly, the second system may specifically include: a second database independent of the first database in the first system, a second storage area independent of the first storage area in the first system, and a second cache independent of the first cache in the first system.

[0078] The second database may be used to store user data of the target user. The second storage area may be used to store target index data. The target index data is used to characterize the mapping relationship between the user data of the target user stored in the second database and the corresponding identity of the target user. The second cache may be used to store intermediate parameters whose usage frequency is greater than a preset frequency threshold.

[0079] It should be pointed out that, based on the above structure, the second system can have a heterogeneous data source relatively independent from the first system, so that the second system can independently complete the data processing of the data processing request initiated by the target user.

[0080] In one embodiment, while the first system and the second system are operated in parallel in the above manner to jointly process the received data processing request, data synchronization can be performed on the relevant data of the first system and the second system according to preset synchronization rules corresponding to different data storage structures.

[0081] In one embodiment, the second system and the first system perform data synchronization according to a preset synchronization rule. When specifically implemented, it may include: according to the first synchronization rule, at each first preset time interval, by using the data channel between the second database and the first database, synchronizing the user data of the target user stored in the second database and the first database. The first synchronization rule may be specifically understood as a data synchronization rule for a database.

[0082] The first preset time interval may be 1 minute, 5 minutes, etc. Of course, the first preset time interval listed above is only a schematic illustration. In specific implementation, other durations may be set as the first preset time interval according to specific circumstances and timeliness requirements.

[0083] Specifically, when the second system is constructed according to the preset construction rules, a data (transmission) channel can be constructed between the second database and the first database, and then when the above-mentioned first system and second system are run in parallel, the first database and the second database can exchange the latest updated user data of the target users of both parties through the data channel, so that the user data of the target users stored in the two databases remain synchronized.

[0084] Specifically, for example, on the first system side, the first system can detect whether the user data of the target user in the first database has been updated at a first preset time interval, and when it is determined that an update has occurred, obtain the updated data as the first update data, and determine the update time; then send the first update data carrying the update time to the second system through the data channel.

[0085] After receiving the first update data through the data channel, the second system can trigger detection of whether the user data of the target user in the second database on the second system side has been updated. If it is determined that an update has occurred, the updated data is obtained as the second update data, and the update time is determined. By comparing the update time of the first update data and the update of the second update data, the first update data and the second update data are integrated to obtain the synchronous update data.

[0086] Then, the second system can send the synchronous update data to the first system through the data channel. At the same time, the second system can update the corresponding data in the second database according to the synchronous update data. The first system can update the corresponding data in the first database according to the received synchronous update data. Thus, data synchronization between the first database and the second database can be achieved.

[0087] Of course, the above-listed synchronization methods for databases are only a schematic illustration. In specific implementation, the above-mentioned second database may also store user data of all users. Accordingly, when performing data synchronization for the database, the user data of non-target users stored in the second database may be updated and synchronized based on the user data of non-target users in the first database at every first preset time interval; at the same time, the user data of target users stored in the first database may be updated and synchronized based on the user data of target users in the second database at every first preset time interval. In this way, data synchronization can be achieved between the first database and the second database.

[0088] In one embodiment, the second system and the first system perform data synchronization according to a preset synchronization rule. When specifically implemented, the method may further include: according to the second synchronization rule, at every second preset time interval, by using the user data of the target user stored in the second database, synchronizing the intermediate parameters stored in the second cache and the first cache. The second synchronization rule may be specifically understood as a data synchronization rule for the cache.

[0089] In this embodiment, the second system can be based on the data in the synchronized second database, and update the intermediate parameters in the second cache according to the data in the second database at every second preset time interval. Similarly, the first system can be based on the data in the synchronized first database, and update the intermediate parameters in the first cache according to the data in the first database at every second preset time interval. In this way, the intermediate parameters stored in the first cache can be matched with the first database, and the intermediate parameters stored in the second cache can be matched with the second database, so as to achieve data synchronization of the intermediate parameters stored in the second cache and the first cache.

[0090] In one embodiment, the second system and the first system perform data synchronization according to a preset synchronization rule. When specifically implemented, the following contents may also be included: according to a third synchronization rule, at every third preset time interval, the target index data stored in the second storage area and the first storage area are synchronized by using a kafka message. The third synchronization rule may be specifically understood as a data synchronization rule for the storage area.

[0091] The above-mentioned Kafka is an open source stream processing platform developed by the Apache Software Foundation. It is a high-throughput distributed publish-subscribe messaging system, which is usually used to process all action stream data of consumers on the website.

[0092] In this embodiment, the target index data stored in the second storage area can be updated and synchronized efficiently and accurately by using Kafka messages.

[0093] Through the above method, according to the preset synchronization rules, data synchronization can be performed on multiple storage structures such as databases, caches, storage areas, etc. in the first system and the second system respectively, thereby effectively ensuring the unity of the data held by the two systems, reducing processing errors during the parallel operation, and further processing the received data processing requests more accurately and stably.

[0094] In one embodiment, while the first system and the second system are running in parallel, target parameters can also be recorded by using a preset monitoring log. The recorded target parameters can specifically include the processing time and processing results of each processing stage when the first system and the second system process data processing requests in parallel. Further, the recorded target parameters can also include state characteristic parameters of the first system and state characteristic parameters of the second system. The above-mentioned preset monitoring log can be specifically understood as log data for recording target parameters related to the first system and the second system running in parallel.

[0095] In one embodiment, the state characteristic parameters of the first system may specifically include at least one of the following: a state parameter associated with the first database, a state parameter associated with the first storage area, a state parameter associated with the first cache, and the like.

[0096] Correspondingly, the state characteristic parameters of the second system may specifically include at least one of the following: a state parameter associated with the second database, a state parameter associated with the second storage area, a state parameter associated with the second cache, and the like.

[0097] In addition, the state characteristic parameters of the first system may also include key information related to the NoSQL of the first system, such as the number of related tables, columns, records, related sequences, or the number of related keys.

[0098] Correspondingly, the state characteristic parameters of the second system may also include key information related to the NoSQL of the second system, such as the number of related tables, columns, records, related sequences, or the number of related keys.

[0099] Of course, the state characteristic parameters of the first system and the state characteristic parameters of the second system listed above are only schematic illustrations. In specific implementation, according to specific circumstances and processing requirements, the state characteristic parameters of the first system and the state characteristic parameters of the second system may also include other types of parameter data. This specification does not limit this.

[0100] In one embodiment, the above-mentioned monitoring of the operating status of the second system according to the preset monitoring log may include the following contents when implemented specifically.

[0101] S1: Calculating the synchronization rate and accuracy of the second system and the first system when processing the data processing request according to the preset monitoring log;

[0102] S2: Determine the operating status of the second system according to the synchronization rate and accuracy.

[0103] In this embodiment, during specific implementation, the evaluation parameters such as synchronization rate and accuracy of the first system and the second system when processing data processing requests in parallel can be calculated according to the target parameters recorded in the preset monitoring log. Then, the operation status of the second system can be evaluated according to the synchronization rate and accuracy, and compared with the preset synchronization rate threshold and the preset accuracy threshold, so that the operation status of the second system can be monitored long-term and effectively.

[0104] The evaluation parameters listed above may also include other types of parameter data such as processing efficiency and number of errors for evaluating the operating status of the second system.

[0105] In one embodiment, when it is determined that the calculated synchronization rate is less than a preset synchronization rate threshold and / or the calculated accuracy is less than a preset accuracy threshold based on the synchronization rate and accuracy, it can be determined that an abnormality has occurred in the operation state of the second system. For example, a large-scale synchronization delay has occurred in the second system.

[0106] In one embodiment, when it is determined that the second system is abnormal, an abnormal prompt information can be generated. According to the abnormal prompt information, it can be timely triggered to switch to use the first system to independently process the received data processing request to ensure accurate and stable data processing of the received data processing request. In addition, according to the abnormal prompt information, it can also be timely triggered to repair the second system to eliminate the abnormality in a timely manner.

[0107] In one embodiment, when it is detected that the operating status of the second system is abnormal, after switching to the first system to independently process the received data processing request, the method may also include the following contents when implemented.

[0108] S1: According to the preset monitoring log, determine the abnormal reason that causes the abnormal operation state of the second system. The abnormal reason may specifically include: communication abnormality between the first system and the second system (for example, the data channel between the first database and the second database is interrupted, or the communication network between the first system and the second system is delayed, etc.), and / or, internal abnormality of the second system (for example, data transmission between different components within the second system is interrupted, or the server within the second system is overloaded, etc.).

[0109] S2: Repair the second system according to the abnormal cause.

[0110] In this embodiment, the state characteristic parameters of the first system and the state characteristic parameters of the second system in the same time period can be obtained according to the preset monitoring log; the cause of the abnormality can be determined by comparing the state characteristic parameters of the first system and the state characteristic parameters of the second system in the same time period.

[0111] Through the above method, the abnormal cause causing the abnormal operation state of the second system can be accurately determined according to the preset monitoring log; then, the second system can be repaired in a targeted manner according to the abnormal cause, thereby improving the repair efficiency of the second system.

[0112] In one embodiment, after the second system is repaired according to the cause of the exception, the method may also include the following contents when implemented: obtaining a preset monitoring log based on the repaired second system; determining whether the operating status of the repaired second system meets the requirements based on the preset monitoring log based on the repaired second system; and when determining whether the operating status of the repaired second system meets the requirements, switching between the first system and the second system to process the received data processing requests in parallel according to preset processing rules.

[0113] In one embodiment, when the method is implemented, it may further include the following contents: adjusting the construction structure of the second system and / or the operation configuration parameters according to the preset monitoring log.

[0114] In this embodiment, during the process of running the first system and the second system in parallel, the operating states of the first system and the second system can also be compared according to the target parameters recorded in the preset monitoring log, and then according to the comparison result, the relatively mature and stable first system can be used as a reference to adjust the construction structure of the second system (for example, the software and hardware resources such as the server used) and / or the operating configuration parameters in a targeted manner, so as to debug and optimize the second system, improve the relevant performance of the second system, and obtain a more stable and reliable second system.

[0115] As can be seen from the above, the data processing method provided by the embodiment of this specification, when the first system and the second system running in parallel are used to process the data processing request received, the first system and the second system are used simultaneously according to the preset processing rules to process the data processing request received in parallel; at the same time, by introducing and using the preset monitoring log to record the processing time and processing results of the second system and the first system in each processing stage of the above data processing request, as well as the target parameters such as the state characteristic parameters of the second system and the state characteristic parameters of the first system; and then according to the preset monitoring log, the operation state of the second system can be monitored; and when an abnormality is found in the second system, the first system is switched to be used to process the data processing request alone in a timely manner, so that the operation of the newly constructed second system can be monitored for a long time and effectively, and the data processing request received can be processed stably. In addition, different synchronization rules are introduced for the second database, the second cache area, and the second storage area in the second system respectively to synchronize data with the first data, the first cache area, and the first storage area in the first system, so that the synchronization rate when the first system and the second system are running in parallel can be improved, so that the above dual system can be used to accurately and efficiently process the data processing request received. By using the preset monitoring log, the state characteristic parameters of the second system are analyzed to determine the abnormal reasons that cause the abnormal operation of the second system, so that the second system can be repaired more targetedly and effectively to eliminate the abnormality according to the above abnormal reasons. By comparing the first system and the second system according to the preset monitoring log, the construction structure and / or operation configuration parameters of the second system can be adjusted in a targeted manner, so that the second system can be effectively optimized, the relevant performance of the second system can be improved, and a more reliable and stable second system can be obtained.

[0116] The embodiment of this specification also provides a server, including a processor and a memory for storing processor executable instructions. When the processor is implemented, it can perform the following steps according to the instructions: use the first system and the second system to process the data processing request in parallel according to the preset processing rules; and record the target parameters through the preset monitoring log; wherein the target parameters include: the processing time and processing results of the second system and the first system in each processing stage of processing the data processing request, as well as the state characteristic parameters of the second system and the state characteristic parameters of the first system; monitor the operating status of the second system according to the preset monitoring log; when it is detected that the operating status of the second system is abnormal, switch to use the first system to process the data processing request alone. Wherein, the second system includes a data processing system independent of the first system that is constructed based on the preset construction rules, and the second system and the first system perform data synchronization according to the preset synchronization rules.

[0117] In order to complete the above instructions more accurately, refer to Figure 3 As shown, the embodiment of this specification also provides another specific server, wherein the server includes a network communication port 301, a processor 302 and a memory 303, and the above structures are connected through internal cables so that each structure can perform specific data interaction.

[0118] The network communication port 301 may be specifically used to receive a data processing request.

[0119] The processor 302 can be specifically used to use the first system and the second system to process the received data processing requests in parallel according to preset processing rules; and record target parameters through a preset monitoring log; wherein the target parameters include: the processing time and processing results of the second system and the first system in each processing stage of processing the data processing request, as well as the state characteristic parameters of the second system and the state characteristic parameters of the first system; according to the preset monitoring log, monitor the operating status of the second system; when it is detected that the operating status of the second system is abnormal, switch to using the first system to independently process the received data processing request.

[0120] The memory 303 may be specifically used to store corresponding instruction programs.

[0121] In this embodiment, the network communication port 301 can be a virtual port that is bound to different communication protocols so that different data can be sent or received. For example, the network communication port can be a port responsible for web data communication, a port responsible for FTP data communication, or a port responsible for email data communication. In addition, the network communication port can also be a physical communication interface or communication chip. For example, it can be a wireless mobile network communication chip, such as GSM, CDMA, etc.; it can also be a Wifi chip; it can also be a Bluetooth chip.

[0122] In this embodiment, the processor 302 may be implemented in any appropriate manner. For example, the processor may take the form of a microprocessor or processor and a computer-readable medium storing a computer-readable program code (such as software or firmware) executable by the (micro)processor, a logic gate, a switch, an application specific integrated circuit (ASIC), a programmable logic controller, and an embedded microcontroller, etc. This specification does not limit this.

[0123] In this embodiment, the memory 303 may include multiple levels. In a digital system, anything that can store binary data can be a memory; in an integrated circuit, a circuit with a storage function but no physical form is also called a memory, such as RAM, FIFO, etc.; in a system, a storage device with a physical form is also called a memory, such as a memory stick, TF card, etc.

[0124] The embodiment of this specification also provides a computer storage medium based on the above data processing method, wherein the computer storage medium stores computer program instructions, and when the computer program instructions are executed, the following are implemented: using the first system and the second system to process the data processing request in parallel according to the preset processing rules; and recording the target parameters through the preset monitoring log; wherein the target parameters include: the processing time and processing results of the second system and the first system in each processing stage of processing the data processing request, as well as the state characteristic parameters of the second system and the state characteristic parameters of the first system; monitoring the operating status of the second system according to the preset monitoring log; and switching to use the first system to process the data processing request separately when the operating status of the second system is detected to be abnormal. wherein the second system includes a data processing system independent of the first system constructed based on the preset construction rules, and the second system and the first system perform data synchronization according to the preset synchronization rules.

[0125] In this embodiment, the storage medium includes, but is not limited to, a random access memory (RAM), a read-only memory (ROM), a cache, a hard disk (HDD), or a memory card. The memory may be used to store computer program instructions. The network communication unit may be an interface for network connection communication set in accordance with the standard specified by the communication protocol.

[0126] In this embodiment, the functions and effects specifically implemented by the program instructions stored in the computer storage medium can be explained in comparison with other implementations and will not be described in detail here.

[0127] See also Figure 4 As shown, at the software level, the embodiment of this specification also provides a data processing device, which is applied to the first system and the second system running in parallel; wherein the second system includes a data processing system independent of the first system constructed based on a preset construction rule, and the second system and the first system perform data synchronization according to a preset synchronization rule. The device may specifically include the following structural modules.

[0128] The processing module 401 may be specifically used to use the first system and the second system to process the received data processing request in parallel according to a preset processing rule; and record target parameters through a preset monitoring log; wherein the target parameters include: the processing time and processing results of the second system and the first system at each processing stage of processing the data processing request, and the state characteristic parameters of the second system and the state characteristic parameters of the first system;

[0129] The monitoring module 402 may be specifically configured to monitor the operating status of the second system according to the preset monitoring log;

[0130] The switching module 403 may be specifically configured to switch to using the first system to independently process the received data processing request when it is detected that the operating status of the second system is abnormal.

[0131] In one embodiment, the data processing request may specifically carry an identity identifier of the user.

[0132] In one embodiment, when the processing module 401 is implemented, it can be used to extract the user's identity from the data processing request; determine whether the initiating user of the data processing request belongs to the target user based on a preset name list and the user's identity; wherein the preset name list stores the identities of multiple target users; when it is determined that the initiating user of the data processing request belongs to the target user, send the data processing request to the second system for corresponding processing.

[0133] In one embodiment, the second system may specifically include a second database independent of the first database in the first system, a second storage area independent of the first storage area in the first system, and a second cache independent of the first cache in the first system.

[0134] In one embodiment, the second database can be specifically used to store user data of the target user; the second storage area can be specifically used to store target index data; the second cache can be specifically used to store intermediate parameters whose usage frequency is greater than a preset frequency threshold; wherein the target index data can be specifically used to characterize the mapping relationship between the user data of the target user stored in the second database and the identity identifier of the target user.

[0135] In one embodiment, the device may further include a synchronization module for synchronizing data between the second system and the first system according to a preset synchronization rule. In a specific implementation, the synchronization module may be used to synchronize the user data of the target user stored in the second database and the first database by using a data channel between the second database and the first database at every first preset time interval according to the first synchronization rule.

[0136] In one embodiment, during specific implementation, the synchronization module can also be used to synchronize the intermediate parameters stored in the second cache with the first cache at every second preset time interval according to the second synchronization rule by using the user data of the target user stored in the second database.

[0137] In one embodiment, during specific implementation, the synchronization module may also be used to synchronize the target index data stored in the second storage area with the target index data stored in the first storage area by using Kafka messages at every third preset time interval according to a third synchronization rule.

[0138] In one embodiment, the state characteristic parameters of the second system may specifically include at least one of the following: a state parameter associated with the second database, a state parameter associated with the second storage area, a state parameter associated with the second cache, etc.

[0139] In one embodiment, the monitoring module 402 can be specifically used to calculate the synchronization rate and accuracy of the second system and the first system when processing data processing requests according to the preset monitoring log; and determine the operating status of the second system according to the synchronization rate and accuracy.

[0140] In one embodiment, the device further includes a repair module. In a specific implementation, when the operating state of the second system is detected to be abnormal, the repair module can be used to determine the abnormal cause of the abnormal operating state of the second system according to the preset monitoring log; wherein the abnormal cause includes: communication abnormality between the first system and the second system, and / or internal abnormality of the second system; and the second system is repaired according to the abnormal cause.

[0141] In one embodiment, the device further includes an optimization module, which can be used to adjust the construction structure of the second system and / or the operation configuration parameters according to the preset monitoring log during specific implementation.

[0142] It should be noted that the units, devices or modules described in the above embodiments can be implemented by computer chips or entities, or by products with certain functions. For the convenience of description, the above devices are described separately by functions divided into various modules. Of course, when implementing this specification, the functions of each module can be implemented in the same or more software and / or hardware, or the modules that implement the same function can be implemented by a combination of multiple sub-modules or sub-units. The device embodiments described above are only schematic. For example, the division of the units is only a logical function division. There may be other division methods in actual implementation, such as multiple units or components can be combined or integrated into another system, or some features can be ignored or not executed. Another point is that the mutual coupling or direct coupling or communication connection shown or discussed can be through some interfaces, indirect coupling or communication connection of devices or units, which can be electrical, mechanical or other forms.

[0143] As can be seen from the above, the data processing device provided in the embodiments of this specification can perform long-term and effective monitoring of the operation of the newly constructed second system and stably process the received data processing requests.

[0144] In a specific scenario example, the data processing method provided in this specification can be used to continuously verify the domestic software and hardware heterogeneous platform (corresponding to the second system), and also analyze the system performance differences between the heterogeneous platforms. The specific method steps are as follows.

[0145] For the existing system (corresponding to the first system), a full-scale domestic hardware and software platform (i.e. the second system) can be built. Figure 2As shown in the figure, the server layer uses Taishan servers equipped with Kunpeng 920 chips. The operating system layer uses Zhongbid Kylin operating system. The database layer uses Huawei's Gauss T database. The application server layer replaces the traditional VMWare with Fusionsphere. Application development platform layer. Remove foreign commercial technology components such as Weblogic and Coherence. Application development mainly uses the self-developed microservice framework, the message component uses kafka, the routing and configuration center uses nginx and zookeeper, the cache uses redis, the nosql solution uses Cassandra, the streaming computing uses Storm, and the log search component uses the ELK solution. The application middleware uses tomcat, and the development language package uses OpenJDK. The application core layer is divided into several microservice modules according to business functions. The hardware and technical components used are shown in the figure below.

[0146] After the request (i.e. data processing request) is connected, the diversion strategy can be determined according to the transaction whitelist strategy (the corresponding preset processing rules) and the whitelist information (the corresponding preset list). The requests of customers in the whitelist will be routed to the domestic software and hardware platform for processing, and other customers will maintain the current routing. At the same time, it will cooperate with heterogeneous databases for data synchronization and support customer-free transaction switching. The overall data processing process can be referred to Figure 5 shown.

[0147] Specifically, for NoSQL index data, the domestic ARM system and X86 system are used to use different consumer groups to consume the same Kafka message.

[0148] For cache data, we first rely on the data synchronization mechanism (i.e. the preset synchronization rules) to complete the data synchronization of the database, and then complete the data loading from the database to the cache through automatic tasks.

[0149] In addition, a specific request performance analysis log (i.e., preset monitoring log) is added to record the routing strategy and actual routing machine status of each request, the time consumption of each module, and the transaction success status. The log is sent to es to complete request monitoring. Key information of the database, NoSQL, and cache of the two systems is collected regularly, such as the number of tables, columns, records, sequences, and keys. The accuracy and efficiency of the entire data synchronization mechanism are automatically compared and evaluated.

[0150] If there is a large-scale synchronization delay in the data of the domestic software and hardware platform, the alarm mechanism can be automatically triggered first, and the route can be automatically switched back to the current system; after the data synchronization is restored to normal, the route can be automatically switched again to restore the continuous verification of the domestic system. In serious cases, the automatic route will not be automatically restored after the automatic route is switched back, and the data synchronization needs to be manually resolved before the route is manually restored.

[0151] In this scenario example, it is noted that existing methods often fail to consider cross-platform, data synchronization, real-time monitoring, and automatic routing switching issues. This results in a lack of solution-level support for verification of domestic heterogeneous platforms, and an inability to form a complete solution.

[0152] In this scenario example, the method adopted not only takes into account the data synchronization issues of dual-screen, database, cache, and NOSQL; it also monitors the data and combines it with an automatic routing switching mechanism to build a complete domestic application continuous verification solution to facilitate the continuous verification of domestic software and hardware systems.

[0153] Furthermore, it also integrates the selection of domestic application construction technology, actual transaction routing control, data synchronization mechanism and monitoring system, and combines the automatic routing switching solution to propose a complete set of practical domestic software and hardware solutions; and can reduce the cost of manual intervention during the parallel operation of dual systems while ensuring the safety of the production system.

[0154] In addition, logs are used to record and analyze the operation of the dual systems, providing a basis for further optimizing the platform and system.

[0155] Through the above scenario examples, it is verified that the data processing method provided in this specification can indeed monitor the operation of the newly constructed second system in a long-term and effective manner and stably process the received data processing requests.

[0156] Although the present specification provides method operation steps as described in the embodiments or flow charts, more or less operation steps may be included based on conventional or non-creative means. The order of steps listed in the embodiments is only one way of executing the order of many steps, and does not represent a unique execution order. When the device or client product in practice is executed, it can be executed in sequence or in parallel according to the method shown in the embodiments or the drawings (for example, a parallel processor or a multi-threaded processing environment, or even a distributed data processing environment). The term "include", "include" or any other variant thereof is intended to cover non-exclusive inclusion, so that the process, method, product or device including a series of elements includes not only those elements, but also includes other elements that are not explicitly listed, or also includes elements inherent to such a process, method, product or device. In the absence of more restrictions, it is not excluded that there are other identical or equivalent elements in the process, method, product or device including the elements. The first, second, etc. words are used to represent the name, and do not represent any particular order.

[0157] Those skilled in the art also know that, in addition to implementing the controller in a purely computer-readable program code, the controller can be made to implement the same function in the form of logic gates, switches, application-specific integrated circuits, programmable logic controllers, and embedded microcontrollers by logically programming the method steps. Therefore, such a controller can be considered as a hardware component, and the devices for implementing various functions included therein can also be considered as structures within the hardware component. Or even, the devices for implementing various functions can be considered as both software modules for implementing the method and structures within the hardware component.

[0158] This specification may be described in the general context of computer-executable instructions executed by a computer, such as program modules. Generally, program modules include routines, programs, objects, components, data structures, classes, etc. that perform specific tasks or implement specific abstract data types. This specification may also be practiced in distributed computing environments where tasks are performed by remote processing devices connected through a communication network. In a distributed computing environment, program modules may be located in local and remote computer storage media including storage devices.

[0159] Through the description of the above embodiments, it can be known that those skilled in the art can clearly understand that the present specification can be implemented by means of software plus a necessary general hardware platform. Based on such an understanding, the technical solution of the present specification can essentially be embodied in the form of a software product, which can be stored in a storage medium such as ROM / RAM, a disk, an optical disk, etc., and includes a number of instructions for enabling a computer device (which can be a personal computer, a mobile terminal, a server, or a network device, etc.) to execute the methods described in each embodiment of the present specification or some parts of the embodiments.

[0160] The various embodiments in this specification are described in a progressive manner, and the same or similar parts between the various embodiments can be referred to each other, and each embodiment focuses on the differences from other embodiments. This specification can be used in many general or special computer system environments or configurations. For example: personal computers, server computers, handheld devices or portable devices, tablet devices, multi-processor systems, microprocessor-based systems, set-top boxes, programmable electronic devices, network PCs, minicomputers, mainframe computers, distributed computing environments including any of the above systems or devices, etc.

[0161] Although the present specification is described through embodiments, those skilled in the art will appreciate that there are many modifications and changes to the present specification without departing from the spirit of the present specification, and it is intended that the appended claims include these modifications and changes without departing from the spirit of the present specification.

Claims

1. A data processing method, characterized in that: The method is applied to a first system and a second system running in parallel; wherein the second system includes a data processing system independent of the first system that is constructed based on a preset construction rule, and the second system and the first system perform data synchronization according to a preset synchronization rule; the method includes: Using the first system and the second system to process the received data processing request in parallel according to a preset processing rule; and recording target parameters through a preset monitoring log; wherein the target parameters include: the processing time and processing results of the second system and the first system at each processing stage of processing the data processing request, as well as the state characteristic parameters of the second system and the state characteristic parameters of the first system; Monitoring the operating status of the second system according to the preset monitoring log; When an abnormality is detected in the operation status of the second system, switching to use the first system to independently process the received data processing request; Wherein, the data processing request carries the user's identity identifier; Accordingly, the first system and the second system are used to process the received data processing requests in parallel according to preset processing rules, including: extracting the user's identity from the data processing request; determining whether the initiating user of the data processing request belongs to the target user according to a preset name list and the user's identity; wherein the preset name list stores the identities of multiple target users; when it is determined that the initiating user of the data processing request belongs to the target user, sending the data processing request to the second system through global routing for corresponding processing; the target user includes a user with a high tolerance for errors occurring during the data processing process; the preset name list includes a white list; and, when it is determined that the initiating user of the data processing request does not belong to the target user, sending the data processing request to the first system through global routing for corresponding processing.

2. The method according to claim 1, characterized in that The second system includes a second database independent of the first database in the first system, a second storage area independent of the first storage area in the first system, and a second cache independent of the first cache in the first system.

3. The method according to claim 2, characterized in that The second database is used to store user data of the target user; the second storage area is used to store target index data; the second cache is used to store intermediate parameters whose usage frequency is greater than a preset frequency threshold; wherein the target index data is used to characterize the mapping relationship between the user data of the target user stored in the second database and the corresponding identity identifier of the target user.

4. The method according to claim 3, characterized in that The second system synchronizes data with the first system according to a preset synchronization rule, including: According to the first synchronization rule, at every first preset time interval, the user data of the target user stored in the second database and the first database are synchronized by using the data channel between the second database and the first database.

5. The method according to claim 4, characterized in that The second system performs data synchronization with the first system according to a preset synchronization rule, further comprising: According to the second synchronization rule, at every second preset time interval, the intermediate parameters stored in the second cache and the first cache are synchronized by using the user data of the target user stored in the second database.

6. The method according to claim 4, characterized in that The second system performs data synchronization with the first system according to a preset synchronization rule, further comprising: According to the third synchronization rule, at every third preset time interval, the target index data stored in the second storage area and the first storage area are synchronized by using Kafka messages.

7. The method according to claim 2, characterized in that: The state characteristic parameters of the second system include at least one of the following: a state parameter associated with the second database, a state parameter associated with the second storage area, and a state parameter associated with the second cache.

8. The method according to claim 7, characterized in that Monitoring the operating status of the second system according to the preset monitoring log includes: Calculating the synchronization rate and accuracy of the second system and the first system when processing the data processing request according to the preset monitoring log; An operating state of the second system is determined according to the synchronization rate and accuracy.

9. The method according to claim 8, characterized in that When it is detected that the operating state of the second system is abnormal, after switching to use the first system to independently process the received data processing request, the method further includes: According to the preset monitoring log, determining the abnormal reason causing the abnormal operation state of the second system; wherein the abnormal reason includes: communication abnormality between the first system and the second system, and / or internal abnormality of the second system; The second system is repaired according to the abnormal cause.

10. The method according to claim 1, characterized in that The method further comprises: According to the preset monitoring log, the construction structure of the second system and / or the operation configuration parameters are adjusted.

11. A data processing device, characterized in that: The device is applied to a first system and a second system running in parallel; wherein the second system includes a data processing system independent of the first system constructed based on a preset construction rule, and the second system and the first system perform data synchronization according to a preset synchronization rule; the device includes: A processing module, used to use the first system and the second system to process the received data processing request in parallel according to a preset processing rule; and record target parameters through a preset monitoring log; wherein the target parameters include: the processing time and processing results of the second system and the first system at each processing stage of processing the data processing request, and the state characteristic parameters of the second system and the state characteristic parameters of the first system; A monitoring module, used to monitor the operating status of the second system according to the preset monitoring log; A switching module, configured to switch to using the first system to independently process the received data processing request when an abnormality is detected in the operation status of the second system; Wherein, the data processing request carries the user's identity identifier; Accordingly, the processing module is specifically used to: extract the user's identity from the data processing request; determine whether the initiator of the data processing request belongs to the target user based on a preset list and the user's identity; wherein the preset list stores the identities of multiple target users; when it is determined that the initiator of the data processing request belongs to the target user, send the data processing request to the second system through global routing for corresponding processing; the target user includes a user with a high tolerance for errors occurring during data processing; the preset list includes a whitelist; and, when it is determined that the initiator of the data processing request does not belong to the target user, send the data processing request to the first system through global routing for corresponding processing.

12. A server, characterized in that: The invention comprises a processor and a memory for storing instructions executable by the processor, wherein the steps of the method according to any one of claims 1 to 10 are implemented when the processor executes the instructions.

13. A computer-readable storage medium, characterized in that: Computer instructions are stored thereon, and when the instructions are executed, the steps of the method according to any one of claims 1 to 10 are implemented.

Images