Data processing method and device and electronic equipment

Abstract

The invention provides a data processing method and device and electronic equipment, and the method comprises the steps: obtaining a data processing parameter and a target scene which are used for processing data; determining whether the target scene is a first determination result of a first scene; when the first determination result is that the target scene is the first scene, determining a first link library and a second link library corresponding to the target scene; determining parameter similarity indexes between the data processing parameters and the plurality of reference processing parameters; determining whether there is a similarity determination result in which at least one parameter similarity index is greater than a parameter similarity threshold in the plurality of parameter similarity indexes; and determining a data processing result corresponding to the data processing parameter according to the data processing parameter, the first link library, the second link library and the similarity determination result. According to the data processing method and apparatus, the technical problem of poor data processing effect caused by unstable data processing during data processing in related technologies is solved.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of big data, in particular to a data processing method and device and electronic equipment. BACKGROUND

[0002] In the related art, data carries a large amount of value and multiple application potential, but different scenarios have different requirements for the precision and integrity of the required data. To convert data into usable information, data needs to be processed. However, in the related art, when data is processed, the data processing is unstable, resulting in poor data processing effect.

[0003] To address the above problems, no effective solutions have been proposed so far. SUMMARY

[0004] The main purpose of the present application is to provide a data processing method, device and electronic equipment to at least solve the technical problem of unstable data processing and poor data processing effect in the related art.

[0005] In order to achieve the above purpose, according to one aspect of the present application, a data processing method is provided, comprising: obtaining data processing parameters and a target scene for processing data; determining a first determination result of whether the target scene is a first scene; in the case that the first determination result is that the target scene is a first scene, determining a first link library and a second link library corresponding to the target scene, wherein the second link library is constructed according to the first link library; determining a parameter similarity index between the data processing parameters and a plurality of reference processing parameters; determining whether there is at least one parameter similarity index greater than a parameter similarity threshold in a plurality of parameter similarity indexes, wherein the plurality of parameter similarity indexes correspond one-to-one to the plurality of reference processing parameters; determining a data processing result corresponding to the data processing parameters according to the data processing parameters, the first link library, the second link library, and the similar determination result.

[0006] Optionally, the determining the data processing result corresponding to the data processing parameter according to the data processing parameter, the first link library, the second link library, and the similarity determination result comprises: in a case where the similarity determination result is that there is at least one parameter similarity index greater than a parameter similarity threshold in the plurality of parameter similarity indexes, determining a to-be-compared parameter from the plurality of reference processing parameters, wherein the to-be-compared parameter is a reference processing parameter with a parameter similarity index greater than a parameter similarity threshold in the plurality of reference processing parameters; determining a second determination result of whether the target scene is a to-be-compared scene, wherein the to-be-compared scene is a scene corresponding to the to-be-compared parameter; and determining the data processing result corresponding to the data processing parameter according to the second determination result, the data processing parameter, the first link library, the second link library, and the similarity determination result.

[0007] Optionally, the determining the data processing result corresponding to the data processing parameter according to the second determination result, the data processing parameter, the first link library, the second link library, and the similarity determination result comprises: in a case where the second determination result is that the target scene is not a to-be-compared scene, adjusting a data processing link corresponding to the to-be-compared scene in a third link library to the first link library and the second link library to obtain a first newly-added link corresponding to the first link library and a second newly-added link corresponding to the second link library, wherein the third link library is a link library corresponding to a second scene, the first scene is a scene with a stable demand index greater than a stable demand threshold, the second scene is a scene with a stable demand index less than or equal to the stable demand threshold, and the stable demand index represents a degree of stability required when data is processed; and processing the data processing parameter by using the first newly-added link corresponding to the first link library to obtain the data processing result corresponding to the data processing parameter.

[0008] Optionally, the processing the data processing parameter by using the first newly-added link corresponding to the first link library to obtain the data processing result corresponding to the data processing parameter comprises: determining a link stability index corresponding to the first newly-added link; and in a case where the link stability index is less than a link stability threshold, stopping processing the data processing parameter by using the first newly-added link corresponding to the first link library, and processing the data processing parameter by using the second newly-added link corresponding to the second link library to obtain the data processing result corresponding to the data processing parameter.

[0009] Optionally, the determining the data processing result corresponding to the data processing parameter according to the data processing parameter, the first link library, the second link library, and the similarity determination result comprises: in a case where the similarity determination result is that there is no at least one parameter similarity index greater than a parameter similarity threshold in the plurality of parameter similarity indexes, determining a data association link corresponding to the data processing parameter; determining a processing mode link corresponding to the data association link and to-be-processed data, wherein the processing mode link comprises a plurality of data processing modes; adding the processing mode link to the first link library and the second link library respectively; processing the to-be-processed data according to the processing mode link in the first link library and the processing mode link in the second link library to obtain the data processing result corresponding to the data processing parameter.

[0010] Optionally, the processing the to-be-processed data according to the processing mode link in the first link library and the processing mode link in the second link library to obtain the data processing result corresponding to the data processing parameter comprises: determining a target processing link corresponding to the to-be-processed data, wherein the target processing link comprises any one of the processing mode link in the first link library and the processing mode link in the second link library; processing the to-be-processed data according to a first processing mode according to an execution order of a plurality of data processing modes in the target processing link to obtain first processing data, and processing the first processing data according to a next processing mode to obtain next processing data, until the plurality of data processing modes are executed to obtain target processing data corresponding to the to-be-processed data; and determining the data processing result corresponding to the data processing parameter according to the target processing data.

[0011] Optionally, the determining the processing mode link corresponding to the data association link comprises: determining a plurality of data association relationships corresponding to the data association link respectively, wherein the plurality of data association relationships respectively comprise a plurality of sub-data, original data corresponding to the plurality of sub-data respectively, and data processing modes corresponding to the plurality of original data respectively, the corresponding data processing modes are used for processing the corresponding original data to obtain the corresponding sub-data, and the plurality of original data and the plurality of sub-data are in one-to-one correspondence; determining execution bit positions corresponding to the plurality of data processing modes respectively according to the plurality of sub-data and the original data corresponding to the plurality of sub-data respectively, wherein the plurality of data processing modes and the plurality of original data are in one-to-one correspondence; and constructing the processing mode link corresponding to the data association link according to the execution bit positions corresponding to the plurality of data processing modes respectively.

[0012] According to another aspect of the present application, a data processing apparatus is provided, comprising: an obtaining module configured to obtain a data processing parameter and a target scenario for processing data; a first determining module configured to determine whether the target scenario is a first scenario; a second determining module configured to determine a first link library and a second link library corresponding to the target scenario when the first determining module determines that the target scenario is the first scenario, wherein the second link library is constructed according to the first link library; a third determining module configured to determine a parameter similarity index between the data processing parameter and a plurality of reference processing parameters; a fourth determining module configured to determine whether there is at least one similarity determination result in which a parameter similarity index is greater than a parameter similarity threshold in a plurality of parameter similarity indexes, wherein the plurality of parameter similarity indexes correspond to the plurality of reference processing parameters one by one; and a fifth determining module configured to determine a data processing result corresponding to the data processing parameter according to the data processing parameter, the first link library, the second link library, and the similarity determination result.

[0013] According to still another aspect of the present application, an electronic device is provided, comprising: a processor; a memory configured to store instructions executable by the processor; and wherein the processor is configured to execute the instructions to implement any of the data processing methods described above.

[0014] According to yet another aspect of the present application, a computer readable storage medium is provided, comprising: instructions stored in the computer readable storage medium, which when executed by a processor of an electronic device, enable the electronic device to perform any of the data processing methods described above.

[0015] In the embodiment of the present application, the data processing parameter used for processing data and the target scene are acquired; a first determination result of whether the target scene is a first scene is determined; in the case that the first determination result is that the target scene is the first scene, a first link library corresponding to the target scene and a second link library are determined, wherein the second link library is constructed according to the first link library; a parameter similarity index between the data processing parameter and a plurality of reference processing parameters is determined; a similar determination result that at least one parameter similarity index is greater than a parameter similarity threshold is determined in the plurality of parameter similarity indexes, wherein the plurality of parameter similarity indexes correspond to the plurality of reference processing parameters one by one; and a data processing result corresponding to the data processing parameter is determined according to the data processing parameter, the first link library, the second link library, and the similar determination result. By acquiring the data processing parameter and the target scene, the core requirement of data processing and the application scene attribute are determined, and then it is determined whether the target scene is the first scene. When it is confirmed that the target scene is the first scene, the first link library and the second link library constructed according to the first link library are called to form a function-redundant double-link configuration to strengthen the disaster recovery basis of data processing. At the same time, the similarity index of the data processing parameter and the plurality of reference processing parameters is calculated and it is determined whether there is a case that meets the threshold value, so as to ensure the adaptability of the data processing parameter and the link library. Finally, the data processing result is determined in combination with the data processing parameter, the double-link library and the similar determination result, so as to effectively improve the stability of data processing. BRIEF DESCRIPTION OF DRAWINGS

[0016] The accompanying drawings, which form a part of the specification, are included to provide a further understanding of the application and are incorporated herein in conjunction with the description of the application. The embodiments of the present application, and the descriptions thereof, are used to explain the application and are not intended to limit the application. In the drawings:

[0017] Figure 1 A hardware structure block diagram of a mobile terminal for executing a data processing method provided in an embodiment of the present application is shown;

[0018] Figure 2 A flowchart of a data processing method provided in an embodiment of the present application is shown;

[0019] Figure 3 A general architecture diagram of a conventional data processing provided in an embodiment of the present application is shown;

[0020] Figure 4 A general architecture diagram of improved data processing provided in an embodiment of the present application is shown;

[0021] Figure 5 A data system hierarchical disaster recovery architecture diagram provided in an embodiment of the present application is shown;

[0022] Figure 6A flowchart of a disaster recovery capability configuration process is shown according to an embodiment of the present application;

[0023] Figure 7 A schematic diagram of an inter-system relationship architecture is shown according to an embodiment of the present application;

[0024] Figure 8 A schematic diagram of a batch link and data link relationship is shown according to an embodiment of the present application;

[0025] Figure 9 A structural block diagram of a data processing apparatus is shown according to an embodiment of the present application.

[0026] Among the above figures, the following reference signs are included:

[0027] 102, processor; 104, memory; 106, transmission device; 108, input / output device. DETAILED DESCRIPTION

[0028] It should be noted that the embodiments and features of the embodiments in the present application can be combined with each other without conflict. The technical solutions in the embodiments of the present application will be described in detail below with reference to the accompanying drawings and in combination with the embodiments.

[0029] In order to enable those skilled in the art to better understand the present application, the technical solutions in the embodiments of the present application will be described clearly and completely below with reference to the accompanying drawings in the embodiments of the present application. Obviously, the described embodiments are only a part of the embodiments of the present application, not all the embodiments. Based on the embodiments in the present application, all other embodiments obtained by those skilled in the art without creative labor should be within the scope of protection of the present application.

[0030] It should be noted that the terms "first", "second" and the like in the specification and claims of the present application and the above-mentioned figures are used to distinguish similar objects, and do not necessarily indicate a specific order or sequence. It should be understood that the data thus used can be interchanged under appropriate circumstances, so that the embodiments of the present application described herein can be implemented. In addition, the terms "include" and "have" and any variations thereof are intended to cover non-exclusive inclusion, for example, a process, method, system, product or device including a series of steps or units does not necessarily limit to those steps or units clearly listed, but can include other steps or units not clearly listed or inherent to these processes, methods, products or devices.

[0031] It should be noted that the data and information obtained in this application are all information and data that have been fully authorized by users, and the collection, storage, use, processing, transmission, provision, disclosure, and application of related data or information all comply with the conditions and permissions for data or information processing, necessary confidentiality measures have been taken, and corresponding operation entry points are provided for users to choose to authorize or refuse. For the results of automated decision-making, corresponding operation entry points are also provided for users to choose to agree to or refuse the results of automated decision-making; if the user chooses to refuse, the operations and processing related to automated decision-making will not be executed.

[0032] The technical solutions of the present invention will be clearly and completely described below with reference to the accompanying drawings in the embodiments of the present invention.

[0033] The methods and embodiments provided in this application can be executed on a mobile terminal, computer terminal, or similar computing device. Taking running on a mobile terminal as an example, Figure 1 A hardware structure block diagram of a mobile terminal performing a data processing method according to an embodiment of this application is shown. Figure 1 As shown, a mobile terminal may include one or more ( Figure 1 Only one is shown in the diagram. A processor 102 (which may include, but is not limited to, a microprocessor MCU or a programmable logic device FPGA, etc.) and a memory 104 for storing data are also shown. The mobile terminal may further include a transmission device 106 for communication functions and an input / output device 108. Those skilled in the art will understand that... Figure 1 The structure shown is for illustrative purposes only and does not limit the structure of the mobile terminal described above. For example, the mobile terminal may also include components that are more... Figure 1 The more or fewer components shown, or having the same Figure 1 The different configurations shown.

[0034] The memory 104 can be used to store computer programs, such as software programs of application software and modules, such as a computer program corresponding to the data processing method in the embodiments of the present application. The processor 102 can execute various functions and data processing, i.e., implement the above method, by running the computer program stored in the memory 104. The memory 104 can include a high-speed random access memory, and can further include a non-volatile memory, such as one or more magnetic storage devices, flash memories, or other non-volatile solid-state memories. In some examples, the memory 104 can further include a memory remotely arranged with respect to the processor 102, which can be connected to the mobile terminal through a network. Examples of the above network include, but are not limited to, the Internet, an intranet, a local area network, a mobile communication network, and a combination thereof. The transmission device 106 is used to receive or send data via a network. The specific examples of the above network can include a wireless network provided by a communication provider of the mobile terminal. In one example, the transmission device 106 includes a network adapter (Network Interface Controller, NIC) which can be connected to other network devices through a base station so as to communicate with the Internet. In one example, the transmission device 106 can be a radio frequency (Radio Frequency, RF) module which is used to communicate with the Internet in a wireless manner.

[0035] In the embodiments of the present application, a data processing method running on a mobile terminal, a computer terminal or a similar computing device is provided. It should be noted that the steps shown in the flowchart of the accompanying drawings can be executed in a computer system such as a set of computer executable instructions, and although a logical order is shown in the flowchart, in some cases, the steps shown or described herein can be executed in an order different from that shown.

[0036] Figure 2 A flowchart of a data processing method according to an embodiment of the present application is shown, as shown in Figure 2 The method includes the following steps:

[0037] S202, obtaining data processing parameters and a target scene for processing data.

[0038] In step S202 provided in the present application, data processing parameters and a target scene for processing data are obtained.

[0039] The data processing involves processing data, which is a data processing process for analyzing, converting, calculating, optimizing or other operations on original data to achieve a specific target or requirement.

[0040] The data processing parameter is a demand parameter for data processing, for example, a demand parameter (such as a user target table) initiated by a data user, which can be used to explicitly specify the object, standard, range and other key information of data processing.

[0041] The target scene is an application scene corresponding to the data processing demand, for example, a first scene (high disaster backup level, stable demand index greater than threshold) or a second scene (low disaster backup level, stable demand index less than or equal to threshold).

[0042] By obtaining the data processing parameter and the target scene for processing the data, the core demand and application scene attribute of data processing can be explicitly specified, thereby providing accurate guidance and basis for subsequent data processing.

[0043] S204, a first determination result of determining whether the target scene is a first scene.

[0044] In step S204 provided in the present application, the first determination result of determining whether the target scene is the first scene is determined.

[0045] The first scene is a scene where the stable demand index is greater than the stable demand threshold, and the core is the high requirement for the stability (continuity) of data processing, corresponding to a high disaster backup level. In the first scene, the reliability of data processing (such as uninterrupted) needs to be maintained as much as possible.

[0046] The first determination result is a determination result of whether the target scene is the first scene, for example, the first determination result can be that the target scene is the first scene or the target scene is not the first scene, which is used for subsequent selection of an adaptive link library.

[0047] By determining the first determination result of whether the target scene is the first scene, the demand degree of stability of the current data processing task can be explicitly specified, and then the system can select an adaptive link library according to the scene type for configuration.

[0048] S206, in a case where the first determination result is that the target scene is the first scene, determining a first link library and a second link library corresponding to the target scene, wherein the second link library is constructed according to the first link library.

[0049] In step S206 provided in the present application, in a case where the first determination result is that the target scene is the first scene, a first link library and a second link library corresponding to the target scene are determined.

[0050] The first link library is a link library corresponding to the first scene (high disaster backup level), which includes a data processing link required by data processing in the first scene.

[0051] Among them, the second link library is constructed based on the first link library, and is used for a backup link library to provide redundant support when the first link library fails or cannot meet the demand.

[0052] In the case that the first determination result is that the target scene is the first scene, the corresponding first link library and the second link library constructed based thereon are determined, which can ensure the data processing stability in a high disaster recovery level scene through a dual link redundancy configuration. In addition, the second link library can be multiple.

[0053] S208, determine a parameter similarity index between the data processing parameter and each of the plurality of reference processing parameters.

[0054] In step S208 provided in the present application, the parameter similarity index between the data processing parameter and each of the plurality of reference processing parameters is determined.

[0055] Among them, the plurality of reference processing parameters are the processing parameters corresponding to the existing data processing links in all link libraries (including the first link library, the second link library, and the third link library).

[0056] Among them, the parameter similarity index is used to quantify the similarity between the data processing parameter and each reference processing parameter.

[0057] By determining the parameter similarity index between the data processing parameter and the plurality of reference processing parameters, the matching degree of the current parameter and the existing link can be quantitatively evaluated, so as to ensure the selection of the most suitable data processing path and avoid data processing instability or effect deviation caused by parameter incompatibility. And by quantifying the similarity between the data processing parameter and the plurality of reference processing parameters corresponding to the existing data processing links in all link libraries, it can be quickly judged whether there is an adaptive historical processing link for the current data processing demand, which provides a core basis for subsequent determination of whether a new link needs to be added or an existing link needs to be adjusted, ensures that the selection of the data processing link is more in line with the actual demand, avoids resource waste caused by blind construction of new links, and improves the adaptation efficiency and accuracy of data processing.

[0058] S210, determine whether there is at least one parameter similarity index greater than the parameter similarity threshold in the plurality of parameter similarity indexes, wherein the plurality of parameter similarity indexes correspond one-to-one to the plurality of reference processing parameters.

[0059] In step S210 provided in the present application, it is determined whether there is at least one parameter similarity index greater than the parameter similarity threshold in the plurality of parameter similarity indexes.

[0060] The at least one parameter similarity index is greater than a parameter similarity threshold value in the plurality of parameter similarity indexes.

[0061] The parameter similarity threshold value is a pre-set critical value for judging the degree of parameter similarity, and is used to judge whether the data processing parameter and the reference processing parameter are adapted.

[0062] The similarity determination result is a determination result of whether there is at least one parameter similarity index greater than the parameter similarity threshold value in the plurality of parameter similarity indexes. For example, the similarity determination result can be that there is at least one parameter similarity index greater than the parameter similarity threshold value in the plurality of parameter similarity indexes, or that there is no at least one parameter similarity index greater than the parameter similarity threshold value in the plurality of parameter similarity indexes.

[0063] By determining whether there is an index greater than a pre-set critical value (the parameter similarity threshold value) in the plurality of parameter similarity indexes, it can be quickly determined whether there is an existing reference processing parameter and a corresponding link that is adapted to the current data processing parameter, ensuring that the dual-link library and the data processing demand are accurately matched in the first scenario, avoiding processing fluctuations or interruptions caused by improper link adaptation, and strengthening the stability of data processing from the link matching level. In addition, it can also provide a clear direction for subsequent selection of link adjustment, addition or switching strategies.

[0064] S212, determining a data processing result corresponding to the data processing parameter according to the data processing parameter, the first link library, the second link library, and the similarity determination result.

[0065] In step S212 provided in the present application, the data processing result corresponding to the data processing parameter is determined according to the data processing parameter, the first link library, the second link library, and the similarity determination result.

[0066] The data processing result is a result obtained by performing data processing on the corresponding data processing link and meeting the processing demand in combination with the data processing parameter, the first link library and the second link library, and the similarity determination result.

[0067] Determining the data processing result in combination with the data processing parameter, the dual-link library and the similarity determination result can effectively avoid processing interruption caused by single link failure or improper parameter adaptation, and improve the stability and effect of data processing.

[0068] As an optional embodiment, the determining the data processing result corresponding to the data processing parameter according to the data processing parameter, the first link library, the second link library, and the similarity determination result comprises: in a case where the similarity determination result is that there is at least one parameter similarity index greater than the parameter similarity threshold in the plurality of parameter similarity indexes, determining a to-be-compared parameter from the plurality of reference processing parameters, wherein the to-be-compared parameter is a reference processing parameter in the plurality of reference processing parameters and the parameter similarity index of which is greater than the parameter similarity threshold; determining a second determination result of whether the target scene is a to-be-compared scene, wherein the to-be-compared scene is a scene corresponding to the to-be-compared parameter; and determining the data processing result corresponding to the data processing parameter according to the second determination result, the data processing parameter, the first link library, the second link library, and the similarity determination result.

[0069] In this embodiment, specific steps of determining the data processing result corresponding to the data processing parameter according to the data processing parameter, the first link library, the second link library, and the similarity determination result are described.

[0070] The to-be-compared parameter is a reference processing parameter in the plurality of reference processing parameters and the parameter similarity index of which is greater than the parameter similarity threshold.

[0071] The to-be-compared scene is a stability requirement scene of data processing corresponding to the to-be-compared parameter, for example, the first scene or the second scene.

[0072] The second determination result is a determination result of whether the target scene is the to-be-compared scene, for example, the second determination result is that the target scene is the to-be-compared scene or the target scene is not the to-be-compared scene.

[0073] In a case where the similarity determination result is that there is a parameter similarity index meeting the threshold, a to-be-compared parameter highly matching the data processing parameter is screened out, and then it is judged whether the target scene is consistent with a to-be-compared scene corresponding to the to-be-compared parameter, so that the configuration of the first link library and the second link library can be accurately adjusted based on the scene matching condition, the double-link library is adapted to the target scene with high disaster recovery requirement, the operation of the link is prevented from being unstable due to the scene mismatch, reliable support is provided for data processing by relying on the adapted link, and the problems of unstable data processing and poor effect in the related art are further solved.

[0074] As an optional embodiment, according to the second determination result, the data processing parameter, the first link library, the second link library, and the similar determination result, the data processing result corresponding to the data processing parameter is determined, including: in the case that the second determination result is that the target scene is not the scene to be compared, adjusting the data processing link corresponding to the scene to be compared in the third link library to the first link library and the second link library to obtain the first newly added link corresponding to the first link library and the second newly added link corresponding to the second link library, wherein the third link library is a link library corresponding to the second scene, the first scene is a scene with a stable demand index greater than a stable demand threshold, the second scene is a scene with a stable demand index less than or equal to the stable demand threshold, and the stable demand index represents the degree of stability required when data is processed; and the data processing parameter is processed by using the first newly added link corresponding to the first link library to obtain the data processing result corresponding to the data processing parameter.

[0075] In this embodiment, the specific steps of determining the data processing result corresponding to the data processing parameter according to the second determination result, the data processing parameter, the first link library, the second link library, and the similar determination result are described.

[0076] Among them, the third link library is involved, which is a collection of data processing links corresponding to the second scene (low disaster recovery level).

[0077] Among them, the data processing link is involved, which is a specific path in the link library that carries the data processing process and contains the execution logic of a series of data processing operations. For example, in the data processing process, the data processing link can be a collection of a series of processing steps and logic from data input to output, and each link corresponds to multiple data processing methods (at least one data processing method).

[0078] Among them, the first newly added link is involved, which is a newly added link in the first link library obtained by adjusting the data processing link corresponding to the scene to be compared from the third link library to the first link library in the case that the target scene is not the scene to be compared.

[0079] Among them, the second newly added link is involved, which is a newly added link in the second link library obtained by adjusting the data processing link corresponding to the scene to be compared from the third link library to the second link library in the case that the target scene is not the scene to be compared, and forms a redundancy with the original link in the second link library to guarantee the disaster recovery demand of the first scene.

[0080] Among them, the second scene is involved, which is a scene with a stable demand index less than or equal to a stable demand threshold, and has a low requirement for data processing stability, corresponding to a low disaster recovery level.

[0081] The stability demand index is an index quantifying the degree of demand for stability in the data processing scene, and the numerical value of the stability demand index directly reflects the strength of the demand for uninterrupted and undisturbed processing in the scene.

[0082] The stability demand threshold is a critical value for distinguishing between the first scene and the second scene, and is used to distinguish between a high stability demand scene (the first scene) and a low stability demand scene (the second scene).

[0083] The data processing link of the to-be-compared scene in the third link library corresponding to the low stability demand scene is synchronously adjusted to the first link library and the second link library to form a new link, which not only supplements the processing resources of the dual-link configuration of the high stability demand scene to ensure that the dual link always maintains consistent functionality and redundancy backup capability, but also reuses the existing link resources to avoid repeated construction. The new link of the first link library is used to process the data processing parameters, which can ensure the adaptability and continuity of data processing in the high stability demand scene, effectively avoid processing interruption caused by single link failure or resource shortage, and further improve the stability and effect of data processing.

[0084] As an optional embodiment, the first new link corresponding to the first link library is used to process the data processing parameters to obtain a data processing result corresponding to the data processing parameters, including: determining a link stability index corresponding to the first new link; in the case that the link stability index is less than a link stability threshold, stopping using the first new link corresponding to the first link library to process the data processing parameters, and using the second new link corresponding to the second link library to process the data processing parameters to obtain a data processing result corresponding to the data processing parameters.

[0085] In this embodiment, the specific steps of using the first new link corresponding to the first link library to process the data processing parameters to obtain a data processing result corresponding to the data processing parameters are described.

[0086] The link stability index is an index quantifying the stability of the first new link when processing data.

[0087] The link stability threshold is a preset critical value for determining whether the link stability index meets the demand for data processing. If the link stability index is lower than the threshold, it is considered that the stability of the link is insufficient.

[0088] The data processing stability degree of the first newly added link is quantified to obtain a link stability index, and when the index is lower than a preset threshold (a link stability threshold), the link is stopped from being used and the second newly added link is switched to process data, so that the processing interruption or abnormal result caused by unstable link can be avoided in time, thereby ensuring the continuity of data processing in the first scenario based on the redundancy disaster recovery mechanism of the double newly added links.

[0089] As an optional embodiment, according to the data processing parameter, the first link library, the second link library, and the similarity determination result, a data processing result corresponding to the data processing parameter is determined, including: in the case that the similarity determination result is that there is no at least one parameter similarity index greater than the parameter similarity threshold in the plurality of parameter similarity indexes, determining a data association link corresponding to the data processing parameter; determining a processing mode link corresponding to the data association link and to-be-processed data, wherein the processing mode link includes a plurality of data processing modes; adding the processing mode link to the first link library and the second link library respectively; processing the to-be-processed data according to the processing mode link in the first link library and the processing mode link in the second link library to obtain the data processing result corresponding to the data processing parameter.

[0090] In this embodiment, the specific steps of determining the data processing result corresponding to the data processing parameter according to the data processing parameter, the first link library, the second link library, and the similarity determination result are illustrated.

[0091] Among them, the data association link is involved, which is a data association path corresponding to the data processing parameter, and can be a table-level data association path established based on data blood relationship. For example, the data association link can be a data blood link, which can comprehensively reflect the source, flow direction and dependency relationship of data, etc.

[0092] Among them, the processing mode link is involved, which is a data processing path corresponding to the data association link and containing a plurality of data processing modes, including each processing step and operation that the data needs to go through in the processing process.

[0093] Among them, the to-be-processed data is involved, which is the original data or intermediate data covered by the data association link and needs to be executed through a series of operations of the processing mode link, and can be the object of data processing.

[0094] Among them, the plurality of data processing modes are involved, which are specific operation logics for processing data contained in the processing mode link, and are basic units for realizing data processing, such as data preprocessing, cleaning, conversion, calculation, summarization, etc., which are used to gradually transform the to-be-processed data into target results.

[0095] When the parameter similarity is insufficient, the data associated link matching the data processing parameter is determined, so as to accurately locate the to-be-processed data and the flow path thereof; the corresponding processing mode link and the original data are obtained based on the associated link, so as to ensure that the processing logic is highly adapted to the data characteristics, and the processing mode link is added to the first link library and the second link library, and finally the data is processed through the double link libraries, so as to significantly improve the processing accuracy and the disaster recovery capability.

[0096] As an optional embodiment, the to-be-processed data is processed according to the processing mode link in the first link library and the processing mode link in the second link library, so as to obtain the data processing result corresponding to the data processing parameter, including: determining the target processing link corresponding to the to-be-processed data, wherein the target processing link includes any one of the following: the processing mode link in the first link library, the processing mode link in the second link library; according to the execution order of the plurality of data processing modes in the target processing link, the to-be-processed data is processed according to the first processing mode to obtain the first processing data, and the first processing data is processed according to the next processing mode to obtain the next processing data, until the plurality of data processing modes are executed to obtain the target processing data corresponding to the to-be-processed data; and determining the data processing result corresponding to the data processing parameter according to the target processing data.

[0097] In this embodiment, the specific steps of processing the to-be-processed data according to the processing mode link in the first link library and the processing mode link in the second link library to obtain the data processing result corresponding to the data processing parameter are described.

[0098] Among them, the target processing link is involved, which is a specific link selected from the processing mode link in the first link library or the second link library, used for executing the to-be-processed data processing flow, and is the core execution path of data processing.

[0099] Among them, the processing mode link in the first link library is involved, which is an ordered execution path stored in the first link library and containing a plurality of data processing modes.

[0100] Among them, the processing mode link in the second link library is involved, which is an ordered execution path stored in the second link library and functionally equivalent to the processing mode link in the first link library, used for disaster backup.

[0101] Among them, the execution order is involved, which is the execution logic of the plurality of data processing modes in the processing mode link, and the step flow of data processing is clear, ensuring that the processing process is orderly promoted.

[0102] The first processing mode is a specific data processing operation first performed in the target processing link and is an initial operation for the processed data to enter the processing flow.

[0103] The first processing data is intermediate data obtained after the processed data is processed by the first processing mode and is input data for subsequent processing steps.

[0104] The next processing mode is a specific data processing operation performed after the current processing mode (e.g., the first processing mode) and is used to further process the data obtained in the previous step.

[0105] The next processing data is intermediate data obtained after the current processing data (e.g., the first processing data) is processed by the next processing mode and is sequentially processed to gradually obtain the target processing data.

[0106] The target processing data is the final data obtained after the processed data is sequentially processed by all data processing modes in the target processing link.

[0107] According to the predefined execution order in the link, the data processing modes are sequentially called to ensure that the processed data is gradually converted according to the standardized process, avoid errors caused by disordered operation sequence, ensure the logical coherence in the data processing process, and further generate more reliable and accurate results based on the target processing data and parameters.

[0108] As an optional embodiment, determining a processing mode link corresponding to a data association link includes: determining a plurality of data association relationships corresponding to the data association link, respectively, wherein the plurality of data association relationships respectively include a plurality of sub-data, a plurality of original data corresponding to the plurality of sub-data, and a plurality of data processing modes corresponding to the plurality of original data, the corresponding data processing mode is used to process the corresponding original data to obtain the corresponding sub-data, and the plurality of original data and the plurality of sub-data are one-to-one corresponding; determining execution bit positions corresponding to the plurality of data processing modes according to the plurality of sub-data and the plurality of original data corresponding to the plurality of sub-data, wherein the plurality of data processing modes and the plurality of original data are one-to-one corresponding; and constructing the processing mode link corresponding to the data association link according to the execution bit positions corresponding to the plurality of data processing modes.

[0109] In this embodiment, the specific steps of determining the processing mode link corresponding to the data association link are described.

[0110] The plurality of sub-data is result data obtained after the corresponding original data is processed by the corresponding data processing mode in the data association relationship.

[0111] Among them, a plurality of original data are involved, which are original data (i.e., parent data corresponding to sub-data) that have not been processed by the corresponding data processing mode in the data association relationship.

[0112] Among them, the execution bit is involved, which is the execution order of each data processing mode determined according to the association logic of the sub-data and the corresponding original data, and the execution order of the different operations in the processing mode link is clear.

[0113] Determining a plurality of data association relationships corresponding to the data association link can clarify the source and flow of data, ensure the logical clarity and traceability of data processing, so as to determine the execution bit of the data processing mode according to the sub-data and the original data, and accurately determine the data processing flow that completely matches the data association link.

[0114] Through the above embodiment, the data processing parameter and the target scene used for processing the data are obtained; a first determination result of whether the target scene is a first scene is determined; in the case that the first determination result is that the target scene is the first scene, a first link library and a second link library corresponding to the target scene are determined, wherein the second link library is constructed according to the first link library; a parameter similarity index between the data processing parameter and a plurality of reference processing parameters is determined; it is determined whether there is at least one similar determination result in which a parameter similarity index is greater than a parameter similarity threshold in a plurality of parameter similarity indexes, wherein the plurality of parameter similarity indexes correspond one-to-one to the plurality of reference processing parameters; and a data processing result corresponding to the data processing parameter is determined according to the data processing parameter, the first link library, the second link library, and the similar determination result. By obtaining the data processing parameter and the target scene, the core requirement and the application scene attribute of data processing are determined, and then it is determined whether the target scene is the first scene. When it is confirmed that the target scene is the first scene, the first link library and the second link library constructed according to the first link library are called to form a function-redundant double-link configuration to strengthen the disaster recovery basis of data processing, while the similarity index of the data processing parameter and the plurality of reference processing parameters is calculated and it is determined whether there is a case that meets the threshold, to ensure the adaptability of the data processing parameter and the link library. Finally, the data processing result is determined in combination with the data processing parameter, the double-link library and the similar determination result, effectively avoiding processing interruption caused by single link failure or improper parameter adaptation, and improving the stability and effect of data processing.

[0115] In order for those skilled in the art to more clearly understand the technical solutions of the present application, the implementation process of the data processing method of the present application will be described in detail below in conjunction with specific embodiments.

[0116] Figure 3 The overall architecture diagram of the traditional data processing provided by the embodiments of the present application is shown, Figure 4An overall architecture diagram of improved data processing provided by embodiments of the present application is shown in FIG. 1. Figure 3 and Figure 4 As shown in FIG. 1, the overall architecture of data processing (i.e., the overall architecture of improved data processing) of the present application supports different disaster recovery level application scenarios by batch link hierarchical disaster recovery of a data system, realizes consistent disaster recovery capability for different scenarios, and also realizes sharing and reuse of data processing capability and storage capability of the data system to maximize resource utilization. In the traditional overall architecture of data processing, data processing logic such as data processing is implemented by each application for high disaster recovery level application scenarios. Differences in technical capabilities of different users result in unstable data processing. In addition, in the traditional disaster recovery construction of big data batch applications, in addition to the generally low disaster recovery capability of the data system, data processing scenarios with high data processing continuity (i.e., stability) requirements are constructed independently of the data system, which results in the inability to reuse the processed data of the data system during data processing, a large amount of repeated processing and redundant storage, and a great waste of resource investment. In this case, the subsequent overall improvement of the disaster recovery capability of the data system will require more resources. The core capabilities of the data system (multi-source data integration, hybrid storage, and elastic computing) can support efficient data processing.

[0117] The present embodiment relates to a specific data processing method, which can be implemented by a data system. The data system includes a first scenario (i.e., a high disaster recovery level), a second scenario (i.e., a low disaster recovery level), a first processing module and a second processing module corresponding to the first scenario, and a third processing module corresponding to the second scenario. The second processing module is functionally equivalent to the first processing module. The third processing module can be connected to a target detection module for controlling the running stability of the third processing module. The first scenario is a scenario with a stability requirement index greater than a stability requirement threshold. The second scenario is a scenario with a stability requirement index less than or equal to the stability requirement threshold. The stability requirement index represents the degree of stability required when processing data, thereby realizing the improvement of the hierarchical disaster recovery capability at the system level. Further, the first processing module includes a first link library, the second processing module includes a second link library, and the third processing module includes a third link library.

[0118] Figure 5 An overall architecture diagram of the hierarchical disaster recovery architecture of the data system provided by embodiments of the present application is shown in FIG. 2. Figure 5 As shown in FIG. 2, the data system includes:

[0119] Data processing center 1: includes a first module and a third module to establish a data processing end of the data system and deploy all batch link (data processing flow) data processing ends thereof;

[0120] The data processing center 2 comprises a second module to establish a disaster recovery end of the data system, and deploy the disaster recovery end of the batch link of the high disaster recovery level (i.e., the first scenario) to support the data processing requirement scenario with high data processing continuity requirement.

[0121] Based on the above setting, the data processing continuity requirement of different data processing scenarios can be met, and the hierarchical disaster recovery capability at the system level can be constructed, and the rapid configuration of the disaster recovery capability in the batch link dimension is realized, thereby solving the technical problem of unstable data processing and poor data processing effect in the related art when data is processed. The following will be described in detail in combination with specific processing steps.

[0122] Step S1: obtaining data processing parameters and a target scenario for processing data;

[0123] The data processing parameters can be demand parameters initiated by a target object (i.e., a user of the data), specifically, a user target table. The target scenario can be any one of the first scenario (i.e., high disaster recovery level) and the second scenario (i.e., low disaster recovery level).

[0124] Step S2: determining a first determination result of whether the target scenario is the first scenario;

[0125] For example, it is determined whether the target scenario is a high disaster recovery level.

[0126] Step S3: in the case that the first determination result is that the target scenario is the first scenario, determining a first link library corresponding to the target scenario and a second link library, wherein the second link library is constructed according to the first link library;

[0127] For example, if the target scenario is a high disaster recovery level, the first link library of the first module and the second link library of the second module are used subsequently, and data processing operation is performed based on the data processing parameters.

[0128] Step S4: determining a parameter similarity index between the data processing parameters and a plurality of reference processing parameters;

[0129] For example, it is first determined whether the data processing parameters themselves exist corresponding data processing links. For this, the data processing parameters are respectively compared with reference processing parameters corresponding to all data processing links in all link libraries (including the first link library, the second link library and the third link library) to determine whether the data processing parameters themselves exist corresponding data processing links in the data system.

[0130] Step S5: determining whether there is at least one similarity determination result in which a parameter similarity index is greater than a parameter similarity threshold value in the plurality of parameter similarity indexes, wherein the plurality of parameter similarity indexes correspond to the plurality of reference processing parameters one by one;

[0131] For example, after performing similarity comparison on the data processing parameter and the reference processing parameters corresponding to all data processing links in all link libraries (including the first link library, the second link library, and the third link library) respectively, it is determined whether there is at least one similarity determination result in which a parameter similarity index is greater than a parameter similarity threshold value. If there is, it can be considered that the data processing parameter itself has a corresponding data processing link in the data system. Otherwise, it can be considered that the data processing parameter itself does not have a corresponding data processing link in the data system.

[0132] Step S6: determining a data processing result corresponding to the data processing parameter according to the data processing parameter, the first link library, the second link library, and the similarity determination result.

[0133] Further, S6 can further include:

[0134] In the case that the similarity determination result is that there is at least one parameter similarity index greater than the parameter similarity threshold value in the plurality of parameter similarity indexes, a to-be-compared parameter is determined from the plurality of reference processing parameters, wherein the to-be-compared parameter is a reference processing parameter in the plurality of reference processing parameters whose parameter similarity index is greater than the parameter similarity threshold value; a second determination result of whether a target scene is a to-be-compared scene is determined, wherein the to-be-compared scene is a scene corresponding to the to-be-compared parameter; and a data processing result corresponding to the data processing parameter is determined according to the second determination result, the data processing parameter, the first link library, the second link library, and the similarity determination result.

[0135] For example, in the case that the data processing parameter itself has a corresponding data processing link in the data system, the to-be-compared parameter is determined from the plurality of reference processing parameters, which can be considered as the data processing parameter. However, it is considered that the data user may switch the scene requirement of data processing, for example, the scene corresponding to the to-be-compared parameter is the second scene, but the scene corresponding to the data processing parameter is the first scene. In this case, the data processing link corresponding to the to-be-compared parameter needs to be adjusted in the link library to realize scene switching.

[0136] Further, determining a data processing result corresponding to the data processing parameter according to the second determination result, the data processing parameter, the first link library, the second link library, and the similarity determination result can further include:

[0137] In a case that the second determination result is that the target scene is not the to-be-compared scene, data processing links corresponding to the to-be-compared scene in the third link library are adjusted to the first link library and the second link library, to obtain a first newly-added link corresponding to the first link library and a second newly-added link corresponding to the second link library, wherein the third link library is a link library corresponding to a second scene, the first scene is a scene in which a stable demand index is greater than a stable demand threshold, the second scene is a scene in which the stable demand index is less than or equal to the stable demand threshold, and the stable demand index represents a degree of stability required when data is processed.

[0138] For example, the scene corresponding to the to-be-compared parameter is the second scene, but the scene corresponding to the data processing parameter is the first scene. In this case, the data processing links corresponding to the to-be-compared scene in the third link library need to be adjusted to the first link library and the second link library, to obtain the first newly-added link corresponding to the first link library and the second newly-added link corresponding to the second link library, and then the first newly-added link corresponding to the first link library is used to process the data processing parameter, to obtain the data processing result corresponding to the data processing parameter.

[0139] In a case that the second determination result is that the target scene is not the to-be-compared scene, data processing links corresponding to the to-be-compared scene in the third link library are adjusted to the first link library and the second link library, to obtain a first newly-added link corresponding to the first link library and a second newly-added link corresponding to the second link library.

[0140] Further, processing the data processing parameter by using the first newly-added link corresponding to the first link library to obtain the data processing result corresponding to the data processing parameter can further include:

[0141] determining a link stability index corresponding to the first newly-added link; in a case that the link stability index is less than a link stability threshold, stopping processing the data processing parameter by using the first newly-added link corresponding to the first link library, and processing the data processing parameter by using the second newly-added link corresponding to the second link library to obtain the data processing result corresponding to the data processing parameter.

[0142] For example, the scenario corresponding to the to-be-compared parameter is the second scenario, but the scenario corresponding to the data processing parameter is the first scenario. In this case, the data processing link corresponding to the to-be-compared scenario in the third link library needs to be adjusted to the first link library and the second link library to obtain the first newly added link corresponding to the first link library and the second newly added link corresponding to the second link library. Then, the first newly added link corresponding to the first link library is used to process the data processing parameter. However, during the processing, the first newly added link corresponding to the first link library has an abnormal problem, which leads to unstable data processing. In this case, the first newly added link corresponding to the first link library is stopped to process the data processing parameter, and the second newly added link corresponding to the second link library is used to process the data processing parameter to obtain the data processing result corresponding to the data processing parameter, thereby realizing the stability of the data processing of the high disaster recovery level.

[0143] Further, S6 can further include:

[0144] In a case where the similarity determination result is that at least one parameter similarity index in the plurality of parameter similarity indexes is not greater than the parameter similarity threshold, a data processing link corresponding to the data processing parameter is determined; a processing mode link corresponding to the data processing link and to-be-processed data are determined, the processing mode link including a plurality of data processing modes; the processing mode link is added to the first link library and the second link library; and the to-be-processed data is processed according to the processing mode link in the first link library and the processing mode link in the second link library to obtain a data processing result corresponding to the data processing parameter.

[0145] For example, in a case where the data processing parameter itself does not exist in the data system, a data bloodline link (i.e., a data association link) corresponding to the data processing parameter is determined, and a batch job link (i.e., a processing mode link) corresponding to the data association link and to-be-processed data are determined, so as to sequentially process the to-be-processed data according to the batch job link to obtain a data processing result corresponding to the data processing parameter.

[0146] Further, determining the processing mode link corresponding to the data association link includes:

[0147] Determine a plurality of data association relationships corresponding to the data association link respectively, wherein the plurality of data association relationships respectively include a plurality of sub-data, a plurality of original data corresponding to the plurality of sub-data, and a plurality of data processing modes corresponding to the plurality of original data, the corresponding data processing mode is used to process the corresponding original data to obtain the corresponding sub-data, the plurality of original data and the plurality of sub-data are one-to-one correspondence; according to the plurality of sub-data and the original data corresponding to the plurality of sub-data, determine the execution bit corresponding to the plurality of data processing modes respectively, wherein the plurality of data processing modes and the plurality of original data are one-to-one correspondence; according to the execution bit corresponding to the plurality of data processing modes respectively, construct the processing mode link corresponding to the data association link. Specifically includes:

[0148] A1, obtain the data link (i.e. data association link) by using the target table (i.e. data processing parameter), including: according to the target table, combined with the data blood relationship table (i.e. data association relationship), obtaining the data link.

[0149] A2, obtain the batch link (i.e. processing mode link) based on the data link, including: according to the corresponding relationship between the table on the data link and the batch job (i.e. the corresponding relationship between the data blood relationship table in the data association link and the data processing mode in the processing mode link), combined with the batch job relationship table, obtaining the batch processing link (i.e. batch link), realizing the generation of batch processing link from blood data link.

[0150] A3, pull the program of each batch job on the batch link, mainly according to the corresponding relationship between batch job and program code, pull from the code library storing program code.

[0151] A4, pull the database table structure involved in each batch job on the batch link, according to the data link, obtain the involved data table, obtain the table structure from the structure library storing the data table structure.

[0152] A5, deploy the program and data table of the batch link which needs to improve the disaster recovery capability in the data processing center 2, mainly including the program of batch job and the table structure of the involved data table.

[0153] A6, configure the newly added data table synchronization task between data processing center 1 and data processing center 2, to realize that the processing mode link is respectively added to the first link library and the second link library, and then realize the disaster recovery between the master and standby data processing centers.

[0154] The data link relationship (i.e. data association relationship) is the data relationship at the data table level established by data blood analysis, and the batch job generating each data table is synchronized to locate the data ownership, so that the data association link can be obtained. Table 1 is a data blood relationship table, as shown in table 1.

[0155] Table 1

[0156]

[0157] The batch job relationship is based on the dependency relationship between batch jobs within and between systems. According to the batch job relationship, a batch link can be constructed. Table 2 is a batch job relationship table, as shown in Table 2.

[0158] Table 2

[0159]

[0160] Through the batch link library, the full-link processing batch job flow of a certain data processing scene can be obtained: first, the data link is obtained according to the data blood relationship table through the target table of the user; second, the batch processing link is obtained according to the batch job relationship table through the corresponding relationship between the table on the data link and the batch job, so as to realize the generation of the batch processing link from the blood data link.

[0161] In addition, the above steps can also be used to construct any one of all link libraries (including the first link library, the second link library, and the third link library). Specifically, for the construction of any one link library, the data blood link (i.e., the data link) and the batch job link (i.e., the batch link), and the relationship between the batch link and the data link can be used for construction. That is, based on the data link and the batch link, the batch link library is constructed, so as to support the disaster recovery capability of the adaptive data processing link by fusing the blood relationship at the data table level and the link relationship of the batch job node.

[0162] Figure 6 Fig. 1 shows a disaster recovery capability configuration flow diagram provided by an embodiment of the present application, Figure 7 Fig. 2 shows an inter-system relationship architecture diagram provided by an embodiment of the present application, Figure 8 Fig. 3 shows a batch link and data link relationship diagram provided by an embodiment of the present application, as Figure 6 Figure 7 and Figure 8 ​As shown. For the data system, by constructing the disaster recovery capability configuration center, relying on the data blood relationship to realize the full-link penetration of the disaster recovery capability configuration process, the corresponding disaster recovery capability can be configured based on the batch data processing link of the data system, combined with the differentiated needs of data processing continuity in different data processing scenarios, to improve the stability of data processing of the data system. That is, if the data processing continuity level of a certain data processing scenario needs to be improved or a new data processing scenario with high data processing continuity (i.e. stability) requirement is added, the data link is obtained by tracing back from the target table F of the user (i.e. the target table of the data user), and then the batch processing link is located according to the data table on the data link, and the disaster recovery capability of the batch processing link is improved to ensure the data processing continuity requirement of the data processing scenario.

[0163] Further, according to the processing mode link in the first link library and the processing mode link in the second link library, the data to be processed is processed to obtain a data processing result corresponding to the data processing parameter, which can further include:

[0164] determining a target processing link corresponding to the data to be processed, wherein the target processing link includes any one of the following: a processing mode link in the first link library, and a processing mode link in the second link library; according to the execution order of the plurality of data processing modes in the target processing link, the data to be processed is processed according to a first processing mode to obtain first processing data, and the first processing data is processed according to a next processing mode to obtain next processing data, until the plurality of data processing modes are executed to obtain target processing data corresponding to the data to be processed; and determining a data processing result corresponding to the data processing parameter according to the target processing data.

[0165] Among them, when determining the target processing link corresponding to the data to be processed, if the processing mode link in the first link library can be stably run (such as no exception, etc.), the target processing link is the processing mode link in the first link library. If the processing mode link in the first link library cannot be stably run, or can be stably run at the beginning but appears abnormal in the middle, the target processing link is switched to the processing mode link in the second link library.

[0166] Based on the above, when constructing the batch link library from the data blood relationship link and the batch job link respectively, it can be gradually enriched and improved with the continuous improvement of the disaster recovery capability of the big data application scenario. Moreover, by adopting the data blood relationship analysis method to construct the data blood relationship link that can locate the data attribution, automatic positioning of the batch job on the data link and automatic tracing of the batch processing path can be realized, and the accuracy and efficiency of combing the full-link batch job of the data processing scenario can be improved.

[0167] Furthermore, by constructing a system-level hierarchical disaster recovery capability for the data system, and using batch links as the dimension, disaster recovery capabilities can be configured in a refined manner, fulfilling different disaster recovery capability requirements for different batch links on the system as needed. Moreover, by obtaining program information for each batch job on the batch links through system-level integration, one-click deployment and configuration of the disaster recovery terminal can be achieved.

[0168] It should be noted that the steps shown in the flowchart in the accompanying drawings can be executed in a computer system such as a set of computer-executable instructions, and although a logical order is shown in the flowchart, in some cases the steps shown or described may be executed in a different order than that shown here.

[0169] This application also provides a data processing apparatus. It should be noted that the data processing apparatus of this application can be used to execute the data processing method provided in this application. This apparatus is used to implement the above embodiments and preferred embodiments; details already described will not be repeated. As used below, the term "module" can refer to a combination of software and / or hardware that implements a predetermined function. Although the apparatus described in the following embodiments is preferably implemented in software, hardware implementation, or a combination of software and hardware, is also possible and contemplated.

[0170] The data processing apparatus provided in the embodiments of this application will be described below.

[0171] Figure 9 A structural block diagram of a data processing apparatus according to an embodiment of this application is shown. Figure 9 As shown, the device includes: an acquisition module 902, a first determination module 904, a second determination module 906, a third determination module 908, a fourth determination module 910, and a fifth determination module 912.

[0172] The module 902 is used to acquire data processing parameters and a target scene for data processing. A first determining module 904, connected to the acquisition module 902, is used to determine whether the target scene is a first scene (a first determining result). A second determining module 906, connected to the first determining module 904, is used to determine a first link library and a second link library corresponding to the target scene if the first determining result indicates the target scene is the first scene, wherein the second link library is constructed based on the first link library. A third determining module 908, connected to the second determining module 906, is used to determine the parameter similarity index between the data processing parameters and multiple reference processing parameters. A fourth determining module 910, connected to the third determining module 908, is used to determine whether there is at least one similarity determination result among the multiple parameter similarity indices where the parameter similarity index is greater than the parameter similarity threshold, wherein the multiple parameter similarity indices correspond one-to-one with multiple reference processing parameters. A fifth determining module 912, connected to the fourth determining module 910, is used to determine the data processing result corresponding to the data processing parameters based on the data processing parameters, the first link library, the second link library, and the similarity determination result.

[0173] The data processing device may further include a processor and a memory. The aforementioned acquisition module 902, first determining module 904, second determining module 906, third determining module 908, fourth determining module 910, and fifth determining module 912 are all stored as program units in the memory. The processor executes these program units stored in the memory to achieve the corresponding functions. All of the above modules are located in the same processor; alternatively, the modules may be located in different processors in any combination. The processor contains a kernel, which retrieves the corresponding program units from the memory. One or more kernels may be configured, and adjusting the kernel parameters can address the technical problem in related technologies where unstable data processing leads to poor data processing results. The memory may include non-permanent memory in computer-readable media, such as random access memory (RAM) and / or non-volatile memory, such as read-only memory (ROM) or flash RAM. The memory includes at least one memory chip.

[0174] This invention provides an electronic device including one or more processors, a memory, and one or more programs, wherein the one or more programs are stored in the memory and configured to be executed by the one or more processors, and the one or more programs include a data processing method for performing any of the above-described methods. The device described herein may be a server, PC, PAD, mobile phone, etc.

[0175] This application also provides a computer program product, including a computer program that, when executed by a processor, implements any of the above-described data processing methods.

[0176] It is evident that those skilled in the art will understand that the modules or steps of the present invention described above can be implemented using general-purpose computing devices. They can be centralized on a single computing device or distributed across a network of multiple computing devices. They can be implemented using computer-executable program code, thereby allowing them to be stored in a storage device for execution by a computing device. In some cases, the steps shown or described can be performed in a different order than those described herein, or they can be fabricated as separate integrated circuit modules, or multiple modules or steps can be fabricated as a single integrated circuit module. Thus, the present invention is not limited to any particular hardware and software combination. Those skilled in the art will understand that embodiments of this application can be provided as methods, systems, or computer program products. Therefore, this application can take the form of a completely hardware embodiment, a completely software embodiment, or an embodiment combining software and hardware aspects. Moreover, this application can take the form of a computer program product implemented on one or more computer-usable storage media (including but not limited to disk storage, CD-ROM, optical storage, etc.) containing computer-usable program code.

[0177] This application is described with reference to flowchart illustrations and / or block diagrams according to embodiments of this application. It will be understood that each block of the flowchart illustrations and / or block diagrams, and combinations of blocks in the flowchart illustrations and / or block diagrams, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, special-purpose computer, embedded processor, or other programmable data processing apparatus to produce a machine, such that the instructions, which execute via the processor of the computer or other programmable data processing apparatus, generate instructions for implementing the flowchart illustrations. Figure 1 One or more processes and / or boxes Figure 1 A device that provides the functions specified in one or more boxes.

[0178] These computer program instructions may also be stored in a computer-readable storage medium that can direct a computer or other programmable data processing device to function in a particular manner, such that the instructions stored in the computer-readable storage medium produce an article of manufacture including instruction means, which are implemented in a process Figure 1 One or more processes and / or boxes Figure 1 The functions specified in one or more boxes. These computer program instructions may also be loaded onto a computer or other programmable data processing apparatus to cause a series of operational steps to be performed on the computer or other programmable apparatus to produce a computer-implemented process, thereby providing instructions that execute on the computer or other programmable apparatus for implementing the process. Figure 1 One or more processes and / or boxes Figure 1The steps of the function specified in one or more boxes.

[0179] In a typical configuration, a computing device includes one or more processors (CPUs), input / output interfaces, network interfaces, and memory. Memory may include non-persistent memory in computer-readable media, random access memory (RAM), and / or non-volatile memory such as read-only memory (ROM) or flash RAM. Memory is an example of computer-readable media. Computer-readable media includes both permanent and non-persistent, removable and non-removable media that can store information by any method or technology. Information can be computer-readable instructions, data structures, modules of programs, or other data. Examples of computer storage media include, but are not limited to, phase-change memory (PRAM), static random access memory (SRAM), dynamic random access memory (DRAM), other types of random access memory (RAM), read-only memory (ROM), electrically erasable programmable read-only memory (EEPROM), flash memory or other memory technologies, CD-ROM, digital versatile optical disc (DVD) or other optical storage, magnetic tape, disk storage or other magnetic storage devices, or any other non-transfer medium that can be used to store information that can be accessed by the computing device. As defined in this article, computer-readable media do not include transient media, such as modulated data signals and carrier waves.

[0180] The technical features of the above embodiments can be combined in any way. For the sake of brevity, not all possible combinations of the technical features in the above embodiments are described. However, as long as there is no contradiction in the combination of these technical features, they should be considered to be within the scope of this specification.

[0181] It should also be noted that the terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such process, method, article, or apparatus. Unless otherwise specified, an element defined by the phrase "comprising one..." does not exclude the presence of other identical elements in the process, method, article, or apparatus that includes that element.

[0182] As can be seen from the above description, the embodiments of this application achieve the following technical effects:

[0183] 1) Compared with related technologies, this invention obtains data processing parameters and target scenarios, clarifies the core requirements and application scenario attributes of data processing, and then determines whether the target scenario is the first scenario. When the target scenario is confirmed to be the first scenario, the first link library and the second link library built based on the first link library are called to form a functionally redundant dual-link configuration to strengthen the disaster recovery foundation of data processing. At the same time, the similarity index between the data processing parameters and multiple reference processing parameters is calculated and it is determined whether there is a situation that meets the threshold to ensure the compatibility between the data processing parameters and the link library. Finally, the data processing result is determined by combining the data processing parameters, the dual link library and the similarity determination result, which effectively avoids processing interruption caused by single link failure or improper parameter adaptation, and improves the stability and effect of data processing.

[0184] 2) Compared with related technologies, this invention simultaneously adjusts the data processing links of the comparison scenarios in the third link library corresponding to low-stability demand scenarios to the first and second link libraries to form new links. This not only supplements the dual-link configuration with suitable processing resources for high-stability demand scenarios, ensuring that the dual links always maintain functional consistency and redundancy backup capabilities, but also reuses existing link resources to avoid repeated construction. Then, the data processing parameters are processed through the new links in the first link library. This can ensure the adaptability and continuity of data processing under high-stability demand scenarios, effectively avoid processing interruptions caused by single link failures or insufficient resources, and further improve the stability and effectiveness of data processing.

[0185] 3) Compared with related technologies, the present invention obtains a link stability index by quantifying the data processing stability of the first newly added link. When the index is lower than the preset threshold (link stability threshold), the link is stopped and the data is switched to the second newly added link. This can avoid processing interruption or abnormal results caused by unstable links in a timely manner, thereby ensuring the continuity of data processing in the first scenario based on the redundancy disaster recovery mechanism of the two newly added links.

[0186] The above description is merely a preferred embodiment of this application and is not intended to limit this application. Various modifications and variations can be made to this application by those skilled in the art. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of this application should be included within the protection scope of this application.

Claims

1. A data processing method, characterized in that, include: Obtain the data processing parameters and target scenario used to process the data; The first determination result is to determine whether the target scene is the first scene; If the first determination result is that the target scenario is the first scenario, a first link library and a second link library corresponding to the target scenario are determined, wherein the second link library is constructed based on the first link library; Determine the parameter similarity index between the data processing parameters and multiple reference processing parameters; Among multiple parameter similarity indices, it is determined whether there is at least one parameter similarity index that is greater than the parameter similarity threshold, wherein the multiple parameter similarity indices correspond one-to-one with the multiple reference processing parameters; Based on the data processing parameters, the first link library, the second link library, and the similarity determination result, a data processing result corresponding to the data processing parameters is determined.

2. The method according to claim 1, characterized in that, The step of determining the data processing result corresponding to the data processing parameters based on the data processing parameters, the first link library, the second link library, and the similarity determination result includes: If the similarity determination result is that at least one parameter similarity index among the plurality of parameter similarity indices is greater than the parameter similarity threshold, a parameter to be compared is determined from the plurality of reference processing parameters, wherein the parameter to be compared is a reference processing parameter among the plurality of reference processing parameters whose parameter similarity index is greater than the parameter similarity threshold; A second determination result for determining whether the target scene is a scene to be compared, wherein the scene to be compared is the scene corresponding to the comparison parameter; Based on the second determination result, the data processing parameters, the first link library, the second link library, and the similarity determination result, a data processing result corresponding to the data processing parameters is determined.

3. The method according to claim 2, characterized in that, The step of determining the data processing result corresponding to the data processing parameters based on the second determination result, the data processing parameters, the first link library, the second link library, and the similarity determination result includes: If the second determination result is that the target scenario is not the scenario to be compared, the data processing links in the third link library corresponding to the scenario to be compared are adjusted to the first link library and the second link library to obtain the first new link corresponding to the first link library and the second new link corresponding to the second link library. The third link library is the link library corresponding to the second scenario. The first scenario is a scenario where the stability demand index is greater than the stability demand threshold, and the second scenario is a scenario where the stability demand index is less than or equal to the stability demand threshold. The stability demand index represents the degree of stability required when processing data. The data processing parameters are processed using the first newly added link corresponding to the first link library to obtain the data processing result corresponding to the data processing parameters.

4. The method according to claim 3, characterized in that, The step of using the first newly added link corresponding to the first link library to process the data processing parameters and obtain the data processing result corresponding to the data processing parameters includes: Determine the link stability index corresponding to the first newly added link; If the link stability index is less than the link stability threshold, stop using the first newly added link corresponding to the first link library, process the data processing parameters, and use the second newly added link corresponding to the second link library to process the data processing parameters to obtain the data processing result corresponding to the data processing parameters.

5. The method according to claim 1, characterized in that, The step of determining the data processing result corresponding to the data processing parameters based on the data processing parameters, the first link library, the second link library, and the similarity determination result includes: If the similarity determination result is that there is no parameter similarity index among the plurality of parameter similarity indices that is greater than the parameter similarity threshold, the data association link corresponding to the data processing parameter is determined. Determine the processing method link and the data to be processed corresponding to the data association link, wherein the processing method link includes multiple data processing methods; Add the processing method links to the first link library and the second link library respectively; Based on the processing method links in the first link library and the processing method links in the second link library, the data to be processed is processed to obtain the data processing result corresponding to the data processing parameters.

6. The method according to claim 5, characterized in that, The step of processing the data to be processed according to the processing method links in the first link library and the processing method links in the second link library to obtain the data processing result corresponding to the data processing parameters includes: Determine the target processing link corresponding to the data to be processed, wherein the target processing link includes any one of the following: the processing method link in the first link library, and the processing method link in the second link library; According to the execution order of multiple data processing methods in the target processing chain, the data to be processed is processed according to the first processing method to obtain the first processed data, and the first processed data is processed according to the next processing method to obtain the next processed data, until multiple data processing methods are completed to obtain the target processed data corresponding to the data to be processed. Based on the target processing data, determine the data processing result corresponding to the data processing parameters.

7. The method according to any one of claims 5 to 6, characterized in that, Determining the processing method link corresponding to the data association link includes: Determine multiple data association relationships corresponding to the data association links, wherein each of the multiple data association relationships includes multiple sub-data, original data corresponding to each of the multiple sub-data, and data processing methods corresponding to each of the multiple original data. The corresponding data processing methods are used to process the corresponding original data to obtain the corresponding sub-data, and the multiple original data and the multiple sub-data correspond one-to-one. Based on the plurality of sub-data and the original data corresponding to each of the plurality of sub-data, the execution position corresponding to each of the plurality of data processing methods is determined, wherein the plurality of data processing methods correspond one-to-one with the plurality of original data; Based on the execution position corresponding to the multiple data processing methods, a processing method link corresponding to the data association link is constructed.

8. A data processing apparatus, characterized in that, include: The acquisition module is used to acquire data processing parameters and target scenarios for data processing. The first determining module is used to determine whether the target scene is a first determining result of the first scene; The second determining module is used to determine a first link library and a second link library corresponding to the target scenario when the first determining result is that the target scenario is the first scenario, wherein the second link library is constructed based on the first link library; The third determining module is used to determine the parameter similarity index between the data processing parameters and multiple reference processing parameters, respectively; The fourth determining module is used to determine whether there is a similarity determination result among multiple parameter similarity indices where at least one parameter similarity index is greater than the parameter similarity threshold, wherein the multiple parameter similarity indices correspond one-to-one with the multiple reference processing parameters; The fifth determining module is used to determine the data processing result corresponding to the data processing parameters based on the data processing parameters, the first link library, the second link library, and the similarity determining result.

9. An electronic device, characterized in that, include: processor; Memory used to store the processor's executable instructions; The processor is configured to execute the instructions to implement the data processing method as described in any one of claims 1 to 7.

10. A computer-readable storage medium, characterized in that, When the instructions in the computer-readable storage medium are executed by the processor of the electronic device, the electronic device is able to perform the data processing method as described in any one of claims 1 to 7.

Images