Data processing risk detection method and device, electronic equipment and storage medium

Abstract

The invention discloses a data processing risk detection method and device, electronic equipment and a storage medium. The method comprises the steps of obtaining a target processing script in response to triggering of a data processing risk detection event; wherein the target processing script is a script file used for processing the at least two data tables; determining table structure information of each data table related to the target processing script and a field processing link between the data tables based on the target processing script; the field processing link comprises a logic conversion relationship between each target field in the target data table and a source field in the source data table; and judging whether a data processing risk exists when the data processing task is executed based on the target processing script based on the table structure information and the field processing link. According to the scheme, data processing risk detection can be automatically carried out, developers can be helped to systematically solve problems before production instead of depending on manual troubleshooting or post remedy, and the reliability of data processing is remarkably improved.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of data processing, and in particular to a data processing risk detection method and device, electronic equipment and a storage medium. BACKGROUND

[0002] In the current data processing flow, developers mainly rely on manual verification or test running to ensure the correctness of the processing script. However, manual verification needs to review the table building statements and processing statements in the processing script line by line, and determine whether the field types match and whether the operation logic can cause overflow or truncation through experience, which is inefficient and easy to miss data processing anomalies in complex scenarios (such as multi-field joint operation and extreme value overflow caused by aggregation functions). Test running relies on sample data or limited use cases, which is difficult to cover boundary conditions (such as addition operation when INT field takes maximum value, precision truncation after DECIMAL field extreme value multiplication and division, and super-long truncation after string concatenation). These problems are particularly prominent in scenarios with strict data precision requirements, therefore, an automatic and full-coverage data processing risk prediction scheme is urgently needed to block data quality problems from the source. SUMMARY

[0003] The present application provides a data processing risk detection method, device, electronic equipment and storage medium, which can automatically detect data processing risks.

[0004] According to an aspect of the present application, a data processing risk detection method is provided, comprising:

[0005] In response to a data processing risk detection event being triggered, a target processing script is obtained; wherein the target processing script is a script file used for processing at least two data tables;

[0006] Based on the target processing script, the table structure information of each data table involved in the target processing script and the field processing link between data tables are determined; the field processing link includes the logical conversion relationship between each target field in the target data table and the source field in the source data table;

[0007] Based on the table structure information and the field processing link, it is judged whether there is a data processing risk when executing a data processing task based on the target processing script.

[0008] According to another aspect of the present application, a data processing risk detection device is provided, comprising:

[0009] A target processing script acquisition module is configured to obtain a target processing script in response to a data processing risk detection event being triggered; wherein the target processing script is a script file used for processing at least two data tables;

[0010] determine table structure information of each data table involved in the target processing script and field processing link between data tables based on the target processing script; the field processing link includes logical conversion relationship between each target field in the target data table and source field in the source data table;

[0011] a data processing risk judgment module configured to judge whether there is a data processing risk when executing a data processing task based on the target processing script based on the table structure information and the field processing link.

[0012] According to another aspect of the present application, an electronic device is provided, which comprises:

[0013] at least one processor; and

[0014] a memory connected to the at least one processor in communication; wherein,

[0015] the memory stores a computer program executable by the at least one processor, and the computer program is executed by the at least one processor to enable the at least one processor to execute the data processing risk detection method according to any one of the embodiments of the present application.

[0016] According to another aspect of the present application, a computer readable storage medium is provided, which stores computer instructions for enabling a processor to implement the data processing risk detection method according to any one of the embodiments of the present application when executed.

[0017] The data processing risk detection scheme according to the embodiments of the present application, in response to a data processing risk detection event being triggered, acquires a target processing script; wherein the target processing script is a script file used for processing at least two data tables; determines table structure information of each data table involved in the target processing script and field processing link between data tables based on the target processing script; the field processing link includes logical conversion relationship between each target field in the target data table and source field in the source data table; judges whether there is a data processing risk when executing a data processing task based on the target processing script based on the table structure information and the field processing link. Through the technical scheme provided by the embodiments of the present application, automatic data processing risk detection can be realized, which helps developers to solve problems systematically before production, rather than relying on manual troubleshooting or after-the-fact remediation, and significantly improves the reliability of data processing.

[0018] It should be understood that the content described in this part is not intended to identify key or important features of the embodiments of the present application, nor to limit the scope of the present application. Other features of the present application will become apparent from the following description. Attached Figure Description

[0019] To more clearly illustrate the technical solutions in the embodiments of the present invention, the accompanying drawings used in the description of the embodiments will be briefly introduced below. Obviously, the accompanying drawings described below are only some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0020] Figure 1 A flowchart of a data processing risk detection method provided in an embodiment of the present invention;

[0021] Figure 2 This is a schematic diagram of the structure of a data processing risk detection device provided in an embodiment of the present invention;

[0022] Figure 3 A schematic diagram of the structure of an electronic device for implementing the data processing risk detection method of this invention. Detailed Implementation

[0023] To enable those skilled in the art to better understand the present invention, the technical solutions of the present invention will be clearly and completely described below with reference to the accompanying drawings of the embodiments of the present invention. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort should fall within the scope of protection of the present invention.

[0024] It should be noted that the terms "first," "second," etc., in the specification, claims, and accompanying drawings of this invention are used to distinguish similar objects and are not necessarily used to describe a specific order or sequence. It should be understood that such data can be interchanged where appropriate so that the embodiments of the invention described herein can be implemented in orders other than those illustrated or described herein. Furthermore, the terms "comprising" and "having," and any variations thereof, are intended to cover a non-exclusive inclusion; for example, a process, method, system, product, or apparatus that comprises a series of steps or units is not necessarily limited to those steps or units explicitly listed, but may include other steps or units not explicitly listed or inherent to such processes, methods, products, or apparatus.

[0025] Figure 1 This is a flowchart illustrating a data processing risk detection method according to an embodiment of the present invention. This embodiment is applicable to situations requiring data processing risk detection. The method can be executed by a data processing risk detection device, which can be implemented in hardware and / or software and can be configured in an electronic device.Figure 1 The method comprises the following steps:

[0026] S110, in response to a data processing risk detection event being triggered, obtaining a target processing script; wherein the target processing script is a script file used for processing at least two data tables.

[0027] In the embodiment of the present application, when a data processing risk detection request is received, it is determined that a data processing risk detection event is triggered. In response to the data processing risk detection event being triggered, a target processing script is obtained; wherein the target processing script is a script file used for processing at least two data tables. A data table is a table used for storing data in a database, which is composed of rows (records) and columns (fields). Each row represents a data record, and each column represents a field for storing data of a specific type. A data table is the basic unit of organizing and storing data in a database management system, which allows users to store, query and manage data in a structured manner. For example, a script storage path is determined, and the target processing script is read based on the script storage path.

[0028] S120, based on the target processing script, determining the table structure information of each data table involved in the target processing script and the field processing link between the data tables; the field processing link contains the logical conversion relationship between each target field in the target data table and the source field in the source data table.

[0029] In the embodiment of the present application, the target processing script is analyzed to determine the table structure information of each data table involved in the target processing script and the field processing link between the data tables. For example, the target processing script is input into a pre-trained script analysis model, and the table structure information of each data table involved in the target processing script and the field processing link between the data tables are determined according to the output result of the script analysis model. The table structure information can include the names, data types and precision of all fields of the data table and other related information. The field processing link contains the logical conversion relationship between each target field in the target data table and the source field in the source data table.

[0030] Optionally, determining the table structure information of each data table involved in the target processing script and the field processing link between the data tables based on the target processing script comprises: determining the table structure information of a corresponding data table based on each table creation statement in the target processing script; and determining the field processing link between the data tables involved in the target processing script based on the processing statements in the target processing script. In the embodiment of the present application, the target processing script includes the table creation statements of each data table involved and the processing statements for processing the data tables, and therefore each table creation statement in the target processing script can be analyzed respectively to determine the table structure information of the data table corresponding to the table creation statement, for example, the field name is a_decimal, the type is DECIMAL, the maximum integer bit is 8 bits, and the decimal bit is 2 bits, which are extracted from a_decimal DECIMAL(10, 2). The processing statements in the target processing script are analyzed to determine the field processing link between the data tables involved in the target processing script. The field processing link can also be referred to as field conversion logical relationship, which is used to explicitly indicate how each target field of the target data table is calculated from the source field in the source data table. For example, the target field B.b_int is calculated by A.a_int1+A.a_int2.

[0031] In S130, it is judged whether there is a data processing risk when the data processing task is executed based on the target processing script based on the table structure information and the field processing link.

[0032] In the embodiment of the present application, the table structure information and the field processing link are analyzed to judge whether there is a data processing risk when the data processing task is executed based on the target processing script. For example, the table structure information and the field processing link can be input into a pre-trained data processing risk analysis model to judge whether there is a data processing risk when the data processing task is executed based on the target processing script according to the output result of the data processing risk analysis model.

[0033] Optionally, judging whether there is a data processing risk when the data processing task is executed based on the target processing script based on the table structure information and the field processing link comprises: for each target field in the target data table, determining a source field corresponding to the target field based on the field processing link; determining source data information of the source field based on table structure information corresponding to a source data table where the source field is located, and determining theoretical data information of the target field based on table structure information corresponding to the target data table; determining real data information of the target field based on the field processing link corresponding to the target field and the source data information; and judging whether there is a data processing risk when the data processing task is executed based on the target processing script based on the real data information and the theoretical data information. This arrangement has the advantage that it can accurately judge whether there is a data processing risk when the data processing task is executed based on the target processing script.

[0034] In the embodiment of the application, since the field processing link is used to reflect the logical conversion relationship between each target field in the target data table and the source field in the source data table, the field processing link can be analyzed for each target field in the target data table to determine the source field that has a logical conversion relationship with the target field. The source field can be one or multiple. The data table where the source field is located is taken as the source data table, and the source data information of the source field is determined based on the table structure information corresponding to the source data table, and the theoretical data information of the target field is determined based on the table structure information corresponding to the target data table. The theoretical data information can be understood as the data information of the target field declared in the table creation statement corresponding to the target data table. Then, the real data information of the target field is determined based on the field processing link corresponding to the target field and the source data information. The real data information can be understood as the actual data information of the target field after the source field is processed according to the processing expression corresponding to the field processing link to be converted into the target field. The data information can include data range, data precision, string length and other related information. It can be understood that when the data information is the data range, the source data information is the source data range, the theoretical data information is the theoretical data range, and the real data information is the real data range; when the data information is the data precision, the source data information is the source data precision, the theoretical data information is the theoretical data precision, and the real data information is the real data precision; and when the data information is the string length, the source data information is the source string length, the theoretical data information is the theoretical string length, and the real data information is the real string length. The real data information and the theoretical data information are compared, and whether there is a data processing risk when the data processing task is executed based on the target processing script is judged based on the comparison result.

[0035] Optionally, the data processing risk includes a data overflow risk, a precision loss risk, and a string truncation risk; and based on the real data information and the theoretical data information, determining whether there is a data processing risk when the data processing task is executed based on the target processing script, includes: if the data information is a data range, when the real data range exceeds the theoretical data range, it is determined that there is a data overflow risk when the data processing task is executed based on the target processing script; if the data information is data precision, when the real data precision is greater than the theoretical data precision, it is determined that there is a precision loss risk when the data processing task is executed based on the target processing script; and if the data information is a string length, when the real string length is greater than the theoretical string length, it is determined that there is a string truncation risk when the data processing task is executed based on the target processing script.

[0036] In the embodiment of the application, if the data information is a data range, the source data information is a source data range, the theoretical data information is a theoretical data range, and the real data information is a real data range, when the real data range exceeds the theoretical data range, it is determined that there is a data overflow risk when the data processing task is executed based on the target processing script. It can be understood that the source data range of the source field is determined based on the table structure information corresponding to the source data table where the source field is located, and the theoretical data range of the target field is determined based on the table structure information corresponding to the target data table. For example, the source fields include a_int1 and a_int2, and the source data range of the two source fields is-2^31~2^31-1, and the target field is b_int, and the theoretical data range of the target field is-2^31~2^31-1. The real data range of the target field is determined based on the logical conversion relationship (i.e. field processing link) between the target field and the source field and the source data range, for example, the logical conversion relationship between the target field and the source field is b_int=a_int1+a_int2, and the real data range of the target field is (-4294967294, 4294967294). It is determined whether the real data range exceeds the theoretical data range, if yes, it is determined that there is a data overflow risk when the data processing task is executed based on the target processing script, otherwise, it is determined that there is no data overflow risk when the data processing task is executed based on the target processing script. For example, for the target field (b_int) and the source field (a_int1 and a_int2) with the logical conversion relationship b_int=a_int1+a_int2, there is a data overflow risk when the data processing task is executed.

[0037] In the embodiment of the present application, if the data information is data precision, the source data information is source data precision, the theoretical data information is theoretical data precision, and the real data information is real data precision. When the real data precision exceeds the theoretical data precision, it is determined that there is a data overflow risk when the data processing task is executed based on the target processing script. It can be understood that the source data precision of the source field is determined based on the table structure information corresponding to the source data table where the source field is located, and the theoretical data precision of the target field is determined based on the table structure information corresponding to the target data table. The real data precision of the target field is determined based on the logical conversion relationship (i.e., the field processing link) between the target field and the source field and the source data precision. It is judged whether the real data precision exceeds the theoretical data precision. If yes, it is determined that there is a precision loss risk when the data processing task is executed based on the target processing script. Otherwise, it is determined that there is no precision loss risk when the data processing task is executed based on the target processing script.

[0038] Optionally, if the data information is data precision, before determining the real data information of the target field based on the field processing link corresponding to the target field and the source data information, the method further includes: judging whether the theoretical data precision is greater than the source data precision; and determining the real data precision of the target field based on the field processing link corresponding to the target field and the source data information, including: when the theoretical data precision is greater than the source data precision, determining the real data precision of the target field based on the field processing link corresponding to the target field and the source data information. Optionally, the method further includes: if the theoretical data precision is less than the source data precision, determining that there is a precision loss risk when the data processing task is executed based on the target processing script.

[0039] For example, assuming that the source field is DECIMAL (a, b), the source data precision is the number of decimal places b, the target field is DECIMAL (c, d), and the theoretical data precision is the number of decimal places d. It is judged whether d is greater than or equal to b and whether c-d is greater than or equal to a-b. If d < b (i.e., the theoretical data precision is less than the source data precision), it is determined that there is a precision loss risk when the data processing task is executed based on the target processing script. If c-d < a-b, it is determined that there is a data overflow risk when the data processing task is executed based on the target processing script. In the embodiment of the present application, if the theoretical data precision is greater than the source data precision, the real data precision of the target field is determined based on the field processing link corresponding to the target field and the source data information, and it is judged whether the real data precision is greater than the theoretical data precision. If yes, it is determined that there is a precision loss risk when the data processing task is executed based on the target processing script.

[0040] For example, the source field is DECIMAL(a, b) and DECIMAL(c, d), the target field is DECIMAL(e, f), if the logical conversion relationship between the target field and the source field is DECIMAL(e, f) = DECIMAL(a, b) / DECIMAL(c, d), the real data precision of the target field is max(6, b + (c - d) + 1) decimal places, and if f < max(6, b + (c - d) + 1), it is determined that there is a risk of precision loss when the data processing task is executed based on the target processing script. If the logical conversion relationship between the target field and the source field is DECIMAL(e, f) = DECIMAL(a, b) * DECIMAL(c, d), the result of DECIMAL(a, b) * DECIMAL(c, d) should be DECIMAL(a + c, b + d), and the real data precision of the target field is b + d. Therefore, if e - f < a + c - (b + d), it is determined that there is a risk of data overflow when the data processing task is executed based on the target processing script, and if f < b + d, it is determined that there is a risk of precision loss when the data processing task is executed based on the target processing script. If the logical conversion relationship between the target field and the source field is DECIMAL(a, b) ± DECIMAL(c, d), the result of DECIMAL(a, b) ± DECIMAL(c, d) should be DECIMAL(max(a - b, c - d) + max(b, d) + 1, max(b, d)), and the real data precision of the target field is max(b, d). Therefore, if e - f < max(a - b, c - d) + 1, it is determined that there is a risk of data overflow when the data processing task is executed based on the target processing script, and if f < max(b, d), it is determined that there is a risk of precision loss when the data processing task is executed based on the target processing script.

[0041] In the embodiment of the present application, if the data information is a string length, the source data information is a source string length, the theoretical data information is a theoretical string length, and the real data information is a real string length, when the real string length is greater than the theoretical string length, it is determined that there is a string truncation risk when the data processing task is executed based on the target processing script. It can be understood that the source string length of the source field is determined based on the table structure information corresponding to the source data table where the source field is located, and the theoretical string length of the target field is determined based on the table structure information corresponding to the target data table. The real string length of the target field is determined based on the logical conversion relationship (i.e. field processing link) between the target field and the source field and the source string length. For example, the logical conversion relationship between the target field and the source field is e = CONCAT (a.b), and the real string length of the target field is LEN (a) + LEN (b). For example, e = CONCAT (a_varchar, '_suffix'), and the real string length of the target field is LEN (a_varchar) + 7; the logical conversion relationship between the target field and the source field is e = REPEAT (c, d), and the real string length of the target field is LEN (c) * d. For example, e = REPEAT (a_varchar, 3), and the real string length of the target field is LEN (a_varchar) * 3. It is determined whether the real string length exceeds the theoretical string length. If yes, it is determined that there is a string truncation risk when the data processing task is executed based on the target processing script, otherwise, it is determined that there is no string truncation risk when the data processing task is executed based on the target processing script. For example, the logical conversion relationship between the target field and the source field is c = CONCAT (a.b), and the real string length of the target field is LEN (a) + LEN (b). If the real string length LEN (a) + LEN (b) is greater than the theoretical string length LEN (c), it is determined that there is a string truncation risk when the data processing task is executed based on the target processing script.

[0042] Optionally, when the logical conversion relationship between the target field and the source field is realized by an aggregation function (such as SUM()OVER()), the real data range of the target field is calculated based on the aggregation function and the source data range of the source field, and it is determined whether the real data range exceeds the theoretical data range of the target field. If yes, it is determined that there is an aggregation risk when the data processing task is executed based on the target processing script. The aggregation risk is a special case of data overflow risk.

[0043] The data processing risk detection method of the embodiment of the present application, in response to a data processing risk detection event being triggered, acquires a target processing script; wherein the target processing script is a script file used for processing at least two data tables; based on the target processing script, table structure information of each data table involved in the target processing script and field processing links between data tables are determined; the field processing links contain logical conversion relationships between each target field in the target data table and the source field in the source data table; based on the table structure information and the field processing links, it is judged whether there is a data processing risk when a data processing task is executed based on the target processing script. Through the technical scheme provided by the embodiment of the present application, automatic data processing risk detection can be realized, which helps developers to solve problems systematically before production, rather than relying on manual troubleshooting or after-the-fact remediation, and significantly improves the reliability of data processing.

[0044] In some embodiments, after judging whether there is a data processing risk when a data processing task is executed based on the target processing script based on the table structure information and the field processing links, it further includes: if it is determined that there is a data processing risk when a data processing task is executed based on the target processing script, data processing warning is performed, and the data processing risk type is determined, the target processing script is processed based on the risk defense measures corresponding to the data processing risk type. The advantage of such setting is that when it is detected that there is a data processing risk, appropriate risk defense measures are taken to process the target processing script in time to eliminate the data processing risk as much as possible.

[0045] In the embodiment of the present application, if it is determined that there is a data processing risk when the data processing task is executed based on the target processing script, a data processing warning is performed, for example, data processing warning information is sent to a data administrator, or a data processing risk prompt is performed in the form of audio. Optionally, the data processing risk type can be further determined, wherein the data processing risk type includes data overflow risk, precision loss risk and string truncation risk, and the target processing script is processed based on the risk defense measures corresponding to the data processing risk type. For example, if the data processing risk is data overflow risk, the theoretical data range corresponding to the target field in the target processing script is adjusted, so that the adjusted theoretical data range is greater than the corresponding real data range, for example, the data type of the target field is changed from INT to BIGINT; if the data processing risk is precision loss risk, the theoretical data precision corresponding to the target field in the target processing script is adjusted, so that the adjusted theoretical data precision is greater than the corresponding real data precision, for example, the target field is adjusted from DECIMAL(10, 3) to DECIMAL(10, 5), or the theoretical data precision corresponding to the target field in the target processing script is forcibly controlled to be a preset precision threshold, for example, the theoretical data precision corresponding to the target field is forcibly controlled to be 3 decimal places by using the control instruction ROUND(a_decimal*2.5, 3); if the data processing risk is string truncation risk, the theoretical string length corresponding to the target field in the target processing script is adjusted, so that the adjusted theoretical string length is greater than the corresponding real string length, for example, the type of the target field is adjusted from VARCHAR(20) to VARCHAR(30), or the theoretical string length corresponding to the target field in the target processing script is forcibly controlled to be a preset string length threshold, for example, the theoretical string length corresponding to the target field is forcibly controlled to be 25 by using the control instruction SUBSTR(CONCAT(a_varchar,'_suffix'),1,25).

[0046] Figure 2 A structural schematic diagram of a data processing risk detection device provided in the embodiment of the present application is shown in FIG. 1. As shown in the figure, the device comprises: Figure 2

[0047] A target processing script acquisition module 210 is configured to acquire a target processing script in response to a data processing risk detection event being triggered, wherein the target processing script is a script file used for processing at least two data tables;

[0048] A field processing link determination module 220 is configured to determine table structure information of each data table involved in the target processing script and a field processing link between data tables based on the target processing script, wherein the field processing link comprises a logical conversion relationship between each target field in the target data table and a source field in the source data table.​

[0049] The data processing risk judgment module 230 is configured to determine whether there is a data processing risk when the data processing task is executed based on the target processing script based on the table structure information and the field processing link.

[0050] Optionally, the field processing link determination module is configured to:

[0051] determine the table structure information of the corresponding data table based on each table creation statement in the target processing script;

[0052] determine the field processing link between the data tables involved in the target processing script based on the processing statements in the target processing script.

[0053] Optionally, the data processing risk judgment module includes:

[0054] The source field determination unit is configured to determine, for each target field in the target data table, a source field corresponding to the target field based on the field processing link;

[0055] The theoretical data information determination unit is configured to determine source data information of the source field based on the table structure information of the source data table corresponding to the source field, and determine theoretical data information of the target field based on the table structure information of the target data table;

[0056] The real data information determination unit is configured to determine real data information of the target field based on the field processing link corresponding to the target field and the source data information;

[0057] The data processing risk judgment unit is configured to determine whether there is a data processing risk when the data processing task is executed based on the target processing script based on the real data information and the theoretical data information.

[0058] Optionally, the data processing risk includes a data overflow risk, a precision loss risk, and a string truncation risk.

[0059] The data processing risk judgment unit is configured to:

[0060] If the data information is a data range, when the real data range exceeds the theoretical data range, it is determined that there is a data overflow risk when the data processing task is executed based on the target processing script.

[0061] If the data information is data precision, when the real data precision is greater than the theoretical data precision, it is determined that there is a precision loss risk when the data processing task is executed based on the target processing script.

[0062] If the data information is a string length, when the real string length is greater than the theoretical string length, it is determined that there is a risk of string truncation when the data processing task is executed based on the target processing script.

[0063] Optionally, the data processing risk detection device further comprises:

[0064] The data precision comparison unit is configured to, if the data information is data precision, determine whether the theoretical data precision is greater than the source data precision before determining the real data information of the target field based on the field processing link corresponding to the target field and the source data information.

[0065] The real data information determination unit is configured to:

[0066] When the theoretical data precision is greater than the source data precision, determine the real data precision of the target field based on the field processing link corresponding to the target field and the source data information.

[0067] Optionally, the data processing risk detection device further comprises:

[0068] The precision loss risk determination unit is configured to, if the theoretical data precision is less than the source data precision, determine that there is a risk of precision loss when the data processing task is executed based on the target processing script.

[0069] Optionally, the data processing risk detection device further comprises:

[0070] The data processing warning module is configured to, after determining whether there is a data processing risk when the data processing task is executed based on the target processing script based on the table structure information and the field processing link, if it is determined that there is a data processing risk when the data processing task is executed based on the target processing script, perform data processing warning, determine a data processing risk type, and process the target processing script based on a risk prevention measure corresponding to the data processing risk type.

[0071] The data processing risk detection device provided in the embodiments of the present application can execute the data processing risk detection method provided in any of the embodiments of the present application, and has the corresponding functional modules and beneficial effects of the execution method.

[0072] Figure 3A schematic diagram of an electronic device 10 that can be used to implement embodiments of the present invention is shown. The electronic device is intended to represent various forms of digital computers, such as laptop computers, desktop computers, workstations, personal digital assistants, servers, blade servers, mainframe computers, and other suitable computers. The electronic device can also represent various forms of mobile devices, such as personal digital processors, cellular phones, smartphones, wearable devices (e.g., helmets, glasses, watches, etc.), and other similar computing devices. The components shown herein, their connections and relationships, and their functions are merely illustrative and are not intended to limit the implementation of the invention described and / or claimed herein.

[0073] like Figure 3 As shown, the electronic device 10 includes at least one processor 11 and a memory, such as a read-only memory (ROM) 12 or a random access memory (RAM) 13, communicatively connected to the at least one processor 11. The memory stores computer programs executable by the at least one processor. The processor 11 can perform various appropriate actions and processes based on the computer program stored in the ROM 12 or loaded from storage unit 18 into the RAM 13. The RAM 13 may also store various programs and data required for the operation of the electronic device 10. The processor 11, ROM 12, and RAM 13 are interconnected via a bus 14. An input / output (I / O) interface 15 is also connected to the bus 14.

[0074] Multiple components in electronic device 10 are connected to I / O interface 15, including: input unit 16, such as keyboard, mouse, etc.; output unit 17, such as various types of displays, speakers, etc.; storage unit 18, such as disk, optical disk, etc.; and communication unit 19, such as network card, modem, wireless transceiver, etc. Communication unit 19 allows electronic device 10 to exchange information / data with other devices through computer networks such as the Internet and / or various telecommunications networks.

[0075] Processor 11 can be a variety of general-purpose and / or special-purpose processing components with processing and computing capabilities. Some examples of processor 11 include, but are not limited to, a central processing unit (CPU), a graphics processing unit (GPU), various special-purpose artificial intelligence (AI) computing chips, various processors running machine learning model algorithms, a digital signal processor (DSP), and any suitable processor, controller, microcontroller, etc. Processor 11 performs the various methods and processes described above, such as data processing risk detection methods.

[0076] In particular, the processes described above with reference to the flowcharts can be implemented as a computer software program in accordance with embodiments of the present application. For example, embodiments of the present application include a computer program product comprising a computer program carried on a non-transitory computer-readable medium, the computer program containing program code for executing the methods illustrated by the flowcharts. In such embodiments, the computer program can be downloaded and installed from a network via the communication unit 19, or installed from the storage unit 18, or installed from the ROM 12. When the computer program is executed by the processor 11, the above-described functions defined in the methods of embodiments of the present application are performed.

[0077] In some embodiments, the data processing risk detection method can be implemented as a computer program tangibly embodied in a computer readable storage medium, such as the storage unit 18. In some embodiments, part or all of the computer program can be loaded onto and / or installed on the electronic device 10 via the ROM 12 and / or the communication unit 19. When the computer program is loaded onto the RAM 13 and executed by the processor 11, one or more steps of the data processing risk detection method described above can be performed. Alternatively, in other embodiments, the processor 11 can be configured to perform the data processing risk detection method by any other suitable means, such as by means of firmware.

[0078] The various implementations of the systems and techniques described above can be realized in digital electronic circuitry, integrated circuitry, a field programmable gate array (FPGA), an application specific integrated circuit (ASIC), a system on a chip (SOC), a complex programmable logic device (CPLD), computer hardware, firmware, software, and / or combinations thereof. These various implementations can include implementation in one or more computer programs that are executable and / or interpretable on a programmable system including at least one programmable processor, which can be special or general purpose, coupled to receive data and instructions from, and to transmit data and instructions to, a storage system, at least one input device, and at least one output device.

[0079] Computer programs used to implement the methods of the present application can be written in any combination of one or more programming languages. These computer programs can be provided to a processor of a general purpose computer, special purpose computer, or other programmable data processing apparatus to produce a machine, such that the computer program, when executed by the processor of the machine, causes the machine to implement the functions / acts specified in the flowcharts and / or block diagrams. The computer program can be executed entirely on a machine, partially on a machine, partially on a machine and partially on a remote machine or entirely on a remote machine or server.

[0080] In the context of the present application, a computer-readable storage medium can be a tangible medium that can contain or store computer programs for use by or in connection with an instruction execution system, apparatus, or device. Computer-readable storage media can include, but are not limited to, electronic, magnetic, optical, electromagnetic, infrared, or semiconductor systems, apparatus, or devices, or any suitable combination of the foregoing. Alternatively, a computer-readable storage medium can be a machine-readable signal medium. More specific examples of a machine-readable storage medium will include one or more lines of a program of instructions in a transitory signal, a portable computer diskette, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or Flash memory), an optical fiber, a portable compact disc read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the foregoing.

[0081] To provide for interaction with a user, the systems and techniques described here can be implemented on an electronic device having a display device (e.g., a CRT (cathode ray tube) or LCD (liquid crystal display) monitor) for displaying information to the user and a keyboard and a pointing device (e.g., a mouse or a trackball) by which the user can provide input to the electronic device. Other kinds of devices can be used to provide for interaction with a user as well; for example, feedback provided to the user can be any form of sensory feedback (e.g., visual feedback, auditory feedback, or tactile feedback); and input from the user can be received in any form, including acoustic, speech, or tactile input.

[0082] The systems and techniques described here can be implemented in a computing system that includes a back end component (e.g., as a data server), or that includes a middleware component (e.g., an application server), or that includes a front end component (e.g., a user computer having a graphical user interface or a Web browser through which a user can interact with an implementation of the systems and techniques described here), or any combination of such back end, middleware, or front end components. The components of the system can be interconnected by any form or medium of digital data communication (e.g., a communication network). Examples of communication networks include a local area network (LAN), a wide area network (WAN), blockchain network, and the Internet.

[0083] The computing system can include clients and servers. A client and server are generally remote from each other and typically interact through a communication network. The relationship of client and server arises by virtue of computer programs running on the respective computers and having a client-server relationship to each other. The server can be a cloud server, also known as a cloud computing server or cloud host, which is a host product in the cloud computing service system, to solve the defects of large management difficulty and weak business scalability in traditional physical host and VPS service.

[0084] It should be understood that the various forms of flow shown above can be used to reorder, add or delete steps. For example, each step described in the present application can be executed in parallel, sequentially or in a different order, as long as the desired results of the technical solutions of the present application can be achieved, which is not limited herein.

[0085] The above detailed description does not constitute a limitation on the scope of protection of the present application. Those skilled in the art should understand that various modifications, combinations, sub-combinations and substitutions can be made according to design requirements and other factors. Any modifications, equivalent replacements and improvements made within the spirit and principles of the present application shall be included in the scope of protection of the present application.

Claims

1. A data processing risk detection method, characterized by, The method comprises the following steps: In response to a data processing risk detection event being triggered, a target processing script is obtained; wherein the target processing script is a script file used for processing at least two data tables; Based on the target processing script, table structure information of each data table involved in the target processing script and field processing links between data tables are determined; the field processing links include logical conversion relationships between each target field in the target data table and a source field in a source data table; Based on the table structure information and the field processing links, it is determined whether there is a data processing risk when a data processing task is executed based on the target processing script.

2. The method of claim 1, wherein, Based on the target processing script, table structure information of each data table involved in the target processing script and field processing links between data tables are determined, comprising: Based on each table creation statement in the target processing script, the table structure information of the corresponding data table is determined; Based on the processing statements in the target processing script, the field processing links between the data tables involved in the target processing script are determined.

3. The method of claim 1, wherein, Based on the table structure information and the field processing links, it is determined whether there is a data processing risk when a data processing task is executed based on the target processing script, comprising: For each target field in the target data table, based on the field processing links, a source field corresponding to the target field is determined; Based on the table structure information of the source data table corresponding to the source field, the source data information of the source field is determined, and based on the table structure information of the target data table, the theoretical data information of the target field is determined; Based on the field processing links corresponding to the target field and the source data information, the real data information of the target field is determined; Based on the real data information and the theoretical data information, it is determined whether there is a data processing risk when a data processing task is executed based on the target processing script.

4. The method of claim 3, wherein, The data processing risk includes data overflow risk, precision loss risk, and string truncation risk; Based on the real data information and the theoretical data information, it is determined whether there is a data processing risk when a data processing task is executed based on the target processing script, comprising: If the data information is a data range, when the real data range exceeds the theoretical data range, it is determined that there is a data overflow risk when a data processing task is executed based on the target processing script; If the data information is data precision, when the real data precision is greater than the theoretical data precision, it is determined that there is a precision loss risk when a data processing task is executed based on the target processing script; If the data information is a string length, when the real string length is greater than the theoretical string length, it is determined that there is a string truncation risk when a data processing task is executed based on the target processing script.

5. The method of claim 4, wherein, If the data information is data precision, before determining the real data information of the target field based on the field processing links corresponding to the target field and the source data information, it further comprises: It is determined whether the theoretical data precision is greater than the source data precision; Based on the field processing links corresponding to the target field and the source data information, the real data information of the target field is determined, comprising: When the theoretical data precision is greater than the source data precision, a real data precision of the target field is determined based on the field processing link corresponding to the target field and the source data information.

6. The method of claim 5, wherein, Further comprising: If the theoretical data precision is less than the source data precision, it is determined that there is a risk of precision loss when data processing tasks are performed based on the target processing script.

7. The method of claim 1, wherein, After determining whether there is a data processing risk when data processing tasks are performed based on the target processing script based on the table structure information and the field processing link, further comprising: If it is determined that there is a data processing risk when data processing tasks are performed based on the target processing script, a data processing warning is given, and a data processing risk type is determined, and the target processing script is processed based on a risk prevention measure corresponding to the data processing risk type.

8. A data processing risk detection apparatus, characterized by, Comprising: A target processing script acquisition module is configured to acquire a target processing script in response to a data processing risk detection event being triggered, wherein the target processing script is a script file used for processing at least two data tables; A field processing link determination module is configured to determine table structure information of each data table involved in the target processing script and a field processing link between data tables based on the target processing script, wherein the field processing link includes a logical conversion relationship between each target field in a target data table and a source field in a source data table. A data processing risk judgment module is configured to determine whether there is a data processing risk when data processing tasks are performed based on the target processing script based on the table structure information and the field processing link.

9. An electronic device, comprising: The electronic device comprises: At least one processor; and The memory is in communication connection with the at least one processor; wherein The memory stores a computer program that can be executed by the at least one processor, and the computer program is executed by the at least one processor to enable the at least one processor to execute the data processing risk detection method of any one of claims 1-7.

10. A computer-readable storage medium, characterized in that, The computer readable storage medium stores computer instructions for enabling the processor to execute the data processing risk detection method of any one of claims 1-7 when executed.

Images