Data debugging method, device, medium, equipment and program product

Abstract

The present application provides a data debugging method, device, medium, equipment and program product. The data debugging method provided in the embodiment of the present application obtains a target data source, and then generates a data source to be tested according to the target data source and a preset data mapping relationship, thereby converting a target data source of a data type that is not originally supported by the debugging system into a data source of an internal preset type of the debugging system, so that the debugging system can use the data source to be tested to debug the debugging logic program in the debugging system, thereby enriching the data source format types supported by the debugging system during debugging to meet the needs of the diversity of data source formats of the current actual platform.

Description

Technical Field

[0001] The present application relates to the field of Internet technology, and in particular to a data debugging method, device, medium, equipment and program product. Background Art

[0002] In recent years, with the development of big data technology, a variety of big data computing engine frameworks have emerged, such as Storm, Spark, Flink and other excellent computing engine frameworks.

[0003] In the field of big data, Structured Query Language (SQL) is the user-oriented Application Programming Interface (API) layer. However, most current open source Flink frameworks do not provide functional support for SQL debugging.

[0004] For some platforms (for example, Alibaba Cloud's Flink SQL platform), although SQL debugging is supported, it only supports debugging using structured data sources. It can be seen that the supported formats are relatively simple and cannot meet the diverse requirements of the data source formats of the current actual platforms. Summary of the invention

[0005] The embodiments of the present application provide a data debugging method, apparatus, medium, device and program product to meet the diverse requirements for actual platform data source formats.

[0006] In a first aspect, an embodiment of the present application provides a data debugging method, which is applied to a debugging system, wherein the debugging system is used to debug an input logic program using a data source of a preset type, and the method includes:

[0007] Acquire a target data source, wherein the target data source includes data for debugging the logic program to be debugged, wherein the data format of the target data source is a target type, and the target type is different from the preset type;

[0008] Generate a data source to be tested according to the target data source and a preset data mapping relationship, wherein the preset data mapping is used to map the data of the target type to the data of the preset type;

[0009] The logic program to be debugged is debugged in the debugging system using the data source to be tested.

[0010] In a possible design, generating a data source to be tested according to the target data source and a preset data mapping relationship includes:

[0011] Acquire target mode information of the target data source, wherein the target mode information includes target data organization information and target data structure information;

[0012] Acquire target data from the target data source according to the target mode information, wherein the target data is a data item required for debugging the logic program to be debugged;

[0013] The preset data mapping relationship is used to convert each field in the target data into a field of the preset type to generate the data source to be tested.

[0014] In a possible design, generating the data source to be tested includes:

[0015] Each field of the preset type is encapsulated through a target object to generate corresponding test data, and the data source to be tested includes the test data corresponding to all fields of the preset type.

[0016] In a possible design, debugging the logic program to be debugged in the debugging system using the data source to be tested includes:

[0017] Calling the test project main class of the debugging system to generate a test task;

[0018] Registering a test resource to process the data source to be tested using the test resource to generate a corresponding data list;

[0019] Execute the logic program to be debugged and call the data list to output the test result;

[0020] The test results are displayed in the form of test logs.

[0021] In a possible design, after generating the data source to be tested according to the target data source and the preset data mapping relationship, the method further includes:

[0022] Displaying the data source to be tested on a debugging page of the debugging system by using a preset display data type;

[0023] Acquire, in the debugging page, a modification instruction for at least one field in the data source to be tested;

[0024] The corresponding fields are modified according to the modification instruction, and the data source to be tested is updated.

[0025] In a possible design, after displaying the data source to be tested on the debugging page of the debugging system by using a preset display data type, the method further includes:

[0026] Generate a data deserializer according to the preset display data type and the target type to convert the updated to-be-tested data source of the preset display data type into the to-be-tested data source of the target type;

[0027] The data source to be tested in the debugging system is generated according to the target type and the converted data source to be tested.

[0028] In a possible design, before acquiring the target data source, the method further includes:

[0029] In the debugging page, obtain the logic program to be debugged;

[0030] It is determined that no data source required for debugging the logic program to be debugged exists in the debugging system.

[0031] In a second aspect, an embodiment of the present application provides a data debugging device, including:

[0032] An acquisition module, used for acquiring a target data source, wherein the target data source includes data for debugging the logic program to be debugged, wherein the data format of the target data source is a target type, and the target type is different from a preset type in the debugging system;

[0033] A processing module, used to generate a data source to be tested according to the target data source and a preset data mapping relationship, wherein the preset data mapping is used to map the data of the target type to the data of the preset type;

[0034] The debugging module is used to debug the logic program to be debugged in the debugging system using the data source to be tested.

[0035] In a possible design, the processing module is specifically used to:

[0036] Acquire target mode information of the target data source, wherein the target mode information includes target data organization information and target data structure information;

[0037] Acquire target data from the target data source according to the target mode information, wherein the target data is a data item required for debugging the logic program to be debugged;

[0038] The preset data mapping relationship is used to convert each field in the target data into a field of the preset type to generate the data source to be tested.

[0039] In a possible design, the processing module is further used to encapsulate each field of the preset type through the target object to generate corresponding test data, and the data source to be tested includes the test data corresponding to all fields of the preset type.

[0040] In a possible design, the debugging module is specifically used to:

[0041] Calling the test project main class of the debugging system to generate a test task;

[0042] Registering a test resource to process the data source to be tested using the test resource to generate a corresponding data list;

[0043] Execute the logic program to be debugged and call the data list to output the test result;

[0044] The test results are displayed in the form of test logs.

[0045] In a possible design, the data debugging device further includes:

[0046] A display module, used to display the data source to be tested on a debugging page of the debugging system by using a preset display data type;

[0047] A modification module, used for obtaining, in the debugging page, a modification instruction for at least one field in the data source to be tested;

[0048] The processing module is used to modify the data of the corresponding field according to the modification instruction and update the data source to be tested.

[0049] In a possible design, the processing module is further used to generate a data deserializer according to the preset display data type and the target type to convert the updated data source to be tested of the preset display data type into the data source to be tested of the target type;

[0050] The processing module is further configured to generate the data source to be tested in the debugging system according to the target type and the converted data source to be tested.

[0051] In a possible design, the acquisition module is further used to acquire the logic program to be debugged in the debugging page;

[0052] The processing module is further used to determine that the data source required for debugging the logic program to be debugged does not exist in the debugging system.

[0053] In a third aspect, an embodiment of the present application further provides an electronic device, including:

[0054] Processor; and

[0055] a memory for storing a computer program for the processor;

[0056] Wherein, the processor is configured to implement any one of the data debugging methods in the first aspect by executing the computer program.

[0057] In a fourth aspect, an embodiment of the present application further provides a storage medium having a computer program stored thereon, which, when executed by a processor, implements any one of the data debugging methods in the first aspect.

[0058] In a fifth aspect, an embodiment of the present application further provides a computer program product, including a computer program, which implements any one of the data debugging methods in the first aspect when executed by a processor.

[0059] The embodiments of the present application provide a data debugging method, apparatus, medium, device and program product, which first obtain a target data source, and then generate a data source to be tested according to the target data source and a preset data mapping relationship, thereby converting a target data source of a data type that is not originally supported by a debugging system into a data source of an internal preset type of the debugging system, so that the debugging system can use the data source to be tested to debug the debugging logic program in the debugging system, thereby enriching the data source format types supported by the debugging system during debugging to meet the diversity of data source formats on current actual platforms. BRIEF DESCRIPTION OF THE DRAWINGS

[0060] In order to more clearly illustrate the embodiments of the present application or the technical solutions in the prior art, the following is a brief introduction to the drawings required for use in the embodiments or the description of the prior art. Obviously, the drawings described below are some embodiments of the present application. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative labor.

[0061] Figure 1 is an application scenario diagram of a data debugging method according to an exemplary embodiment of the present application;

[0062] Figure 2 is a flowchart of a data debugging method according to an exemplary embodiment of the present application;

[0063] Figure 3 is a flowchart of a data debugging method according to another exemplary embodiment of the present application;

[0064] Figure 4 is a flowchart of a data debugging method according to another exemplary embodiment of the present application;

[0065] Figure 5 is a structural schematic diagram of a data debugging device according to an exemplary embodiment of the present application;

[0066] Figure 6is a structural schematic diagram of a data debugging device according to another exemplary embodiment of the present application;

[0067] Figure 7 It is a schematic diagram of the structure of an electronic device according to an exemplary embodiment of the present application. DETAILED DESCRIPTION

[0068] In order to make the purpose, technical solution and advantages of the embodiments of the present application clearer, the technical solution in the embodiments of the present application will be clearly and completely described below in conjunction with the drawings in the embodiments of the present application. Obviously, the described embodiments are part of the embodiments of the present application, not all of the embodiments. Based on the embodiments in the present application, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of this application.

[0069] The terms "first", "second", "third", "fourth", etc. (if any) in the specification and claims of the present application and the above-mentioned drawings are used to distinguish similar objects, and are not necessarily used to describe a specific order or sequence. It should be understood that the data used in this way can be interchangeable where appropriate, so that the embodiments of the present application described herein can be implemented in an order other than those illustrated or described herein. In addition, the terms "including" and "having" and any of their variations are intended to cover non-exclusive inclusions, for example, a process, method, system, product or device that includes a series of steps or units is not necessarily limited to those steps or units that are clearly listed, but may include other steps or units that are not clearly listed or inherent to these processes, methods, products or devices.

[0070] In the field of big data, Structured Query Language (SQL) is the user-oriented Application Programming Interface (API) layer. SQL is a high-level non-procedural programming language that allows users to work on high-level data structures. It does not require users to specify the storage method for data, nor does it require users to understand the specific data storage method. Therefore, different database systems with completely different underlying structures can use the same structured query language as an interface for data input and management. Structured query language statements can be nested, which makes it extremely easy to use and powerful.

[0071] Because it is easy to use, in the traditional streaming computing field, such as Storm and Spark Streaming, some Function or Datastream APIs are provided, and users can write business logic in Java or Scala. Although this method is flexible, it has a certain programming threshold and is difficult to tune. As its version is constantly updated, the provided API will also have many incompatibilities.

[0072] Most of the current open source Flink frameworks do not yet provide functional support for SQL debugging. Some platforms (such as Alibaba Cloud's Flink SQL platform) support SQL debugging, but only support debugging using structured data sources. It can be seen that the supported formats are relatively simple and cannot meet the diverse requirements of data source formats on current actual platforms.

[0073] In addition, Flink SQL is a development language that conforms to standard SQL semantics and is designed by Flink real-time computing to simplify computing models and lower the threshold for users to use real-time computing. Currently, Spark and Flink are actively turning to the SQL platform.

[0074] In the Flink SQL platform, the need to constantly interact with the real upstream and downstream production environments to adjust the SQL statement logic to obtain the final correct SQL statement leads to the problem of high SQL development costs. This application aims to reduce the development costs caused by the above problems, so as to adapt to diverse data sources through the debugging solution proposed in this application, and quickly verify the accuracy of SQL statement logic, so as to shorten the development cycle and reduce costs.

[0075] Figure 1 1 is an application scenario diagram of a data debugging method according to an exemplary embodiment of the present application. Figure 1 As shown, the data debugging method provided in this embodiment is applied to a debugging system, wherein the debugging system can be a debugging platform based on various big data computing engine frameworks, for example, a Flink SQL platform. The debugging platform is mainly used to debug the input logic program using a preset type of data source, that is, to debug the SQL statement to be debugged using the data in the corresponding data source to obtain the final correct SQL statement.

[0076] When debugging, the logic program to be debugged can be written through the debugging interface, and then it is determined whether there is a data source required for debugging the logic program to be debugged in the debugging system. If it does not exist, it means that the data inside the debugging system cannot be debugged when the logic program to be debugged is lacking. At this time, it is necessary to generate the data source to be tested corresponding to the logic program to be debugged before the corresponding debugging work can be performed on the logic program to be debugged. The debugging system can first obtain the external target data source, and then convert the target data source into the data source to be tested supported by the debugging system according to the preset data mapping relationship, and then use the data source to be tested to debug the logic program to be debugged in the debugging system. Among them, it is worth noting that different data source data types correspond to different target mode information (for example: Schema, different data formats have their own specification definitions, Schema describes the fields and types of data, etc.), although the Schema of each data format is different, different Schemas will have a one-to-one mapping relationship with the internal data type of the Flink SQL platform. Therefore, the target data source can be converted into the data source to be tested supported by the debugging system according to the preset data mapping relationship.

[0077] Therefore, before debugging the logic program to be debugged in the debugging system, the target data sources of various data types are first converted into the data sources to be tested supported by the debugging system through the preset data mapping relationship, so that the debugging system can use the target data sources of various data types to test the corresponding logic program to be debugged, so that the debugging system can meet the debugging requirements for the current diverse data source formats.

[0078] Figure 2 FIG. 1 is a flow chart of a data debugging method according to an exemplary embodiment of the present application. Figure 2 As shown, the data debugging method provided in this embodiment includes:

[0079] Step 101: Obtain the target data source.

[0080] In this step, you can first input the logic program to be debugged through the debugging page. When the debugging operation is triggered, the relevant data source needs to be called.

[0081] At this time, if there is a data source required for debugging the logic program to be debugged in the debugging system, the test project main class of the debugging system and the data source that already exists in the debugging system are directly called to generate a test task. Then, the test resources are registered to use the test resources to process the data source to be tested and generate a corresponding data list. The logic program to be debugged is executed, and the data list is called to output the test results, and the test results are displayed in the form of a test log. Therefore, according to the test log, it is determined whether the logic program to be debugged needs to be modified. For example, according to the test results in the test log, it is determined whether the SQL statement needs to be modified.

[0082] If the data source required for debugging the logic program to be debugged does not exist in the debugging system, it is necessary to first obtain the target data source, and generate the corresponding data source to be tested according to the data in the target data source (for example, name, age, address, etc.), and then use the generated test data source for subsequent testing. The target data source includes the data for debugging the logic program to be debugged, and the data format of the target data source is the target type, while the debugging system supports the data source of the preset type, wherein the target type is different from the preset type.

[0083] Specifically, the product side of the SQL platform can call the Rest request, and then call the Spring service in the FlinkSQL platform for processing. The Spring service is mainly used to generate data, parse the Schema to obtain data types, etc. In addition, the generated data can be fed back to the product front end to display the data in Json format, which increases reading comprehension and facilitates data modification.

[0084] Step 102: Generate a data source to be tested according to the target data source and a preset data mapping relationship.

[0085] In this step, the data source to be tested may be generated according to the target data source obtained above and the preset data mapping relationship, wherein the preset data mapping is used to map the target type of data to the preset type of data.

[0086] Specifically, in this step, the Flink SQL platform can automatically generate the data source to be tested, which can provide support for target data sources of different formats (for example, JSON, AVRO, CSV format, etc.). When using the target data source, different data source data formats correspond to different schemas, and the schema needs to be correctly parsed to obtain the correct data type to generate the correct data. If you start with the schema of the data source format, you need to parse the schema of each format and then generate data separately, which is redundant, and each new format of data needs to parse the schema of the format, and then generate data, which has poor scalability. However, although the data format schemas are different, different schemas will have a one-to-one mapping relationship with the internal data types of the Flink SQL platform, that is, the preset types. Therefore, you can start with the internal data types of Flink SQL to generate data, and finally, splice it into JSON format data to display it to the user, so that it can be viewed and modified.

[0087] Step 103: Debug the logic program to be debugged in the debugging system using the data source to be tested.

[0088] After the data source to be tested is generated according to the target data source and the preset data mapping relationship, the data source to be tested can be used in the debugging system to debug the logic program to be debugged. Specifically, the customized TableSource and TableSink can be applied in the Flink SQL platform to complete the rapid verification of SQL logic, where TableSource and TableSink are the interfaces that the customized data source target source in the Flink SQL platform needs to implement.

[0089] In this embodiment, by acquiring the target data source, and then generating the data source to be tested according to the target data source and the preset data mapping relationship, the target data source of the data type that the debugging system originally does not support is converted into a data source of the internal preset type of the debugging system, so that the debugging system can use the data source to be tested to debug the debugged logic program in the debugging system, thereby enriching the data source format types supported by the debugging system during debugging to meet the diversity of data source formats of the current actual platform.

[0090] Figure 3 FIG. 1 is a flow chart of a data debugging method according to another exemplary embodiment of the present application. Figure 3 As shown, the data debugging method provided in this embodiment includes:

[0091] Step 201: Obtain target data source.

[0092] In this step, you can first input the logic program to be debugged through the debugging page. When the debugging operation is triggered, the relevant data source needs to be called.

[0093] At this time, if there is a data source required for debugging the logic program to be debugged in the debugging system, the test project main class of the debugging system and the data source that already exists in the debugging system are directly called to generate a test task. Then, the test resources are registered to use the test resources to process the data source to be tested and generate a corresponding data list. The logic program to be debugged is executed, and the data list is called to output the test results, and the test results are displayed in the form of a test log. Therefore, according to the test log, it is determined whether the logic program to be debugged needs to be modified. For example, according to the test results in the test log, it is determined whether the SQL statement needs to be modified.

[0094] If the data source required for debugging the logic program to be debugged does not exist in the debugging system, it is necessary to first obtain the target data source, and generate the corresponding data source to be tested according to the data in the target data source (for example, name, age, address, etc.), and then use the generated test data source for subsequent testing. The target data source includes the data for debugging the logic program to be debugged, and the data format of the target data source is the target type, while the debugging system supports the data source of the preset type, wherein the target type is different from the preset type.

[0095] Specifically, the product side of the SQL platform can call the Rest request, and then call the Spring service in the FlinkSQL platform for processing. The Spring service is mainly used to generate data, parse the Schema to obtain data types, etc. In addition, the generated data can be fed back to the product front end to display the data in Json format, which increases reading comprehension and facilitates data modification.

[0096] Step 202: Obtain target mode information of the target data source.

[0097] Specifically, the target schema information of the target data source can be obtained, wherein the target schema information includes target data organization information and target data structure information. It is worth understanding that the above target schema information can be a Schema, that is, different data formats have their own specification definitions, wherein the Schema describes information such as the fields and types of the data.

[0098] Step 203: Acquire target data from the target data source according to the target mode information.

[0099] Then, target data is acquired from a target data source according to the target mode information, wherein the target data is data items required for debugging the logic program to be debugged, such as name, age, address and other data items.

[0100] Step 204: Utilize the preset data mapping relationship to convert each field in the target data into a field of a preset type.

[0101] Provide support for target data sources in different formats (for example: JSON, AVRO, CSV format, etc.). When using the target data source, different data source data formats correspond to different schemas, and the schema needs to be correctly parsed to obtain the correct data type in order to generate the correct data.

[0102] The following is an explanation of the definitions of different Schemas using examples in JSON, AVRO, and CSV formats:

[0103] The first type, Schema in JSON format:

[0104] {

[0105] type:'object',

[0106] properties:

[0107] field1:{type:'integer'},

[0108] field2:{type:'string'},

[0109] field3:{type:'string',format:'date-time'}

[0110] }

[0111] }

[0112] The second type, Schema in CSV format:

[0113] field1:int&field2:String&field3:Timestamp

[0114] The third type, Schema in AVRO format:

[0115] {

[0116] "type":"record",

[0117] "name":"jdwdata",

[0118] "namespace":"com.jd.bdp.jdw.avro",

[0119] "fields":[

[0120] {"name":"field1","type":"int"},

[0121] {"name":"field2","type":"string"},

[0122] {"name":"field3","type":{"type":"long","logicalType":"timestamp-millis"}}

[0123] From the definitions of different schemas in the above JSON, AVRO, and CSV formats, it can be seen that the schemas of the three formats are different. However, they can all be mapped to the same type in Flink SQL, that is, the type corresponding to field 1 is integer, the type corresponding to field 2 is String, and the type corresponding to field 3 is Timestamp.

[0124] Step 205: Encapsulate each field of the preset type through the target object to generate corresponding test data.

[0125] Each field of a preset type may be encapsulated through a target object to generate corresponding test data, wherein the data source to be tested includes the test data corresponding to all fields of the preset types.

[0126] Specifically, after obtaining the type corresponding to each field in the specific Flink SQL platform, data can be generated according to the actual type and encapsulated into the Row object. Row encapsulates the type and name information of each field, as well as a generated test data. A class inside Flink is mainly used to encapsulate data. Finally, the data in the Row object is parsed, and test data in JSON format is generated according to the field information.

[0127] Step 206: Debug the logic program to be debugged in the debugging system using the data source to be tested.

[0128] In this embodiment, by obtaining the target data source, and then generating the data source to be tested according to the target data source and the preset data mapping relationship, the target data source of the data type that the debugging system does not originally support is converted into a data source of the internal preset type of the debugging system, so that the debugging system can use the data source to be tested to debug the debugging logic program in the debugging system, thereby enriching the data source format types supported by the debugging system during debugging to meet the needs of the diversity of data source formats of the current actual platform. Specifically, in this embodiment, the corresponding data types inside the corresponding Flink SQL platform are obtained by parsing the corresponding Schema of different data formats, so as to maintain the correct type matching, and data is generated according to the internal type of Flink SQL to reduce the interference of different external data format Schemas. Each piece of data generated in the end is encapsulated in a Row object, and can generate JSON format data based on field information and then feedback to the platform for the business party to view the data. If the data does not meet the requirements, it can be modified based on this data template to facilitate subsequent SQL logic debugging.

[0129] Figure 4 FIG. 1 is a flow chart of a data debugging method according to another exemplary embodiment of the present application. Figure 4 The data debugging method provided in this embodiment includes:

[0130] Step 301: Data debugging method.

[0131] In this step, you can first input the logic program to be debugged through the debugging page. When the debugging operation is triggered, the relevant data source needs to be called.

[0132] If the data source required for debugging the logic program to be debugged exists in the debugging system, the main class of the test project of the debugging system and the existing data source in the debugging system are directly called to generate a test task. Then, the test resources are registered to process the data source to be tested using the test resources and generate a corresponding data list. The logic program to be debugged is executed and the data list is called to output the test results, which are displayed in the form of a test log. Thus, according to the test log, it is determined whether the logic program to be debugged needs to be modified, for example, according to the test results in the test log, it is determined whether the SQL statement needs to be modified.

[0133] If the data source required for debugging the logic program to be debugged does not exist in the debugging system, it is necessary to first obtain the target data source, and generate the corresponding data source to be tested according to the data in the target data source (for example, name, age, address, etc.), and then use the generated test data source for subsequent testing. The target data source includes the data for debugging the logic program to be debugged, and the data format of the target data source is the target type, while the debugging system supports the data source of the preset type, wherein the target type is different from the preset type.

[0134] Step 302: Generate a data source to be tested according to the target data source and a preset data mapping relationship.

[0135] Specifically, the Schema of the target data source can be obtained, that is, different data formats have their own specification definitions, where the Schema describes the fields and types of the data. Then, the target data is obtained from the target data source according to the target schema information, where the target data is the data items required for debugging the logic program to be debugged, such as name, age, address and other data items.

[0136] Provide support for target data sources in different formats (for example, JSON, AVRO, CSV, etc.). When using the target data source, different data source data formats correspond to different schemas, and the schema needs to be correctly parsed to obtain the correct data type in order to generate the correct data.

[0137] Step 303: Call the test project main class of the debugging system to generate a test task.

[0138] After generating the test data in the data source to be tested, you need to transfer the data to the main class of the test project to parse and run the debugging task. Specifically, you can use the customized TableSource and TableSink in the Flink SQL platform to complete the rapid verification of SQL logic. TableSource and TableSink are the interfaces that need to be implemented by the customized data source target source in the Flink SQL platform.

[0139] Step 304: register the test resources to process the data source to be tested using the test resources to generate a corresponding data list.

[0140] Register the test resource (Debug Table Source) to use the test resource to process the data source to be tested and generate the corresponding data list (Table).

[0141] Specifically, if the data format of the target data source is JSON, the JSON deserializer is used to obtain the Row Typeinformation corresponding to Flink according to the corresponding Schema, where Row is a class inside Flink and RowTypeinformation is the type information corresponding to the class. If the data format of the target data source is CSV, the custom parser is used to obtain the Row Typeinformation corresponding to Flink according to the corresponding Schema. If the data format of the target data source is AVRO, the AVRO deserializer can be used to obtain the RowTypeinformation corresponding to Flink according to the corresponding Schema.

[0142] During debugging, you can only use the field names obtained from the schema and the corresponding Flink SQL internal type information. Regardless of the format of the actual data source, the data is parsed in JSON format. Therefore, the fields obtained from different Shemas and the corresponding Flink SQL internal types need to be mapped to the corresponding JSON deserializers to correctly parse the data.

[0143] Step 305: Display the data source to be tested in the debugging page of the debugging system by using a preset display data type.

[0144] In this step, the data source to be tested is displayed on the debugging page of the debugging system by presetting the display data type. Specifically, it can be processed by calling the Rest request on the product side of the SQL platform, and then calling the Spring service in the FlinkSQL platform. The Spring service is mainly used to generate data, parse the Schema to obtain data types, and other functions. In addition, the generated data can be fed back to the product front end to display the data in Json format, which can increase reading comprehension and facilitate data modification.

[0145] Step 306: Obtain, in the debugging page, a modification instruction for at least one field in the data source to be tested.

[0146] After the data in the data source to be tested is displayed in the debugging page of the debugging system, the data in the data source to be tested can be modified in the debugging page, thereby modifying the data in the data source to be tested by obtaining a modification instruction for at least one field in the data source to be tested.

[0147] Step 307: modify the data of the corresponding fields according to the modification instruction, and update the data source to be tested.

[0148] After modifying the data of the corresponding fields according to the modification instruction, the data source to be tested can be updated to provide more suitable data for debugging of subsequent SQL statements.

[0149] Step 308: Generate a data deserializer according to the preset display data type and the target type to convert the updated to-be-tested data source of the preset display data type into the to-be-tested data source of the target type.

[0150] Then, a data deserializer is generated according to the preset display data type and the target type to convert the updated test data source of the preset display data type into the test data source of the target type, thereby providing more suitable data for the subsequent debugging of SQL statements.

[0151] Step 309: Generate a data source to be tested in the debugging system according to the target type and the converted data source to be tested.

[0152] Step 310: Debug the logic program to be debugged in the debugging system using the data source to be tested.

[0153] After the data source to be tested is generated according to the target data source and the preset data mapping relationship, the data source to be tested can be used in the debugging system to debug the logic program to be debugged. Specifically, the customized TableSource and TableSink can be applied in the Flink SQL platform to complete the rapid verification of SQL logic, where TableSource and TableSink are the interfaces that the customized data source target source in the Flink SQL platform needs to implement.

[0154] Figure 5 FIG. 1 is a schematic diagram of a data debugging device according to an exemplary embodiment of the present application. Figure 5 As shown, the data debugging device 400 provided in this embodiment includes:

[0155] An acquisition module 401 is used to acquire a target data source, wherein the target data source includes data for debugging the logic program to be debugged, and the data format of the target data source is a target type, and the target type is different from a preset type in the debugging system;

[0156] A processing module 402 is used to generate a data source to be tested according to the target data source and a preset data mapping relationship, wherein the preset data mapping is used to map the data of the target type to the data of the preset type;

[0157] The debugging module 403 is used to debug the logic program to be debugged in the debugging system using the data source to be tested.

[0158] In a possible design, the processing module 402 is specifically configured to:

[0159] Acquire target mode information of the target data source, wherein the target mode information includes target data organization information and target data structure information;

[0160] Acquire target data from the target data source according to the target mode information, wherein the target data is a data item required for debugging the logic program to be debugged;

[0161] The preset data mapping relationship is used to convert each field in the target data into a field of the preset type to generate the data source to be tested.

[0162] In a possible design, the processing module 402 is further used to encapsulate each field of the preset type through the target object to generate corresponding test data, and the data source to be tested includes the test data corresponding to all fields of the preset type.

[0163] In a possible design, the debugging module 403 is specifically used to:

[0164] Calling the test project main class of the debugging system to generate a test task;

[0165] Registering a test resource to process the data source to be tested using the test resource to generate a corresponding data list;

[0166] Execute the logic program to be debugged and call the data list to output the test result;

[0167] The test results are displayed in the form of test logs.

[0168] exist Figure 5 Based on the embodiment shown, Figure 6 FIG. 1 is a schematic diagram of a data debugging device according to another exemplary embodiment of the present application. Figure 6 As shown, the data debugging device 400 provided in this embodiment further includes:

[0169] A display module 404 is used to display the data source to be tested in a debugging page of the debugging system by using a preset display data type;

[0170] A modification module 405 is used to obtain, in the debugging page, a modification instruction for at least one field in the data source to be tested;

[0171] The processing module 402 is used to modify data in corresponding fields according to the modification instruction and update the data source to be tested.

[0172] In a possible design, the processing module 402 is further used to generate a data deserializer according to the preset display data type and the target type to convert the updated data source to be tested of the preset display data type into the data source to be tested of the target type;

[0173] The processing module 402 is further configured to generate the data source to be tested in the debugging system according to the target type and the converted data source to be tested.

[0174] In a possible design, the acquisition module 401 is further used to acquire the logic program to be debugged in the debugging page;

[0175] The processing module 402 is further configured to determine that the debugging system does not have a data source required for debugging the logic program to be debugged.

[0176] The data debugging device provided in this embodiment can be used to execute the steps in the above method embodiment. For details not disclosed in the device embodiment of this application, please refer to the method embodiment of this application.

[0177] Figure 7 is a schematic diagram of the structure of an electronic device according to an exemplary embodiment of the present application. Figure 7 As shown, this embodiment provides an electronic device 500, including:

[0178] Processor 501; and

[0179] A memory 502, used to store executable instructions of the processor, and the memory may also be a flash memory;

[0180] The processor 501 is configured to execute each step in the above method by executing the executable instructions.

[0181] Optionally, the memory 502 may be independent or integrated with the processor 501 .

[0182] When the memory 502 is a device independent of the processor 501, the electronic device 500 may further include:

[0183] The bus 503 is used to connect the processor 501 and the memory 502 .

[0184] This embodiment further provides a readable storage medium, in which a computer program is stored. When at least one processor of an electronic device executes the computer program, the electronic device executes each step in the above method.

[0185] This embodiment also provides a program product, which includes a computer program stored in a readable storage medium. At least one processor of the electronic device can read the computer program from the readable storage medium, and at least one processor executes the computer program so that the electronic device implements each step in the above method.

[0186] Those skilled in the art can understand that all or part of the steps of implementing the above-mentioned method embodiments can be completed by hardware related to program instructions. The aforementioned program can be stored in a computer-readable storage medium. When the program is executed, the steps of the above-mentioned method embodiments are executed; and the aforementioned storage medium includes: ROM, RAM, disk or optical disk and other media that can store program codes.

[0187] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present application, rather than to limit it. Although the present application has been described in detail with reference to the aforementioned embodiments, those skilled in the art should understand that they can still modify the technical solutions described in the aforementioned embodiments, or replace some or all of the technical features therein with equivalents. However, these modifications or replacements do not cause the essence of the corresponding technical solutions to deviate from the scope of the technical solutions of the embodiments of the present application.

Claims

1. A data debugging method, It is characterized in that Applied to a debugging system, the debugging system is used to debug an input logic program using a data source of a preset type, the method comprising: Acquire a target data source, wherein the target data source includes data for debugging the logic program to be debugged, wherein the data format of the target data source is a target type, and the target type is different from the preset type; Generate a data source to be tested according to the target data source and a preset data mapping relationship, wherein the preset data mapping is used to map the data of the target type to the data of the preset type; Displaying the data source to be tested on a debugging page of the debugging system by using a preset display data type; If a modification instruction for at least one field in the data source to be tested is obtained in the debugging page, data of the corresponding field is modified according to the modification instruction, and the data source to be tested is updated, and a data deserializer is generated according to the preset display data type and the target type to convert the updated data source to be tested of the preset display data type into the data source to be tested of the target type; the data source to be tested in the debugging system is generated according to the target type and the converted data source to be tested; The logic program to be debugged is debugged in the debugging system using the generated data source to be tested.

2. The data debugging method according to claim 1, It is characterized in that The step of generating a data source to be tested according to the target data source and a preset data mapping relationship includes: Acquire target mode information of the target data source, wherein the target mode information includes target data organization information and target data structure information; Acquire target data from the target data source according to the target mode information, wherein the target data is a data item required for debugging the logic program to be debugged; The preset data mapping relationship is used to convert each field in the target data into a field of the preset type to generate the data source to be tested.

3. The data debugging method according to claim 2, It is characterized in that The generating the data source to be tested comprises: Each field of the preset type is encapsulated through a target object to generate corresponding test data, and the data source to be tested includes the test data corresponding to all fields of the preset type.

4. The data debugging method according to any one of claims 1 to 3, It is characterized in that The step of debugging the logic program to be debugged in the debugging system by using the data source to be tested includes: Calling the test project main class of the debugging system to generate a test task; Registering a test resource to process the data source to be tested using the test resource to generate a corresponding data list; Execute the logic program to be debugged and call the data list to output the test result; The test results are displayed in the form of test logs.

5. The data debugging method according to claim 1, It is characterized in that Before obtaining the target data source, the method further includes: In the debugging page, obtain the logic program to be debugged; Determine that there is no data source required for debugging the logic program to be debugged in the debugging system.

6. A data debugging device Characterized in that Comprising: An acquisition module, configured to acquire a target data source, where the target data source includes data for debugging a logic program to be debugged, the data format of the target data source is a target type, and the target type is different from a preset type in the debugging system; A processing module, configured to generate a data source to be tested according to the target data source and a preset data mapping relationship, where the preset data mapping is used to map data of the target type to data of the preset type; A display module, configured to display the data source to be tested on a debugging page of the debugging system through a preset display data type; A modification module, configured to, if a modification instruction for at least one field in the data source to be tested is obtained on the debugging page, modify the data of the corresponding field according to the modification instruction and update the data source to be tested; The processing module is further configured to generate a data deserialization device according to the preset display data type and the target type, so as to convert the updated data source to be tested of the preset display data type into a data source to be tested of the target type; And generate a data source to be tested in the debugging system according to the target type and the converted data source to be tested of the target type; A debugging module, configured to debug the logic program to be debugged in the debugging system by using the generated data source to be tested.

7. An electronic device Characterized in that Comprising: A processor; And A memory, configured to store a computer program of the processor; Wherein, the processor is configured to implement the data debugging method according to any one of claims 1 to 5 by executing the computer program.

8. A computer-readable storage medium, on which a computer program is stored Characterized in that The computer program, when executed by a processor, implements the data debugging method according to any one of claims 1 to 5.

9. A computer program product, including a computer program Characterized in that The computer program, when executed by a processor, implements the data debugging method according to any one of claims 1 to 5.

Images