Script generation method and device

Abstract

The invention provides a script generation method and device.The method comprises the steps that duplicate removal is conducted on a data source table selected by a user, and to-be-matched data corresponding to the data source table is obtained; determining a mapping relationship between the field of the to-be-matched data and a preset target table field; and generating a target script based on the data source table and the mapping relationship, and writing the data in the data source table into the target table based on the target script. According to the embodiment, the script development efficiency can be improved, the script development threshold is reduced, and the script code readability is improved.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of computer, in particular to a script generation method and device. BACKGROUND

[0002] It is particularly important for data analysis to extract data with analysis value or expected from massive data. However, in the prior art, the relevant personnel and the developer are usually communicated first to determine the expected extracted data, and then the developer develops the corresponding script according to the demand. This results in high communication cost in the development process, reduces the script development efficiency, and only professional developers can complete the script writing based on the demand, which reduces the readability of the script code and the development threshold. SUMMARY

[0003] Therefore, the embodiments of the present application at least provide a script generation method and device, electronic equipment and storage medium, which can improve the script development efficiency, reduce the script development threshold, and improve the readability of the script code.

[0004] In a first aspect, the embodiments of the present application provide a script generation method, comprising:

[0005] De-duplicating the data source table selected by the user to obtain the to-be-matched data corresponding to the data source table;

[0006] Determining the mapping relationship between the fields of the to-be-matched data and the preset target table fields;

[0007] Generating a target script based on the data source table and the mapping relationship, so as to write the data in the data source table into the target table based on the target script.

[0008] Optionally, before de-duplicating the data source table selected by the user, the method further comprises:

[0009] Searching for the candidate table corresponding to the keyword input by the user in the data warehouse layer selected by the user;

[0010] In response to the number of data source tables required by the user being one, determining the candidate table as the data source table based on the drag operation initiated by the user on the candidate table in the visual interface;

[0011] In response to the number of data source tables required by the user being multiple, determining at least two connected candidate tables as the data source tables based on the connection operation initiated by the user on the candidate table in the visual interface.

[0012] Optionally, the de-duplication method for de-duplicating the data source table selected by the user includes sorting de-duplication or aggregation de-duplication.

[0013] Optionally, the deduplication method is sorting deduplication; the user-selected data source table is deduplicated to obtain the to-be-matched data corresponding to the data source table, including:

[0014] Grouping the data in the data source table based on the first preset field;

[0015] In each group of the data source table, the data is arranged in descending order according to the time stamp corresponding to the data;

[0016] Filtering the data in each group except the data in the first place in the arrangement order, and taking the data in each group except the data in the first place in the arrangement order as the to-be-matched data corresponding to the data source table.

[0017] Optionally, the deduplication method is aggregation deduplication; the user-selected data source table is deduplicated to obtain the to-be-matched data corresponding to the data source table, including:

[0018] Determining the data with coinciding field values under the second preset field in the data source table;

[0019] Performing an aggregation operation on the data with coinciding field values under the second preset field to obtain the to-be-matched data corresponding to the data source table.

[0020] Optionally, the mapping relationship between the field of the to-be-matched data and the preset target table field is determined, including:

[0021] Determining the name matching value between the field of the to-be-matched data and the target table field;

[0022] In response to the name matching value being not less than a first threshold, determining the mapping relationship between the field of the to-be-matched data and the preset target table field;

[0023] In response to the name matching value being less than the first threshold and greater than or equal to a second threshold, pushing the matching result to the user, and in the case that the user initiates a confirmation operation, determining the mapping relationship between the field of the to-be-matched data and the preset target table field.

[0024] Optionally, the mapping relationship between the field of the to-be-matched data and the preset target table field is determined, including:

[0025] Performing natural semantic matching on the field of the to-be-matched data and the target table field to determine the mapping relationship between the field of the to-be-matched data and the preset target table field based on the natural semantic matching result.

[0026] Optionally, based on the data source table and the mapping relationship, a target script is generated, including:

[0027] Based on the data magnitude of the data source table, determining the execution parameters for generating the target script;

[0028] The target script is generated according to the execution parameter based on the data source table, the to-be-matched data corresponding to the data source table, and the mapping relationship.

[0029] In a second aspect, the embodiments of the present disclosure provide a script generation apparatus, comprising:

[0030] The deduplication module is configured to deduplicate the data source table selected by the user to obtain the to-be-matched data corresponding to the data source table.

[0031] The determination module is configured to determine a mapping relationship between a field of the to-be-matched data and a preset target table field.

[0032] The generation module is configured to generate a target script based on the data source table and the mapping relationship, and write data in the data source table into a target table based on the target script.

[0033] In a third aspect, the embodiments of the present disclosure further provide an electronic device, comprising a processor, a memory and a bus, the memory stores machine readable instructions executable by the processor, when the computer device is running, the processor and the memory communicate through the bus, and the machine readable instructions are executed by the processor to perform the steps of the first aspect or any optional embodiment in the first aspect.

[0034] In a fourth aspect, the embodiments of the present disclosure further provide a computer readable storage medium, the computer readable storage medium stores a computer program, and the computer program is executed by the processor to perform the steps of the first aspect or any optional embodiment in the first aspect.

[0035] In a fifth aspect, the embodiments of the present disclosure further provide a computer program product, comprising a computer program, and the computer program is executed by the processor to implement the method of any embodiment.

[0036] In any aspect or any implementation of any aspect described above, the to-be-matched data is obtained by deduplicating the user-selected data source table, thereby reducing data redundancy and avoiding the influence of duplicate data on subsequent matching and writing, so as to improve the accuracy and consistency of data processing. Secondly, by explicitly defining the mapping relationship between the to-be-matched data field and the target table field, the field matching process is more intuitive and systematic, thereby reducing the workload of manually analyzing the corresponding relationship of the fields and reducing the complexity of script development. In addition, the method automatically generates a target script based on the data source table and the mapping relationship, without the need for a developer to manually write a large number of code statements or logic, thereby reducing repetitive labor, improving development efficiency, and reducing the requirement for code ability of the developer, so that non-professional developers can also quickly generate effective scripts, thereby reducing the development threshold. Since the generation process of the script is automated, the consistency and standardization of the code structure can be guaranteed, the readability problem caused by different personal coding styles can be reduced, and the script is easier to understand and maintain. In addition, the above method can ensure that the script logic is clear and the field mapping relationship is clear, thereby reducing the error rate that may occur due to manual script writing, and making the data processing more efficient and reliable.

[0037] The effects of the script generation apparatus, the electronic device, and the storage medium described above can refer to the description of the script generation method described above, and will not be repeated here. BRIEF DESCRIPTION OF DRAWINGS

[0038] In order to more clearly illustrate the technical solutions of the embodiments of the present disclosure, the following will briefly introduce the drawings needed to be used in the embodiments. The drawings are incorporated into the specification and form a part of the specification, which show the embodiments consistent with the present disclosure, and are used to explain the technical solutions of the present disclosure together with the specification. It should be understood that the following drawings only show some embodiments of the present disclosure, and therefore should not be regarded as a limitation on the scope, and other related drawings can also be obtained by those skilled in the art without creative labor.

[0039] Figure 1 A flowchart of a script generation method provided by an embodiment of the present disclosure is shown;

[0040] Figure 2 A flowchart of a script generation method provided by an embodiment of the present disclosure is shown;

[0041] Figure 3 A schematic diagram of a target table model provided by an embodiment of the present disclosure is shown;

[0042] Figure 4 A schematic diagram of a selected data source table provided by an embodiment of the present disclosure is shown;

[0043] Figure 5A schematic diagram of a correlation data source table provided by an embodiment of the present disclosure is shown.

[0044] Figure 6 A schematic diagram of determining a mapping relationship provided by an embodiment of the present disclosure is shown.

[0045] Figure 7 A schematic diagram of a script generation device provided by an embodiment of the present disclosure is shown.

[0046] Figure 8 An exemplary system architecture to which embodiments of the present disclosure can be applied is shown.

[0047] Figure 9 A structural schematic diagram of a computer system of a terminal device or a server for implementing an embodiment of the present disclosure is shown. DETAILED DESCRIPTION

[0048] To make the objectives, technical solutions, and advantages of the embodiments of the present disclosure clearer, the following will clearly and completely describe the technical solutions in the embodiments of the present disclosure with reference to the accompanying drawings. Obviously, the described embodiments are only a part of the embodiments of the present disclosure, rather than all the embodiments. The components of the embodiments of the present disclosure described and shown in the accompanying drawings can be arranged and designed in various different configurations. Therefore, the following detailed description of the embodiments of the present disclosure provided in the accompanying drawings is not intended to limit the scope of the claimed present disclosure, but only represents selected embodiments of the present disclosure. Based on the embodiments of the present disclosure, all other embodiments obtained by those skilled in the art without creative work fall within the scope of protection of the present disclosure.

[0049] It should be noted that in the technical solutions of the present disclosure, the collection, use, storage, sharing, and transfer of user personal information and other processing comply with relevant laws and regulations, and the user is informed and obtains the consent or authorization of the user. When applicable, the user personal information is subjected to technical processing of de-identification and / or anonymization and / or encryption.

[0050] The above problems and solutions are the results obtained by the inventors after practice and careful research. The discovery process of the above problems and the solutions proposed for the above problems should be the contribution of the inventors to the present disclosure in the process of the present disclosure.

[0051] The technical solutions in the present disclosure will be described clearly and completely in the present disclosure in combination with the drawings in the present disclosure. Obviously, the described embodiments are only some of the embodiments of the present disclosure, but not all the embodiments. The components of the present disclosure described and shown in the drawings can be arranged and designed in various different configurations. Therefore, the following detailed description of the embodiments of the present disclosure provided in the drawings is not intended to limit the scope of the claimed present disclosure, but only represents selected embodiments of the present disclosure. Based on the embodiments of the present disclosure, all other embodiments obtained by those skilled in the art without creative work fall within the scope of protection of the present disclosure.

[0052] It should be noted that similar reference numerals and letters represent similar items in the following drawings, so once an item is defined in one drawing, it does not need to be further defined and explained in subsequent drawings.

[0053] To facilitate the understanding of the present embodiment, first, a script generation method disclosed by the embodiments of the present disclosure will be described in detail. The execution subject of the script generation method provided by the embodiments of the present disclosure is generally a computer device with certain computing power, which includes, for example, a terminal device or a server or other processing device. The terminal device can be a user equipment (User Equipment, UE), a mobile device, a user terminal, a terminal, a cellular phone, a cordless phone, a personal digital assistant (Personal Digital Assistant, PDA), a handheld device, a computing device, a vehicle-mounted device, a wearable device, etc. In some possible implementation manners, the script generation method can be realized by a processor calling computer readable instructions stored in a memory.

[0054] Referring to Figure 1 FIG. 1 shows a flowchart of a script generation method provided by the embodiments of the present disclosure. The method includes S101-S103, wherein:

[0055] S101: De-duplicate the data source table selected by the user to obtain the to-be-matched data corresponding to the data source table.

[0056] In the embodiments of the present disclosure, as Figure 2 shown, the target table model can be defined first. The target table can refer to the final database table to which the data is stored after cleaning, conversion or mapping. The target table model can be a design scheme for the structure, field definition, data type, index and constraint of the target table, which ensures that the data can be stored according to the predetermined rules and meets the query and analysis requirements. As Figure 3As shown, it is a schematic diagram of a target table model provided by an embodiment of the present disclosure. In the target table model, the user can first define the target table name, and then can configure the field names in the target table, such as user_id, order_id, transaction_date, etc., and can also configure the field type of each field, such as int, string, etc. In addition, description information can also be added for each field in the target table model, such as the description information of user_id can be "user unique identification code, unique for each user", the description information of username can be "username, length not more than 50 characters", etc. It should be noted that the above target table model is only used as an example to illustrate a feasible implementation in the embodiment of the present disclosure, and the target table model can also be adjusted according to the actual situation in actual application, such as adding constraints for different fields, the constraint of user_id can be "primary key", the constraint of username can be "not null", etc. The embodiment of the present disclosure does not make specific limitation, and the structure and content of the target table model can be adjusted according to the needs in actual application, as long as it can realize its function.

[0057] In the embodiment of the present disclosure, the data source table can refer to a database table storing raw data, and the data in it is usually not processed. In order to improve the efficiency of data analysis, the data in the data source table can be deduplicated to obtain the expected extracted data to be matched. In addition to the redundancy of the data in the data source table, the demand for the data source table can also be different in different scenarios, such as for an e-commerce platform, in a user behavior analysis scenario, the data source table can be a data table containing user operation logs, and the browsing, clicking, purchasing, etc. The behavior recorded in the data source table can be used to analyze user behavior; while in the financial reconciliation scenario, the required data source table can be an order transaction data table containing order amount, payment method, transaction time, etc.

[0058] As described above, different application scenarios can require different data source tables, so in actual application, the user can select the corresponding data source table according to his own needs. Specifically, before deduplicating the data source table selected by the user, it further includes: searching for a candidate table corresponding to the keyword input by the user in the data warehouse layer selected by the user; in response to the number of data source tables required by the user being one, determining the candidate table as the data source table based on the drag operation initiated by the user on the candidate table in the visualization interface; in response to the number of data source tables required by the user being multiple, determining at least two connected candidate tables as the data source table based on the connection operation initiated by the user on the candidate table in the visualization interface.

[0059] In a specific implementation, the user can first select a corresponding data warehouse layer, such as an ODS (Operational DataStore), a DWD (Data Warehouse Detail), a DWS (Data Warehouse Summary), and the like. After selecting the data warehouse layer, the user can search for a related data table in the corresponding data warehouse layer by inputting relevant field names, table names, or theme keywords, and the like. The execution subject of the embodiment of the present disclosure can return a candidate table list meeting the requirements of the user according to the information input by the user. The arrangement order of each candidate table in the candidate table list can be determined according to the degree of relevance with the information input by the user, or can be determined according to the time of searching, which is not limited in the embodiment of the present disclosure. As shown in FIG. 13, if the number of candidate tables returned by the system is one, the user can directly drag it to the target area to determine it as a data source table. Or, when the number of candidate tables returned by the system is more than one, the user can also select multiple data source tables by dragging at least one candidate table meeting the requirements to the target area. Figure 4 As shown in FIG. 14, in the process of data processing, the data source table required by the user is often more than one, and usually multiple data source tables are needed to be acquired together. In this case, the user can determine the association mode of multiple candidate tables by dragging multiple candidate tables and establishing a connection relationship between the tables. When the user completes the connection, the execution subject of the embodiment of the present disclosure can identify a logically reasonable table connection scheme by analyzing the relationship established by the user, and determine at least two connected candidate tables as the final data source tables. In this way, the user can freely combine multiple data tables to meet specific business analysis requirements when facing a complex data environment. Figure 5 As shown in FIG. 14, in the process of data processing, the data source table required by the user is often more than one, and usually multiple data source tables are needed to be acquired together. In this case, the user can determine the association mode of multiple candidate tables by dragging multiple candidate tables and establishing a connection relationship between the tables. When the user completes the connection, the execution subject of the embodiment of the present disclosure can identify a logically reasonable table connection scheme by analyzing the relationship established by the user, and determine at least two connected candidate tables as the final data source tables. In this way, the user can freely combine multiple data tables to meet specific business analysis requirements when facing a complex data environment.

[0060] It should be noted that when establishing the connection relationship of multiple data tables, there can be multiple connection forms, such as left join, inner join, right join, and union, and the like. Among them, the left join can return all records in the left table, and if there is no matching row in the right table, the part of the result will be filled with NULL; the inner join only returns the matching rows in the two tables, and the unmatched records will not appear in the result; the right join is opposite to the left join, and can return all records in the right table, and if there is no matching row in the left table, the part of the result will be filled with NULL; the union can be used to combine multiple query results, and will not be deduplicated, that is, all records including duplicate rows are retained. In a specific implementation, the different association modes described above can be distinguished by the direction or solidness of the connection line, which is not limited in the embodiment of the present disclosure, and the function can be realized.

[0061] In another possible implementation, the user can also be helped to select the data source table in an intelligent recommendation manner. Specifically, the system can perform intelligent analysis based on historical usage records, user behavior preferences, common query patterns, metadata of the data table, and the like, and automatically recommend the data source table that is most likely to meet the user's needs. For example, when the user inputs a keyword, the system not only returns the matched candidate tables, but also sorts them according to the frequency of use, data quality scores, business relevance, and the like, and displays them on the interface in different recommendation levels. In addition, the system can also analyze historical queries similar to the user's needs by combining a machine learning algorithm, and automatically suggest the optimal combination of data source tables. In this way, the selection cost of the user is further reduced, the accuracy and efficiency of data matching are improved, and the selection of data is more intelligent and personalized. It should be noted that the above-mentioned manner of selecting the data source table by the user is only used as an example to illustrate a possible implementation in the embodiments of the present disclosure, and does not constitute an improper limitation on the present application. In actual applications, it can be set according to actual needs, and the embodiments of the present disclosure do not specifically limit the manner of selecting the data source table by the user, as long as the function can be realized.

[0062] In the embodiments of the present disclosure, the deduplication method for the data source table selected by the user includes sorting deduplication or aggregation deduplication.

[0063] Specifically, the deduplication method is sorting deduplication; the deduplication is performed on the data source table selected by the user to obtain the to-be-matched data corresponding to the data source table, including: grouping the data in the data source table based on a first preset field; arranging the data in each group of the data source table in descending order according to the time stamp corresponding to the data; and filtering out the data in each group except the data in the first arrangement order, and taking the data in the first arrangement order in each group as the to-be-matched data corresponding to the data source table.

[0064] Exemplarily, it is assumed that there is a data source table recording the latest status of orders, in which the first data is "order 101, paid, 2024-03-01 10:00", the second data is "order 101, to be paid, 2024-03-01 9:57", the third data is "order 102, shipped, 2024-03-01 10:05", and the fourth data is "order 102, in stock, 2024-03-01 09:45". At this time, the first preset field can be set as the order number, then the two data belonging to order 101 can be divided into one group, and the two data with order number 102 can be divided into another group. Within the two order groups, the in-group descending order can be arranged according to the corresponding time stamp, and each data in the group can be assigned a number according to the sorting result, so that in the group of order 101, "order 101, paid, 2024-03-01 10:00" has a number 1, and "order 101, to be paid, 2024-03-01 9:57" has a number 2, then only the data row with number 1 can be retained to achieve the purpose of retaining only the latest data record. Similarly, for the group of order 102, the final retained data is "order 102, shipped, 2024-03-01 10:05". Thus, the data to be matched corresponding to the data source table is obtained, and it should be noted that the first preset field can be one or multiple, and the first preset field corresponding to different data source tables can be different, which can be selected and set by the user according to the actual situation, and the embodiments of the present disclosure do not make specific limitations thereto, and the function thereof is used as the criterion.

[0065] In the embodiments of the present disclosure, the deduplication method is aggregation deduplication; the data to be matched corresponding to the data source table is obtained by deduplicating the data source table selected by the user, including: determining the data with coinciding field values in the second preset field in the data source table; and performing an aggregation operation on the data with coinciding field values in the second preset field to obtain the data to be matched corresponding to the data source table.

[0066] Exemplarily, it is assumed that a data source table records sales of a store, where the first data is "store A, product X, sales 100 yuan", the second data is "store A, product X, sales 150 yuan", the third data is "store A, product Y, sales 200 yuan", and the fourth data is "store B, product X, sales 300 yuan". Assuming that the second preset field is the store and the product, the first data and the second data can be regarded as coincident data, and at this time, the above two data can be de-duplicated, and after de-duplication, an aggregation operation can be performed on the sales field, so that the first data and the second data are combined into "store A, product X, sales 200 yuan". It should be noted that the above method of de-duplication and addition is only used as an example of a feasible implementation in the embodiments of the present disclosure, and in actual application, other subtraction or multiplication operation functions can be selected after de-duplication according to different actual scenes, to obtain the to-be-matched data corresponding to the data source table.

[0067] In another possible implementation, in addition to the de-duplication method described above, a conditional de-duplication method can also be used to filter the data source table to obtain the to-be-matched data corresponding to the data source table. Specifically, the user can pre-configure a filtering condition to filter out data meeting the condition to obtain the to-be-matched data corresponding to the data source table. In a specific implementation, conditional de-duplication can include various filtering methods, such as single-condition filtering, such as querying a specific user; range filtering, such as querying data in a certain time period; multi-condition filtering, such as filtering by region and time at the same time; fuzzy matching, such as searching for a field name starting with a certain character. In addition, it can also include null value filtering, subquery filtering, custom function filtering, and partition filtering.

[0068] In the embodiments of the present disclosure, in addition to integrating data of multiple data source tables to obtain to-be-matched data, a two-dimensional structure can also be used to organize and associate data processing nodes. Specifically, multi-source data source table association refers to connecting multiple different data source tables to integrate information of multiple data source tables to form a complete to-be-matched data; and a two-dimensional node is a two-dimensional structure used to organize and associate data processing nodes, and the output of one node can become the input of another node to form a data flow or a dependency chain. Exemplarily, for example, some development scenarios require two data source tables to be associated first, and then the associated data source tables are filtered and processed, and finally associated with other tables. In the face of such complex processing scenarios, it is difficult to implement using a one-dimensional structure, at which time the above-mentioned two-dimensional node configuration method can be used. The specific process of configuring the data source table using the two-dimensional structure is similar to the configuration process of the data source table described above, and will not be described here.

[0069] S102: Determine the mapping relationship between the fields of the to-be-matched data and the preset target table fields.

[0070] In this embodiment of the disclosure, determining the mapping relationship between a field of the data to be matched and a preset target table field includes: determining a name matching value between the field of the data to be matched and the target table field; determining the mapping relationship between the field of the data to be matched and the preset target table field in response to the name matching value being not less than a first threshold; pushing the matching result to the user in response to the name matching value being less than the first threshold and greater than or equal to a second threshold, and determining the mapping relationship between the field of the data to be matched and the preset target table field when the user initiates a confirmation operation.

[0071] In practice, since the target table model has predefined the target table fields to be extracted, the mapping relationship between the target table field names and the field names of the data to be matched can be obtained directly using the name matching values. For example... Figure 6 As shown, the system first calculates the name matching value between the field name in the data to be matched and the field name in the target table. This matching value can be calculated based on methods such as string similarity, spelling similarity, and semantic matching to measure the similarity between the two field names. When the name matching value reaches or exceeds the first threshold, such as 99%, it indicates that the two field names are highly similar, and the mapping relationship between the field in the data to be matched and the field in the target table can be directly determined without additional manual intervention. If the name matching value is less than the first threshold but still higher than or equal to the second threshold, such as a matching value of 94%, which is greater than the second threshold of 90% but less than the first threshold of 99%, it indicates that there is a certain similarity between the two fields, but the similarity is not enough to directly confirm the mapping relationship. In this case, the matching result can be pushed to the user, providing an opportunity for manual review. The user can confirm based on their own business needs and data understanding. If the user clearly confirms the mapping relationship, the system finally determines the mapping between the field to be matched and the field in the target table. Thus, while ensuring the efficiency of automated matching, it also provides an opportunity for manual intervention in matching cases with low similarity, ensuring the accuracy and rationality of the data mapping. In addition, the matching process can also combine multi-dimensional features such as field type, context information, and historical matching records to improve the accuracy of the matching.

[0072] In this embodiment of the disclosure, in addition to the methods described above, the mapping relationship between fields can also be determined by natural semantic matching. Specifically, determining the mapping relationship between fields of the data to be matched and preset target table fields includes: performing natural semantic matching on the fields of the data to be matched and the target table fields, so as to determine the mapping relationship between the fields of the data to be matched and the preset target table fields based on the natural semantic matching results.

[0073] In a specific implementation, when determining the mapping relationship between the to-be-matched data field and the target table field, a natural semantic matching technology can also be used to improve the accuracy of matching. First, the system can perform word segmentation, part-of-speech analysis, and vectorization processing on the field name of the to-be-matched data and the target table field name, and convert the field name into a semantic feature vector. Then, based on a pre-trained semantic matching model or a word vector model, the semantic similarity between the to-be-matched field and the target table field can be calculated to measure the degree of association between them at the natural language level. In addition, multi-dimensional information such as field description, field type, and historical matching records can also be combined to enhance the reliability of matching. For example, if the name of the to-be-matched field is "customer number" and the name of the target table field is "user identification", although the character matching degree of the two is low, they still have a high similarity at the semantic level. Therefore, the system can identify their potential association through a semantic matching algorithm and automatically establish a mapping relationship. When the semantic matching degree reaches a set threshold, the system can directly determine the mapping relationship. If the matching degree is within the threshold range but does not reach complete matching, the system can push the matching suggestion to the user for confirmation to ensure the rationality of the final mapping.

[0074] It should be noted that the above two mapping relationship determination methods can be used alone or simultaneously, for example, the field matching value determined based on the string similarity can be used to preliminarily determine the mapping relationship between the to-be-matched data field and the target table field, and then the natural semantic matching can be used to determine the mapping relationship again to check for missing data. In addition, if there are still target table fields that have not been matched after using the above two methods, the mapping relationship between the fields can be determined through manual input to ensure the integrity and accuracy of the data source.

[0075] S103: Generating a target script based on the data source table and the mapping relationship, to write data in the data source table into the target table based on the target script.

[0076] In the embodiments of the present disclosure, generating a target script based on the data source table and the mapping relationship includes: determining an execution parameter for generating the target script based on the data volume of the data source table; and generating the target script based on the data source table, the to-be-matched data corresponding to the data source table, and the mapping relationship, according to the execution parameter.

[0077] In specific implementation, real-time analysis can be performed based on the data magnitude of the data source table to determine the computing requirement of the current task. If the data amount is small, the default execution parameter can be maintained to avoid resource waste; but when the data amount is large, the execution parameter can be dynamically adjusted in combination with the hardware resource condition of the current cluster to optimize the computing performance and improve the task execution efficiency. For example, the system can appropriately adjust the maximum number of executors used to control the dynamic resource allocation of the task according to the data scale, so that the computing resource can be automatically expanded when the task is highly loaded to speed up the data processing speed. In addition, the parameter that determines the default parallelism of the RDD (Resilient Distributed Dataset) can also be adjusted, and reasonable parallelism can make the task splitting more uniform, thereby improving the task execution efficiency and avoiding computing bottlenecks. Meanwhile, the execution parameter can also include the memory size of each executor, and appropriate memory configuration can effectively reduce data overflow and disk IO (Input / Output), thereby improving the overall computing performance. Alternatively, the above parameters can also be manually configured by relevant personnel to optimize the configuration according to the actual business requirement, and the specific method for adjusting the execution parameter is not limited in the present disclosure, as long as the function thereof can be realized. In addition, as shown in Figure 2 After the target table model is defined, the processing frequency can also be defined, which is used to execute the configuration script according to the processing frequency when the script generation task is executed. The parameter can be determined by the computing resource of the current cluster or pre-configured by relevant personnel.

[0078] In specific implementation, after the execution parameter is determined, the process of generating the target script can first construct the SQL (Structured Query Language) statement according to the mapping relationship between the data source table and the target table. The field mapping rule can be determined according to the field matching result, and data filtering can be performed. If multiple data sources are involved, the data of multiple data source tables can be merged according to the set association manner to ensure that the target data is complete and meets the business requirement. Then, for the execution of the data processing task, the execution parameter of the task can be automatically adjusted according to the data amount, thereby optimizing the resource utilization and improving the computing efficiency. The target script obtained in this way not only contains complete SQL logic to ensure correct data extraction, but also combines the dynamic parameter adjustment mechanism, so that the data processing task can be efficiently executed and the computing resource can be reasonably utilized under different data scales.

[0079] According to a second aspect of the present disclosure, as shown in Figure 7 a script generation apparatus 700 is provided, which comprises:

[0080] The deduplication module 701 is configured to deduplicate the data source table selected by the user to obtain the to-be-matched data corresponding to the data source table.

[0081] The determination module 702 is configured to determine a mapping relationship between a field of the to-be-matched data and a preset target table field.

[0082] The generation module 703 is configured to generate a target script based on the data source table and the mapping relationship, so as to write data in the data source table into a target table based on the target script.

[0083] Optionally, the deduplication module 701 is further configured to:

[0084] search, in the user-selected data warehouse layer, for candidate tables corresponding to the keyword input by the user;

[0085] in response to the number of data source tables required by the user being one, determine the candidate table as the data source table based on a drag operation initiated by the user on the candidate table in the visual interface;

[0086] in response to the number of data source tables required by the user being multiple, determine at least two connected candidate tables as the data source tables based on a connection operation initiated by the user on the candidate tables in the visual interface.

[0087] Optionally, the deduplication method for deduplicating the data source table selected by the user includes sorting deduplication or aggregation deduplication.

[0088] Optionally, the deduplication method is sorting deduplication, and the deduplication module 701 is specifically configured to:

[0089] group data in the data source table based on a first preset field;

[0090] arrange the data in each group of the data source table in descending order according to a time stamp corresponding to the data;

[0091] filter out data in each group except data in the first arrangement order, and take the data in the first arrangement order in each group as the to-be-matched data corresponding to the data source table.

[0092] Optionally, the deduplication method is aggregation deduplication, and the deduplication module 701 is specifically configured to:

[0093] determine data with coinciding field values in a second preset field in the data source table;

[0094] perform an aggregation operation on the data with the coinciding field values in the second preset field to obtain the to-be-matched data corresponding to the data source table.

[0095] Optionally, the determination module 702 is specifically configured to:

[0096] determine a name matching value between the field of the to-be-matched data and the target table field;

[0097] In response to the name matching value being not less than a first threshold, determine a mapping relationship between the field of the to-be-matched data and the preset target table field;

[0098] In response to the name matching value being less than the first threshold and greater than or equal to a second threshold, push the matching result to a user, and in a case where the user initiates a confirmation operation, determine the mapping relationship between the field of the to-be-matched data and the preset target table field.

[0099] Optionally, the determining module 702 is specifically configured to:

[0100] perform natural semantic matching on the field of the to-be-matched data and the target table field, to determine the mapping relationship between the field of the to-be-matched data and the preset target table field based on a natural semantic matching result.

[0101] Optionally, the generating module 703 is specifically configured to:

[0102] determine an execution parameter for generating the target script based on a data magnitude of the data source table;

[0103] generate the target script according to the execution parameter, based on the data source table, the to-be-matched data corresponding to the data source table, and the mapping relationship.

[0104] According to a third aspect of the embodiments of the present disclosure, an electronic device for script generation is provided, including one or more processors, and a storage device configured to store one or more programs, when the one or more programs are executed by the one or more processors, the one or more processors implement the method provided in the first aspect of the embodiments of the present disclosure.

[0105] According to a fourth aspect of the embodiments of the present disclosure, a computer readable medium is provided, and the computer readable medium stores a computer program, when the computer program is executed by a processor, the method provided in the first aspect of the embodiments of the present disclosure is implemented.

[0106] According to a fifth aspect of the embodiments of the present disclosure, a computer program product is provided, including a computer program, when the computer program is executed by a processor, the method of any of the above embodiments is implemented.

[0107] Figure 8 An exemplary system architecture 800 to which the script generation method or script generation device of the present disclosure can be applied is shown.

[0108] As Figure 8As shown, the system architecture 800 can include terminal devices 801, 802, 803, a network 804 and a server 805. The network 804 is a medium for providing communication links between the terminal devices 801, 802, 803 and the server 805. The network 804 can include various connection types, such as wired, wireless communication links or optical fiber cables, etc.

[0109] The users can use the terminal devices 801, 802, 803 to interact with the server 805 through the network 804 to receive or send messages, etc. Various communication client applications can be installed on the terminal devices 801, 802, 803, such as shopping applications, web browser applications, search applications, instant messaging tools, email clients, social platform software, etc. (only as examples).

[0110] The terminal devices 801, 802, 803 can be various electronic devices with display screens and supporting web browsing, including but not limited to smart phones, tablet computers, laptop computers and desktop computers, etc.

[0111] The server 805 can be a server providing various services, such as a background management server supporting shopping websites browsed by users using the terminal devices 801, 802, 803 (only as an example). The background management server can process received script generation requests and feed back processing results (only as an example) to the terminal devices.

[0112] It should be noted that the script generation method provided by the embodiments of the present application is generally executed by the server 805, and correspondingly, the script generation device is generally arranged in the server 805. The script generation method provided by the embodiments of the present application can also be executed by the terminal devices 801, 802, 803, and correspondingly, the script generation device can be arranged in the terminal devices 801, 802, 803.

[0113] It should be understood that, Figure 8 The number of terminal devices, networks and servers in the system architecture 800 is only illustrative. According to the needs of implementation, there can be any number of terminal devices, networks and servers.

[0114] Reference is made below to Figure 9 which shows a structural schematic diagram of a computer system 900 of a terminal device suitable for implementing the embodiments of the present application. Figure 9 The terminal device shown is only an example and should not bring any limitation to the functions and use range of the embodiments of the present application.

[0115] As Figure 9As shown, the computer system 900 includes a central processing unit (CPU) 901 which can perform various appropriate actions and processes according to programs stored in a read only memory (ROM) 902 or loaded into a random access memory (RAM) 903 from a storage section 908. In the RAM 903, various programs and data required for the operation of the system 900 are also stored. The CPU 901, the ROM 902, and the RAM 903 are connected to each other through a bus 904. An input / output (I / O) interface 905 is also connected to the bus 904.

[0116] Connected to the I / O interface 905 are an input section 906 including a keyboard, a mouse, etc.; an output section 907 including a display such as a cathode ray tube (CRT), a liquid crystal display (LCD), etc., and a speaker, etc.; a storage section 908 including a hard disk, etc.; and a communication section 909 including a network interface card such as a LAN card, a modem, etc. The communication section 909 performs communication processing via a network such as the Internet. A drive 910 is also connected to the I / O interface 905 as necessary. A removable recording medium 911 such as a magnetic disk, an optical disk, a magneto-optical disk, a semiconductor memory, etc. is attached to the drive 910 as necessary, so that a computer program read therefrom is installed into the storage section 908 as necessary.

[0117] In particular, the processes described above with reference to the flow charts can be implemented as a computer software program according to embodiments of the present disclosure. For example, embodiments of the present disclosure include a computer program product comprising a computer program carried on a computer readable medium, the computer program comprising program code for performing the methods illustrated by the flow charts. In such embodiments, the computer program can be downloaded and installed from a network via the communication section 909 and / or installed from the removable recording medium 911. When the computer program is executed by the central processing unit (CPU) 901, the above-described functions defined in the system of the present disclosure are performed.

[0118] It should be noted that the computer-readable medium shown in the present application can be a computer-readable signal medium or a computer-readable storage medium or any combination of the above two. The computer-readable storage medium may, for example, but is not limited to, an electrical, magnetic, optical, electromagnetic, infrared, or semiconductor system, device or component, or any combination of the above. More specific examples of computer-readable storage media can include, but are not limited to, an electrical connection having one or more wires, a portable computer diskette, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the above. In the present application, the computer-readable storage medium can be any tangible medium containing or storing a program that can be used by or in conjunction with an instruction execution system, device or component. In the present application, the computer-readable signal medium can include a data signal carried in a baseband or as a part of a carrier wave, which carries computer-readable program code. Such a propagated data signal can take various forms, including but not limited to an electromagnetic signal, an optical signal or any suitable combination of the above. The computer-readable signal medium can also be any computer-readable medium other than the computer-readable storage medium, which can send, propagate or transmit a program for use by or in conjunction with an instruction execution system, device or component. The program code contained on the computer-readable medium can be transmitted by any suitable medium, including but not limited to wireless, wire, optical cable, RF, etc., or any suitable combination of the above.

[0119] The flowcharts and block diagrams in the drawings illustrate the possible implementation architectures, functions and operations of the systems, methods and computer program products according to various embodiments of the present application. In this regard, each block in the flowcharts or block diagrams can represent a module, a program segment or a part of code containing one or more executable instructions for implementing the specified logical functions. It should also be noted that in some alternative implementations, the functions noted in the blocks can occur in different order than that shown in the drawings. For example, two blocks that are shown in succession can actually be executed substantially in parallel, and sometimes in reverse order, depending on the functions involved. It should also be noted that each block in the block diagrams or flowcharts, and the combination of blocks in the block diagrams or flowcharts, can be implemented by a dedicated hardware-based system that performs the specified functions or operations, or can be implemented by a combination of dedicated hardware and computer instructions.

[0120] The modules described in the embodiments of the present application can be implemented in the form of software or hardware. The described modules can also be arranged in a processor, for example, a processor includes a deduplication module, a determination module, a generation module, wherein the names of these modules do not constitute a limitation on the modules themselves in some cases, for example, the deduplication module can also be described as "deduplicating the user-selected data source table to obtain the to-be-matched data corresponding to the data source table".

[0121] As another aspect, the present application also provides a computer readable medium, which can be included in the device described in the above embodiments, or can exist independently without being assembled into the device. The computer readable medium carries one or more programs, when the one or more programs are executed by the device, the device implements the following method: deduplicating the user-selected data source table to obtain the to-be-matched data corresponding to the data source table; determining the mapping relationship between the fields of the to-be-matched data and the preset target table fields; generating a target script based on the data source table and the mapping relationship, so as to write the data in the data source table into the target table based on the target script.

[0122] Finally, it should be noted that: the above embodiments are only specific implementations of the present disclosure, used to illustrate the technical solutions of the present disclosure, rather than limit them, the protection scope of the present disclosure is not limited thereto, although the present disclosure has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that any person skilled in the art within the technical range disclosed by the present disclosure can still modify or easily think of changes to the technical solutions recorded in the foregoing embodiments, or make equivalent replacement to some technical features thereof; and these modifications, changes or replacements do not make the corresponding technical solutions deviate from the spirit and scope of the technical solutions of the embodiments of the present disclosure, and should be covered in the protection scope of the present disclosure. Therefore, the protection scope of the present disclosure should be subject to the protection scope of the claims.

Claims

1. A script generation method characterized by, The method comprises the following steps: de-duplicating a data source table selected by a user to obtain to-be-matched data corresponding to the data source table; determining a mapping relationship between fields of the to-be-matched data and preset target table fields; generating a target script based on the data source table and the mapping relationship, so as to write data in the data source table into the target table based on the target script.

2. The method of claim 1, wherein, Before de-duplicating the data source table selected by the user, the method further comprises the following steps: searching for candidate tables corresponding to the keyword input by the user in a data warehouse layer selected by the user; in response to the number of data source tables required by the user being one, determining the candidate table as the data source table based on a drag operation initiated by the user on the candidate table in a visual interface; in response to the number of data source tables required by the user being multiple, determining at least two connected candidate tables as the data source tables based on a connection operation initiated by the user on the candidate table in the visual interface.

3. The method of claim 1, wherein, The de-duplication method for de-duplicating the data source table selected by the user comprises sorting de-duplication or aggregation de-duplication.

4. The method of claim 3, wherein, The de-duplication method is sorting de-duplication; the method of de-duplicating the data source table selected by the user to obtain to-be-matched data corresponding to the data source table comprises the following steps: grouping data in the data source table based on a first preset field; arranging the data in each group of the data source table in descending order according to the time stamp corresponding to the data; filtering out data in each group except the data in the first place, and taking the data in the first place in each group as to-be-matched data corresponding to the data source table.

5. The method of claim 3, wherein, The de-duplication method is aggregation de-duplication; the method of de-duplicating the data source table selected by the user to obtain to-be-matched data corresponding to the data source table comprises the following steps: determining data with coinciding field values under a second preset field in the data source table; performing an aggregation operation on the data with coinciding field values under the second preset field to obtain to-be-matched data corresponding to the data source table.

6. The method of claim 1, wherein, The method of determining a mapping relationship between fields of the to-be-matched data and preset target table fields comprises the following steps: determining a name matching value between the fields of the to-be-matched data and the target table fields; in response to the name matching value being not less than a first threshold value, determining the mapping relationship between the fields of the to-be-matched data and the preset target table fields; in response to the name matching value being less than the first threshold value and greater than or equal to a second threshold value, pushing the matching result to the user, and determining the mapping relationship between the fields of the to-be-matched data and the preset target table fields if the user initiates a confirmation operation.

7. The method of claim 1, wherein, The method of determining a mapping relationship between fields of the to-be-matched data and preset target table fields comprises the following steps: performing natural semantic matching on the fields of the to-be-matched data and the target table fields, so as to determine the mapping relationship between the fields of the to-be-matched data and the preset target table fields based on the natural semantic matching result.

8. The method of claim 1, wherein, The method of generating a target script based on the data source table and the mapping relationship comprises the following steps: determining an execution parameter for generating the target script based on the data magnitude of the data source table; The target script is generated according to the execution parameter based on the data source table, the to-be-matched data corresponding to the data source table, and the mapping relationship.

9. A script generation apparatus characterized by comprising: The method comprises: The de-duplication module is configured to de-duplicate the data source table selected by the user to obtain the to-be-matched data corresponding to the data source table. The determination module is configured to determine a mapping relationship between a field of the to-be-matched data and a preset target table field. The generation module is configured to generate a target script based on the data source table and the mapping relationship, and write data in the data source table into the target table based on the target script.

10. An electronic device, comprising: The method comprises: One or more processors; Storage apparatus configured to store one or more programs, When the one or more programs are executed by the one or more processors, the one or more processors implement the method of any one of claims 1-8.

11. A computer readable medium having stored thereon a computer program, characterized in that, The program is executed by the processor to implement the method of any one of claims 1-8.

12. A computer program product comprising a computer program which, when executed by a processor, implements the method of any one of claims 1-8.

Images