A visual data development method, device, equipment and medium

Abstract

The present invention provides a visual data development method, device, equipment and medium, the method comprising: establishing multiple first data tables in a visual data development interface according to a preset data table establishment specification, the multiple first data tables including editable item information; removing redundant data tables in the multiple first data tables according to a preset data table approval specification to obtain at least one second data table; acquiring data synchronization tasks and external data, and based on the data synchronization tasks, synchronizing the external data to at least one second data table according to the editable item information to obtain at least one third data table; performing data processing on the data in at least one third data table to obtain a data processing result. The present invention establishes a data table through a data table establishment specification and a data table approval specification, so that the entire data development process can be controlled, the scheduling dependency can be clearly checked, the produced data processing results have clear lineage, and the redundancy of intermediate tables and scheduling processes is reduced.

Description

Technical Field

[0001] The present application relates to the field of big data analysis technology, and specifically to a method, device, equipment and medium for developing visualized data. Background Art

[0002] With the application of big data technology, each company has a professional data processing team. However, as time goes by and the amount of data increases, the number of people in the original data processing team increases. Everyone in the team has their own habits of writing SQL (Structured Query Language) or other programs, naming files, project directories, and data layering. This will result in no one being able to clearly understand the table associations and scheduling dependency logic in the original system, increasing the difficulty of data processing iteration. In addition, there are a large number of redundant scheduling tasks and intermediate tables in the data processing process, wasting system computing and storage resources. Summary of the invention

[0003] In view of the above-mentioned shortcomings of the prior art, the present invention provides a visualization data development method, device, equipment and medium to solve the above-mentioned technical problems.

[0004] In one embodiment of the present invention, a method for developing visual data is provided, the method comprising:

[0005] Establishing a plurality of first data tables in the visual data development interface according to a preset data table establishment specification, wherein the plurality of first data tables include editable item information;

[0006] According to a preset data table approval specification, redundant data tables are removed from the plurality of first data tables to obtain at least one second data table;

[0007] Acquire a data synchronization task and external data, and based on the data synchronization task, synchronize the external data into the at least one second data table according to the editable item information to obtain at least one third data table;

[0008] Data processing is performed on the data in the at least one third data table to obtain a data processing result.

[0009] In one embodiment of the present invention, the step of establishing a plurality of first data tables in the visual data development interface according to a preset data table establishment specification includes:

[0010] In response to the data development requirement instruction, acquiring editable item information required for establishing the plurality of first data tables, and presetting a data table establishment specification according to the editable item information;

[0011] A plurality of first data tables are established in the visual data development interface according to a preset data table establishment specification.

[0012] In one embodiment of the present invention, based on the data synchronization task, synchronizing the external data into the at least one second data table according to the editable item information to obtain at least one third data table includes:

[0013] Remove redundant data synchronization tasks according to the preset data synchronization task approval specifications to obtain approved data synchronization tasks;

[0014] Connecting the data source of the external data with the at least one second data table, and performing field mapping on the data table in the data source of the external data and the at least one second data table according to the editable item information to obtain a field mapping result;

[0015] In response to the approved data synchronization task, the external data is synchronized to the at least one second data table according to the field mapping result and the preset data synchronization cycle and partition field to obtain the at least one third data table.

[0016] In one embodiment of the present invention, in response to the approved data synchronization task, when the external data is synchronized to the at least one second data table according to the field mapping result and the preset data synchronization cycle and partition field, the external data in the same data table is only synchronized once.

[0017] In one embodiment of the present invention, the connecting the data source of the external data with the at least one second data table, and performing field mapping on the data table in the data source of the external data and the at least one second data table according to the editable item information to obtain the field mapping result includes:

[0018] Acquire first field information of a data table in a data source of the external data and second field information of the at least one second data table, and match the first field information and the second field information according to the editable item information to obtain a matching result;

[0019] According to the matching result, the first field information is mapped to the at least one second data table to obtain a field mapping result.

[0020] In one embodiment of the present invention, the step of processing the data in the at least one third data table to obtain a data processing result includes:

[0021] In response to the scheduling task, scheduling the at least one third data table according to a preset scheduling dependency;

[0022] Data in at least one scheduled third data table is extracted, and multi-dimensional data processing is performed on the extracted data to obtain a data processing result.

[0023] In one embodiment of the present invention, extracting data from at least one third data table that is scheduled, and performing multi-dimensional data processing on the extracted data to obtain a data processing result, comprises:

[0024] Establish a data table blood relationship display module;

[0025] The data table lineage display module is used to visualize the data table lineage, scheduling dependency, and data processing results, wherein the data table lineage is determined during the execution of data synchronization tasks and scheduling tasks.

[0026] In one embodiment of the present invention, a visual data development device is provided, the device comprising:

[0027] A data table establishment module, used to establish a plurality of first data tables in the visual data development interface according to a preset data table establishment specification, wherein the plurality of first data tables include editable item information;

[0028] A data table approval module, configured to remove redundant data tables from the plurality of first data tables according to a preset data table approval specification, to obtain at least one second data table;

[0029] A data synchronization module, used for acquiring a data synchronization task and external data, and based on the data synchronization task, synchronizing the external data into the at least one second data table according to the editable item information, to obtain at least one third data table;

[0030] The data processing module is used to process the data in the at least one third data table to obtain a data processing result.

[0031] In one embodiment of the present invention, an electronic device is provided, the electronic device comprising:

[0032] one or more processors;

[0033] The storage device is used to store one or more programs, and when the one or more programs are executed by the one or more processors, the electronic device implements the visual data development method as described above.

[0034] In one embodiment of the present invention, a computer-readable storage medium is provided, on which a computer program is stored. When the computer program is executed by a processor of a computer, the computer executes the visual data development method as described above.

[0035] The beneficial effects of the present invention are as follows: multiple first data tables are established in a visual data development interface according to preset data table establishment specifications, redundant data tables in multiple first data tables are removed according to preset data table approval specifications, and at least one second data table is obtained, thereby ensuring the specifications and specifications of all data tables in the data development system and ensuring that the established data tables are not redundant; obtaining data synchronization tasks and external data, and based on the data synchronization tasks, synchronizing the external data to at least one second data table according to the editable item information, obtaining at least one third data table, performing data synchronization according to the synchronization tasks, avoiding confusion in the lineage of data tables due to the existence of other development projects in the data development system; performing data processing on the data in at least one third data table to obtain data processing results. The present invention performs data table establishment, approval, and scheduling tasks based on a visual method, thereby achieving controllable data development process, clear and traceable scheduling dependencies, and producing data processing results with clear lineage, thereby reducing redundancy in intermediate tables and scheduling processes.

[0036] It should be understood that the foregoing general description and the following detailed description are exemplary and explanatory only and are not restrictive of the present application. BRIEF DESCRIPTION OF THE DRAWINGS

[0037] The drawings herein are incorporated into the specification and constitute a part of the specification, showing embodiments consistent with the present application, and together with the specification, are used to explain the principles of the present application. Obviously, the drawings described below are only some embodiments of the present application, and for those of ordinary skill in the art, other drawings can be obtained based on these drawings without creative work. In the drawings:

[0038] Figure 1 is a schematic diagram of an implementation environment for visual data development shown in an exemplary embodiment of the present application;

[0039] Figure 2 is a flow chart of a visual data development method shown in an exemplary embodiment of the present application;

[0040] Figure 3 yes Figure 2 Step S210 in the illustrated embodiment is a flow chart in an exemplary embodiment;

[0041] Figure 4 yes Figure 2 Step S230 in the illustrated embodiment is a flow chart in an exemplary embodiment;

[0042] Figure 5 is an implementation architecture diagram of a visual data development method shown in an exemplary embodiment of the present application;

[0043] Figure 6is a block diagram of a visual data development device shown in an exemplary embodiment of the present application;

[0044] Figure 7 A schematic diagram of the structure of a computer system suitable for implementing an electronic device of an embodiment of the present application is shown. DETAILED DESCRIPTION

[0045] The following will describe the embodiments of the present invention with reference to the accompanying drawings and preferred embodiments. Those skilled in the art can easily understand other advantages and effects of the present invention from the contents disclosed in this specification. The present invention can also be implemented or applied through other different specific embodiments, and the details in this specification can also be modified or changed in various ways based on different viewpoints and applications without departing from the spirit of the present invention. It should be understood that the preferred embodiments are only for illustrating the present invention, not for limiting the scope of protection of the present invention.

[0046] It should be noted that the illustrations provided in the following embodiments are only schematic illustrations of the basic concept of the present invention, and thus the drawings only show components related to the present invention rather than being drawn according to the number, shape and size of components in actual implementation. In actual implementation, the type, quantity and proportion of each component may be changed arbitrarily, and the component layout may also be more complicated.

[0047] In the following description, numerous details are discussed to provide a more thorough explanation of the embodiments of the present invention. However, it is obvious to those skilled in the art that the embodiments of the present invention can be implemented without these specific details. In other embodiments, well-known structures and devices are shown in the form of block diagrams rather than in detail to avoid making the embodiments of the present invention difficult to understand.

[0048] Here, the abbreviations and key term definitions related to this application are introduced:

[0049] Data development: Data development involves the company's product capabilities, which mainly include three parts: offline development, real-time development, and algorithm development. Offline development mainly includes data processing, publishing, operation and maintenance management, data analysis, data exploration, online query, and ad hoc analysis-related work; real-time development mainly involves real-time access and real-time processing of data, simplifying the processing of streaming data; algorithm development mainly involves providing simple and easy visual drag and drop and Notebook (research information recording) methods to achieve in-depth mining of data value. This application mainly improves data table establishment, data synchronization, and data processing in the data development process through the method of visualizing the data development process.

[0050] Data development system: The data development system includes a control layer, a service layer, and an application layer. The control layer is the core of data processing in the offline development of the data development platform, and the workflow scheduling engine undertakes the scheduling tasks of the entire data development platform; the service layer provides services for the application layer or other external applications; the application layer directly interacts with the user based on the underlying service and provides the user with a visual operation interface. In this application, the user's interactive instructions are received in the visual interface of the application layer, and the user's interactive instructions are responded to to control the establishment of the data table and the data synchronization process.

[0051] Scheduling dependency: Scheduling dependency is the upstream and downstream dependency relationship of the task node. The downstream task node will start running only after the upstream task node has completed and run successfully. After configuring the scheduling dependency, it can be guaranteed that the scheduled task can obtain the correct data when it is running (after the upstream node that the current node depends on runs successfully, the latest data in the upstream table is recognized through the node running status, and the downstream node will then fetch the data). Avoid problems with downstream node fetching data when the upstream table data has not been normally produced when the downstream node fetches data.

[0052] Field mapping: Field mapping is used in the field of data analysis technology and can be used in the Transfer Attributes tool and other editing tools that can paste attributes from one feature layer to other features. For example, if you need to synchronize the data of a field (column) in the original data table in the external data source to a field (column) in the target data table, you need to establish a corresponding relationship between the data elements of the two data tables. At this time, a mapping relationship is established between the two fields so that the data elements can be accurately transferred to the target data table. This process is called field mapping.

[0053] It should be noted that only when data is integrated and calculated can it be used to gain insight into business rules and tap into potential information, thereby realizing the value of big data and achieving the goal of empowering business and creating value. The source of data in some companies is online data collected from internal business systems and web pages. Faced with massive amounts of data and complex calculations, when business changes frequently, rapid delivery is required, there are too many operation and maintenance tasks, and the system environment is complex, coupled with the high turnover of company employees, it will make it difficult for others to understand the data table associations and data table scheduling methods of previous data developers in the original system, which will increase the difficulty and workload of data development and processing, and affect overall efficiency and quality. As the number of people in the data development team increases with the expansion of the business, everyone has their own habits of writing SQL (Structured Query Language) or other programs, naming files, project directories, and data layering, which will lead to the following problems: the dependencies between tables can only be viewed by the development engineers who develop the model or report in the source code; the calculation logic of the fields can only be obtained by viewing the source code; the code standards are not unified, which increases the time to read the code; manually checking the field calculation caliber and the dependencies between tables takes up the time of development engineers and reduces the efficiency of data development.

[0054] In addition, there are a large number of redundant scheduling tasks and intermediate tables in the data processing process, which will waste too much system computing and storage resources.

[0055] Figure 1 FIG. 1 is a schematic diagram of an implementation environment of visual data development shown in an exemplary embodiment of the present application. Figure 1 As shown, the implementation environment includes terminal devices 101, 102, 103, a network 104 and a server 105. The network 104 can provide a medium for a communication link between the terminal devices 101, 102, 103 and the server. The network 104 can include various connection types, such as wired, wireless communication links or optical fiber cables, etc.

[0056] Users can use terminal devices 101, 102, 103 to interact with server 105 through network 104 to receive or send messages, etc. Terminal devices 101, 102, 103 can be various electronic devices with display screens and supporting web browsing, including but not limited to smart phones, tablet computers, laptop computers, desktop computers, wearable devices, virtual reality devices, smart homes, smart voice interaction devices, smart home appliances, vehicle-mounted terminals, etc.

[0057] The server 105 may be a server that provides various services, such as a background management server that provides support for devices operated by users using the terminal devices 101, 102, and 103. The background management server may analyze and process the received request and other data, and feed back the processing results to the terminal devices 101, 102, and 103.

[0058] Server 105 can be an independent physical server, or a server cluster or distributed system composed of multiple physical servers. It can also be a cloud server that provides cloud services, cloud databases, cloud computing, cloud functions, cloud storage, network services, cloud communications, middleware services, domain name services, security services, CDN (Content Delivery Network), as well as basic cloud computing services such as big data and artificial intelligence platforms, etc. The present disclosure does not impose any restrictions on this.

[0059] The terminal devices 101, 102, 103 may, for example, perform the following steps:

[0060] Establishing a plurality of first data tables in the visual data development interface according to a preset data table establishment specification, wherein the plurality of first data tables include editable item information;

[0061] According to a preset data table approval specification, redundant data tables are removed from the plurality of first data tables to obtain at least one second data table;

[0062] Acquire a data synchronization task and external data, and based on the data synchronization task, synchronize the external data into the at least one second data table according to the editable item information to obtain at least one third data table;

[0063] Data processing is performed on the data in the at least one third data table to obtain a data processing result.

[0064] See also Figure 2 , Figure 2 is a flowchart of a visual data development method shown in an exemplary embodiment of the present application. The method provided in the embodiment of the present application can be executed by any electronic device with computing processing capabilities. For example, the method can be executed by the above Figure 1 The server 105 or the terminal devices 101, 102, 103 in the embodiment may also be executed by the server 105 and the terminal devices 101, 102, 103. In the following embodiments, the terminal devices 101, 102, 103 are used as the execution subjects for illustration, but the present disclosure is not limited thereto.

[0065] Exemplarily, before executing the following visualization data development method, the terminal devices 101, 102, and 103 first determine the basic functions that the data development platform needs to have by clarifying the business demand scenarios and the usage needs of the enterprise or users; obtain the function implementation layer according to the basic functions that the data development platform needs to have, including but not limited to the data collection layer, the data storage and processing layer, the data analysis layer, and the data application layer. The data collection layer is mainly responsible for collecting and parsing the data in the underlying data source to integrate the scattered data, the data storage and processing layer is mainly responsible for storing the collected and parsed data, the data analysis layer can, for example, perform simple processing on the data through the reporting system and the analysis system, and then conduct in-depth analysis and mining, and the data application layer can, for example, be divided into different categories of applications according to business needs, including but not limited to visualization display; then establish a data warehouse, and the data warehouse technically adopts a three-tier architecture design: ODS (operational data warehouse), DW (data warehouse), and DM (data mart).

[0066] Through the above method, the preliminary construction of the data development platform can be completed.

[0067] like Figure 2 As shown, in an exemplary embodiment, the visualization data development method includes at least steps S210 to S240, which are described in detail as follows:

[0068] Step S210: establishing a plurality of first data tables in the visual data development interface according to a preset data table establishment specification.

[0069] Among them, before performing step S210, the data development requirement instructions issued by the user can be obtained first, and the editable item information of the user's requirements, as well as the data table establishment specifications and data approval specifications can be obtained through the data development requirement instructions. The editable item information in different data development requirements of different companies is also different. The data development requirement instructions can be instructions obtained by the user through manipulation on the terminal devices 101, 102, and 103. This application does not make specific restrictions on the data table establishment specifications and data approval specifications. In addition, the data development method in this application is carried out in a visual data development interface to visually manage the entire data development process and realize the controllability of the entire data development process.

[0070] It should be noted that, in one embodiment of the present application, in order to better illustrate the data table creation and approval process in step S210, as shown in FIG. Figure 3 As shown, Figure 3 yes Figure 2 Step S210 in the illustrated embodiment is a flow chart in an exemplary embodiment, and step S210 at least includes steps S310 to S320, which are described in detail as follows:

[0071] S310 . In response to a data development requirement instruction, obtain editable item information required for establishing a plurality of first data tables, and preset a data table establishment specification according to the editable item information.

[0072] Exemplarily, before performing the operation of creating the first data table, first obtain the data development demand instruction, respond to the data development demand instruction, obtain the demand for editable item information in each data development process, and then establish the editable item information according to the demand for editable item information in each data development process. Obtain the editable item information required for establishing multiple first data tables. In this embodiment, the editable item information includes but is not limited to the data table name, annotations, life cycle, administrator information, save location, field, field Chinese name, field remarks, table partitions, and the editable information can be pre-set and adjusted according to user needs and company requirements. After setting the data table establishment specifications according to the editable item information, during the data development process, all data development content must use the established data table for data development and scheduling to ensure that the lineage and scheduling dependencies of the data tables used in the entire data development process are clear.

[0073] S320: Establish a plurality of first data tables in the visual data development interface according to a preset data table establishment specification.

[0074] In one embodiment of the present application, the whole process of visual table building is realized in the visual data development interface. During the table building process, the table building approval is carried out according to the preset data table building specifications, and after the table building is completed, the data table approval is completed through the table building approval process to complete the establishment of the data table, thereby ensuring the specifications of all data tables in the data development system. In addition, in the subsequent data development process, it is stipulated that only the data tables established according to the data table establishment specifications and approved by the approval specifications are allowed to be used, and the use of code to build tables during the data development process is not supported, so as to ensure that data development engineers using the same data development platform can clearly and efficiently understand the association relationship and scheduling dependency logic of the data tables, while avoiding the situation of confusion in the lineage of data tables.

[0075] In step S220, redundant data tables in the plurality of first data tables are removed according to a preset data table approval specification to obtain at least one second data table.

[0076] In one embodiment of the present application, after the first data table is established, it is necessary to review whether it complies with the specifications. A data table that does not comply with the specifications is a redundant data table, and at least one second data table is obtained by removing the redundant data tables in the multiple first data tables. In this embodiment, the approval specifications can be pre-established based on the editable item information, or can be established based on other customer demand information. After the data table is established, it is approved through the approval process set by the internal process to ensure that the established data table is not redundant. Redundant data tables will increase data transmission overhead, waste storage space, and reduce the reliability of data and data development results.

[0077] In step S230, a data synchronization task and external data are acquired, and based on the data synchronization task, the external data is synchronized into at least one second data table according to the editable item information to obtain at least one third data table.

[0078] Exemplarily, before data synchronization, it is necessary to establish an association relationship between external data and the data development platform. When the association relationship between external data and the data development platform has been established, obtain the data synchronization instruction issued by the user on the visual data synchronization page, and parse the data synchronization instruction to obtain the data synchronization task. In response to the data synchronization task, through the connection, mapping and other relationships between the data table and the external data, perform synchronization operations such as synchronization time, synchronization cycle, field mapping, table partitioning, etc., synchronize the external source data in the data table of the data development platform to obtain at least one third data table, which can be tested and put online later.

[0079] It should be noted that external data refers to data outside the data development platform, that is, data obtained from an external data source. External data exists in the form of a data table in the external data source. A data table in an external data source may include data elements of one field or data elements of multiple fields. External data sources may include data sources within the enterprise and data sources outside the enterprise. The data in the data source within the enterprise refers to the data generated in the production process of the main business of the enterprise; while the data in the data source outside the enterprise refers to the data not generated by the enterprise itself. This type of data can be used to solve the core problems of the growth of the enterprise's business, including but not limited to user growth data, competitive product analysis data, industry analysis data, decision support data and certain business innovation data. In this embodiment, external data can be extracted from the business system of the enterprise, downloaded on the Internet, or obtained through data collection tools such as crawlers.

[0080] In one embodiment of the present application, the data synchronization process is described in detail, see Figure 4 , Figure 4 yes Figure 2 Step S230 in the illustrated embodiment is a flow chart in an exemplary embodiment, which specifically includes the following steps:

[0081] S410. Remove redundant data synchronization tasks according to preset data synchronization task approval specifications to obtain approved data synchronization tasks.

[0082] In one embodiment of the present application, after obtaining data synchronization tasks, if some data synchronization tasks have duplicate tasks, the preset data synchronization task approval specification is used for approval, and redundant data synchronization tasks are removed to reduce the generation of redundant data scheduling, thereby avoiding the generation of redundant data table lineage and scheduling dependencies. It should be noted that the data synchronization task approval specification in this embodiment can be pre-set according to the actual needs of the company, and this application does not make specific restrictions on the data synchronization task approval specification.

[0083] S420: Connect the data source of the external data to at least one second data table, and perform field mapping on the data table in the data source of the external data and the at least one second data table according to the editable item information to obtain a field mapping result.

[0084] In one embodiment of the present application, at least one second data table is connected to an external data source, and then a data table selection and a second data table selection in the external data source are performed. Field mapping is performed based on the selected data table and the second data table in the external data source to obtain a field mapping result.

[0085] In this embodiment, before data processing or data analysis, it is necessary to sort out which data tables are used when calculating various indicators and dimensions, as well as the flow of data elements in the fields in the data tables of external data sources, that is, it is necessary to obtain the field relationship mapping table of the ODS, DW, and DM layers, which constitutes the blood relationship between tables. Through the blood relationship between tables, the dependency relationship between tables can be obtained. When errors occur in the upstream data batch run, the downstream data can be automatically run; through the blood relationship at the field level, the calculation formula of the field is understood, reducing the learning cost of the data development platform.

[0086] Exemplarily, after the data source of the external data is connected to at least one second data table, the data table name and storage location in the external data source are selected according to the editable item information in this application; the data table in the external data source is extracted according to the data table name in the data table storage location, and then the field area in the data table that needs to be mapped is selected through the table partition, and then the field selection is performed through the Chinese name of the field. The above operation can also be performed in at least one second data table to determine the field corresponding to the field in the data table of the external data source in the second data table, which can be, for example, the same field name; finally, the field mapping is performed through the determined field information. The specific method of field mapping is not limited in this application, and the mapping can be performed by the user issuing instructions in the visual operation interface, or the field mapping can be automatically implemented through the code. For example, there is a data table with a field name of "field 1" in the external data source, in which the data element described in "field 1" is stored. After the external data source is connected to the second data table, it is determined that there is a second data table of "field 1" in at least one second data table, and then "field 1" is mapped in the data table in the external data source and the second data table.

[0087] In one embodiment of the present application, the field mapping step in step S420 is described in detail: obtaining the first field information of a data table in a data source of external data and the second field information of at least one second data table, and matching the first field information and the second field information according to the editable item information to obtain a matching result; according to the matching result, mapping the first field information to at least one second data table to obtain a field mapping result.

[0088] It should be noted that when mapping fields, the field names of some external data are inconsistent with the names required in the second data table. For example, a field in the data table of the external data source is named "user_id", but when creating the field name in the second data table, the field name is required to be created in camel case. Then, when mapping fields, the underlined field name "user_id" needs to be changed to a camel case field name. For example, the field information can be modified through the database's as, Django's model, dictionary, etc.

[0089] S430. In response to the approved data synchronization task, synchronize the external data into at least one second data table according to the field mapping result and the preset data synchronization cycle and partition field to obtain at least one third data table.

[0090] Exemplarily, before synchronizing external data to the second data table, the user can pre-set the data synchronization cycle and partition field in the data development platform, and perform data synchronization according to the partition field within the preset data synchronization cycle. In this embodiment, the partition field is equivalent to the index established for the field. When the target field is used as a filter, the query range can be quickly located according to the established index, thereby speeding up the query efficiency of the chart data.

[0091] It should be noted that, in this embodiment, in response to the approved data synchronization task, when the external data is synchronized to the at least one second data table according to the field mapping results and the preset data synchronization cycle and partition fields, the external data in the same data table is only synchronized once to avoid confusion in the data table blood relationship.

[0092] In step S240, data in at least one third data table is processed to obtain a data processing result.

[0093] In one embodiment of the present application, visual dependencies are edited when scheduling data during the data processing process. The codes used by all modules in the scheduling process cannot perform table creation operations, and can only process at least one third data table to ensure that the data table information generated in the data development platform can only be developed through the data tables used for data synchronization or other tables generated by other scheduling, thereby achieving clear bloodline in the development process.

[0094] In one embodiment of the present application, the data processing step in step S240 is defined as follows:

[0095] In response to the scheduling task, scheduling at least one third data table according to a preset scheduling dependency;

[0096] Data in at least one scheduled third data table is extracted, and multi-dimensional data processing is performed on the extracted data to obtain a data processing result.

[0097] In this embodiment, the scheduling dependency can be obtained according to the response scheduling task, and part or all of at least one third data table is scheduled through the scheduling dependency, and then data is extracted and processed to obtain a data processing result. It should be noted that the data processing result in this embodiment may include a data processing table, and the data processing table stores all data obtained after data processing.

[0098] In one embodiment of the present application, after data processing is performed on the data in at least one third data table to obtain the data processing result, a step of displaying the data processing result and the data lineage relationship is further included, which is specifically described as follows:

[0099] Establish a data table blood relationship display module;

[0100] The data table lineage display module is used to visualize the data table lineage, scheduling dependencies, and data processing results. The data table lineage is determined during the execution of data synchronization tasks and scheduling tasks.

[0101] In this embodiment, the data table lineage, scheduling dependencies and data processing results are visualized, allowing users to provide feedback and adjustments to data table establishment specifications, data table approval specifications, editable item information, etc., so that the adjusted data processing platform can output more accurate data processing results.

[0102] For ease of understanding, in one embodiment of the present application, a specific example is used to illustrate the visualization data development method. Figure 5 As shown, Figure 5 This is an implementation architecture diagram of a visual data development method shown in an exemplary embodiment of the present application. First, establish data table 1 and data table 2 that meet the data table establishment specification; in response to data synchronization task A and data synchronization task B, synchronize external data in data table 1 and data table 2 to obtain synchronized data table 1 and synchronized data table 2; in response to scheduling task A, process the data in data table 1 and data table 2 to obtain processed table 1.

[0103] It should be noted that although the steps of the method in the present application are described in a specific order in the drawings, this does not require or imply that the steps must be performed in this specific order, or that all the steps shown must be performed to achieve the desired results. Additionally or alternatively, some steps may be omitted, multiple steps may be combined into one step, and / or one step may be decomposed into multiple steps, etc.

[0104] Figure 6 is a block diagram of a visual data development device shown in an exemplary embodiment of the present application. The device can be applied to Figure 1 The implementation environment shown is specifically configured in terminal devices 101, 102, and 103. The device may also be applicable to other exemplary implementation environments and specifically configured in other devices. This embodiment does not limit the implementation environment to which the device is applicable.

[0105] like Figure 6 As shown, the visual data development device includes a data table creation module 601, a data table approval module 602, a data synchronization module 603, and a data processing module 604.

[0106] The data table establishment module 601 is used to establish a plurality of first data tables in the visual data development interface according to a preset data table establishment specification, wherein the plurality of first data tables include editable item information.

[0107] The data table approval module 602 is used to remove redundant data tables from the plurality of first data tables according to a preset data table approval specification to obtain at least one second data table.

[0108] The data synchronization module 603 is used to obtain a data synchronization task and external data, and based on the data synchronization task, synchronize the external data into the at least one second data table according to the editable item information to obtain at least one third data table.

[0109] The data processing module 604 is used to process the data in the at least one third data table to obtain a data processing result.

[0110] It should be noted that the visualization data development device provided in the above embodiment and the visualization data development method provided in the above embodiment belong to the same concept, wherein the specific manner in which each module and unit performs operations has been described in detail in the method embodiment and will not be repeated here. In practical applications, the visualization data development device provided in the above embodiment can allocate the above functions to different functional modules as needed, that is, divide the internal structure of the device into different functional modules to complete all or part of the functions described above, and this is not limited here.

[0111] An embodiment of the present application also provides an electronic device, comprising: one or more processors; a storage device for storing one or more programs, when the one or more programs are executed by the one or more processors, the electronic device implements the visual data development method provided in the above-mentioned embodiments.

[0112] Figure 7 The structure diagram of the computer system suitable for implementing the electronic device of the embodiment of the present application is shown. It should be noted that: Figure 7 The computer system of the electronic device shown is only an example and should not bring any limitation to the functions and scope of use of the embodiments of the present application.

[0113] like Figure 7As shown, the computer system includes a central processing unit (CPU) 701, which can perform various appropriate actions and processes according to the program stored in the read-only memory (ROM) 702 or the program loaded from the storage part 708 to the random access memory (RAM) 703, such as executing the method described in the above embodiment. In RAM 703, various programs and data required for system operation are also stored. CPU 701, ROM 702 and RAM 703 are connected to each other through bus 704. Input / output (I / O) interface 705 is also connected to bus 704.

[0114] The following components are connected to the I / O interface 705: an input section 706 including a keyboard, a mouse, etc.; an output section 707 including a cathode ray tube (CRT), a liquid crystal display (LCD), etc., and a speaker, etc.; a storage section 708 including a hard disk, etc.; and a communication section 709 including a network interface card such as a LAN (Local Area Network) card, a modem, etc. The communication section 709 performs communication processing via a network such as the Internet. A drive 710 is also connected to the I / O interface 705 as needed. A removable medium 711, such as a magnetic disk, an optical disk, a magneto-optical disk, a semiconductor memory, etc., is installed on the drive 710 as needed so that a computer program read therefrom is installed into the storage section 708 as needed.

[0115] In particular, according to an embodiment of the present application, the process described above with reference to the flowchart can be implemented as a computer software program. For example, an embodiment of the present application includes a computer program product, which includes a computer program carried on a computer-readable medium, and the computer program includes a computer program for executing the method shown in the flowchart. In such an embodiment, the computer program can be downloaded and installed from a network through a communication section 709, and / or installed from a removable medium 711. When the computer program is executed by a central processing unit (CPU) 701, various functions defined in the system of the present application are executed.

[0116] It should be noted that the computer-readable medium shown in the embodiment of the present application can be a computer-readable signal medium or a computer-readable storage medium or any combination of the above two. The computer-readable storage medium can be, for example, an electrical, magnetic, optical, electromagnetic, infrared, or semiconductor system, device or device, or any combination of the above. More specific examples of computer-readable storage media can include, but are not limited to: an electrical connection with one or more wires, a portable computer disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM), a flash memory, an optical fiber, a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the above. In the present application, a computer-readable signal medium may include a data signal propagated in a baseband or as part of a carrier wave, wherein a computer-readable computer program is carried. This propagated data signal can take a variety of forms, including but not limited to an electromagnetic signal, an optical signal, or any suitable combination of the above. A computer-readable signal medium may also be any computer-readable medium other than a computer-readable storage medium, which may send, propagate or transmit a program for use by or in conjunction with an instruction execution system, apparatus or device. A computer program contained on a computer-readable medium may be transmitted using any appropriate medium, including but not limited to: wireless, wired, etc., or any suitable combination of the above.

[0117] The flowchart and block diagram in the accompanying drawings illustrate the possible architecture, functions and operations of the system, method and computer program product according to various embodiments of the present application. Wherein, each box in the flowchart or block diagram can represent a module, a program segment, or a part of the code, and the above-mentioned module, program segment, or a part of the code contains one or more executable instructions for realizing the specified logical function. It should also be noted that in some alternative implementations, the functions marked in the box can also occur in a different order from the order marked in the accompanying drawings. For example, two boxes represented in succession can actually be executed substantially in parallel, and they can sometimes be executed in the opposite order, depending on the functions involved. It should also be noted that each box in the block diagram or flowchart, and the combination of boxes in the block diagram or flowchart can be implemented with a dedicated hardware-based system that performs a specified function or operation, or can be implemented with a combination of dedicated hardware and computer instructions.

[0118] The units involved in the embodiments described in this application may be implemented by software or hardware, and the units described may also be set in a processor. The names of these units do not, in some cases, constitute limitations on the units themselves.

[0119] Another aspect of the present application provides a computer-readable storage medium having a computer program stored thereon, which, when executed by a processor of a computer, causes the computer to execute the visual data development method as described above. The computer-readable storage medium may be included in the electronic device described in the above embodiment, or may exist independently without being assembled into the electronic device.

[0120] Another aspect of the present application also provides a computer program product or a computer program, which includes computer instructions stored in a computer-readable storage medium. A processor of a computer device reads the computer instructions from the computer-readable storage medium, and the processor executes the computer instructions, so that the computer device executes the visualization data development method provided in each of the above embodiments.

[0121] The above embodiments are merely illustrative of the principles and effects of the present invention, and are not intended to limit the present invention. Anyone familiar with the technology may modify or change the above embodiments without violating the spirit and scope of the present invention. Therefore, all equivalent modifications or changes made by a person of ordinary skill in the art without departing from the spirit and technical ideas disclosed by the present invention shall still be covered by the claims of the present invention.

Claims

1. A visual data development method, characterized in that: The method comprises: Establishing a plurality of first data tables in the visual data development interface according to a preset data table establishment specification, wherein the plurality of first data tables include editable item information; According to a preset data table approval specification, redundant data tables are removed from the plurality of first data tables to obtain at least one second data table; Acquire a data synchronization task and external data, and based on the data synchronization task, synchronize the external data into the at least one second data table according to the editable item information to obtain at least one third data table; Performing data processing on the data in the at least one third data table to obtain a data processing result; The step of synchronizing the external data into the at least one second data table based on the data synchronization task and according to the editable item information to obtain at least one third data table includes: Remove redundant data synchronization tasks according to the preset data synchronization task approval specifications to obtain approved data synchronization tasks; Connecting the data source of the external data with the at least one second data table, and performing field mapping on the data table in the data source of the external data and the at least one second data table according to the editable item information to obtain a field mapping result; In response to the approved data synchronization task, the external data is synchronized to the at least one second data table according to the field mapping result and the preset data synchronization cycle and partition field to obtain the at least one third data table.

2. The visualization data development method according to claim 1, characterized in that: The step of establishing a plurality of first data tables in the visual data development interface according to a preset data table establishment specification includes: In response to the data development requirement instruction, acquiring editable item information required for establishing the plurality of first data tables, and presetting a data table establishment specification according to the editable item information; A plurality of first data tables are established in the visual data development interface according to a preset data table establishment specification.

3. The visualization data development method according to claim 1, characterized in that: In response to the approved data synchronization task, when the external data is synchronized to the at least one second data table according to the field mapping result and the preset data synchronization cycle and partition field, the external data in the same data table is synchronized only once.

4. The visualization data development method according to claim 1, characterized in that: The step of connecting the data source of the external data with the at least one second data table, and performing field mapping on the data table in the data source of the external data and the at least one second data table according to the editable item information to obtain a field mapping result includes: Acquire first field information of a data table in a data source of the external data and second field information of the at least one second data table, and match the first field information and the second field information according to the editable item information to obtain a matching result; According to the matching result, the first field information is mapped to the at least one second data table to obtain a field mapping result.

5. The visualization data development method according to claim 1, characterized in that: The performing data processing on the data in the at least one third data table to obtain a data processing result includes: In response to the scheduling task, scheduling the at least one third data table according to a preset scheduling dependency; Data in at least one scheduled third data table is extracted, and multi-dimensional data processing is performed on the extracted data to obtain a data processing result.

6. The visual data development method according to claim 5, characterized in that: After extracting data from at least one third data table scheduled, and performing multi-dimensional data processing on the extracted data to obtain a data processing result, the method includes: Establish a data table blood relationship display module; The data table lineage display module is used to visualize the data table lineage, scheduling dependency, and data processing results, wherein the data table lineage is determined during the execution of data synchronization tasks and scheduling tasks.

7. A visual data development device, characterized in that: The device comprises: A data table establishment module, used to establish a plurality of first data tables in the visual data development interface according to a preset data table establishment specification, wherein the plurality of first data tables include editable item information; A data table approval module, configured to remove redundant data tables from the plurality of first data tables according to a preset data table approval specification, to obtain at least one second data table; A data synchronization module, used for acquiring a data synchronization task and external data, and based on the data synchronization task, synchronizing the external data into the at least one second data table according to the editable item information, to obtain at least one third data table; A data processing module, used for processing the data in the at least one third data table to obtain a data processing result; The data synchronization module removes redundant data synchronization tasks according to preset data synchronization task approval specifications to obtain approved data synchronization tasks; connects the data source of the external data with the at least one second data table, and performs field mapping on the data table in the data source of the external data and the at least one second data table according to the editable item information to obtain a field mapping result; in response to the approved data synchronization task, synchronizes the external data to the at least one second data table according to the field mapping result and the preset data synchronization cycle and partition field to obtain the at least one third data table.

8. An electronic device, characterized in that: The electronic device comprises: one or more processors; A storage device for storing one or more programs, which, when executed by the one or more processors, enables the electronic device to implement the visual data development method as described in any one of claims 1 to 6.

9. A computer-readable storage medium, characterized in that: A computer program is stored thereon, and when the computer program is executed by a processor of a computer, the computer is caused to execute the visualization data development method according to any one of claims 1 to 6.

Images