Data processing method, device, electronic device and storage medium

Abstract

The present application provides a data processing method, device, electronic device and storage medium, the method comprising: when obtaining a data output request generated by a user operation page, obtaining link information related to the data output request; generating a query request using the link information and parameter information corresponding to the data output request through a query engine service, and performing a data query to a data source corresponding to the link information through the query request; parsing and outputting the queried data. The method can improve data processing efficiency.

Description

Technical Field

[0001] The present invention relates to the field of data processing technology, and in particular to a data processing method, device, electronic equipment and storage medium. Background Art

[0002] A data analysis platform hopes to be able to flexibly build data dashboards, efficiently respond to business changes, and have powerful visualization capabilities. Users and staff do not need to write cumbersome codes. By simply dragging and holding, they hope to create rich icons so that users can use data conveniently and simply.

[0003] The current data analysis platform outputs data through Kylin after data entry, cleaning, and preprocessing. The data analysis platform then merges and formats the output data and outputs it to the report.

[0004] To achieve the above process, users can contact data upload personnel or the scheduling system to connect the existing data source to the hive cluster through collection or scheduling import, and the data processing personnel can write aql to clean and merge the existing hive tables and finally generate the integrated data; configure dimension indicators and other information on the metadata management platform to preload the data into Kylin through scheduling. The data analysis platform obtains the data through the Kylin interface and merges and displays the data in the report.

[0005] The current data analysis platform has a single data source, is highly dependent on Kylin, has a long access link, and has low data processing efficiency. Summary of the invention

[0006] In view of this, the present application provides a data processing method, device, electronic device and storage medium, which can improve data processing efficiency.

[0007] To solve the above technical problems, the technical solution of this application is implemented as follows:

[0008] In one embodiment, a data processing method is provided, the method comprising:

[0009] When a data output request generated by a user operation page is obtained, link information related to the data output request is obtained;

[0010] Generate a query request by using the link information and parameter information corresponding to the data output request through a query engine service, and perform data query on a data source corresponding to the link information through the query request;

[0011] Parse and output the queried data.

[0012] The step of generating a query request by using the link information and parameter information corresponding to the data output request through a query engine service includes:

[0013] Convert the parameter information corresponding to the data output request into parameter information in an input parameter format that meets the requirements of the query engine;

[0014] The query request is generated by using the link information and the parameter information in the input parameter format that meets the query engine requirements through the query engine service.

[0015] The data source is a data file in a local database and / or a remote database.

[0016] Wherein, when the data source is a data file stored in a local database, the data file is uploaded via a data import service dynamic relay protocol DTP and stored in the local database;

[0017] When the data source is a remote database, the method further includes: configuring link information of the database.

[0018] Wherein, the method further comprises:

[0019] Get the data output mode of the user input;

[0020] The step of parsing and outputting the queried data includes:

[0021] The queried data is parsed and outputted in accordance with the acquired data output mode.

[0022] Before querying the data source corresponding to the link information through the query request, the method further comprises:

[0023] The authority information corresponding to the user and the authority information of the data corresponding to the data output request are obtained, and data query is performed to the data source corresponding to the link information through the query request according to the obtained authority information.

[0024] In another embodiment, a data processing device is provided, the device comprising: an acquisition unit, a query engine unit, and an output unit;

[0025] The acquisition unit is used to acquire link information related to the data output request when acquiring the data output request generated by the user operating the page;

[0026] The query engine unit is used to generate a query request using the link information obtained by the acquisition unit and the parameter information corresponding to the data output request through a query engine service, and perform a data query to the data source corresponding to the link information through the query request;

[0027] The output unit is used to parse and output the data queried by the query engine unit.

[0028] in,

[0029] The acquisition unit is further used to convert the parameter information corresponding to the data output request into parameter information in an input parameter format that meets the requirements of the query engine;

[0030] The query engine unit is specifically used to generate a query request through the query engine service using the link information and the parameter information corresponding to the data output request, and to generate the query request through the query engine service using the link information and the parameter information in the input parameter format that meets the query engine requirements after conversion by the acquisition unit.

[0031] The data source is a data file in a local database and / or a remote database.

[0032] Wherein, the device further comprises: an uploading unit and a configuration unit;

[0033] The uploading unit is used for uploading the data file through the data import service dynamic relay protocol DTP and storing it in the local database when the data source is a data file stored in the local database;

[0034] The configuration unit is used to configure the link information of the database when the data source is a remote database.

[0035] in,

[0036] The acquisition unit is further used to acquire the data output mode input by the user;

[0037] The output unit is further used to parse the queried data and output it in accordance with the data output method obtained by the obtaining unit.

[0038] in,

[0039] The acquisition unit is further used to obtain the permission information corresponding to the user and the permission information of the data corresponding to the data output request before the query engine unit performs data query on the data source corresponding to the link information through the query request, and trigger the query engine unit to perform data query on the data source corresponding to the link information through the query request according to the obtained permission information.

[0040] In another embodiment, an electronic device is provided, including a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor implements the steps of the data processing method when executing the program.

[0041] In another embodiment, a computer-readable storage medium is provided, on which a computer program is stored, and when the program is executed by a processor, the steps of the data processing method are implemented.

[0042] As can be seen from the above technical solution, in the above embodiment, the query engine service is called, the link information and parameter information corresponding to the data output request issued by the user are used to generate a query request, and the corresponding data is queried from the data source corresponding to the link information and analyzed and output to the user. This solution can improve data processing efficiency. BRIEF DESCRIPTION OF THE DRAWINGS

[0043] In order to more clearly illustrate the technical solutions in the embodiments of the present application, the drawings required for use in the description of the embodiments will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present application. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying any creative labor.

[0044] Figure 1 A schematic diagram of a data analysis platform architecture provided in an embodiment of the present application;

[0045] Figure 2 This is a schematic diagram of the data processing flow in the embodiment of the present application;

[0046] Figure 3 A schematic diagram of a permission configuration method in an embodiment of the present application;

[0047] Figure 4 This is a schematic diagram of another permission configuration method in an embodiment of the present application;

[0048] Figure 5 A schematic diagram of selecting a data source type provided in an embodiment of the present application;

[0049] Figure 6 A schematic diagram of information related to data connection established in an embodiment of the present application;

[0050] Figure 7 The content displayed after selecting a data connection in the embodiment of the present application;

[0051] Figure 8 A schematic diagram of selecting a chart type given in an embodiment of the present application;

[0052] Fig. 9 This is a schematic diagram of an interface for issuing a data output request in an embodiment of the present application;

[0053] Fig.10 This is a schematic diagram of displaying output data in a general chart format in an embodiment of the present application;

[0054] Fig.11 This is a schematic diagram of the structure of a device applied to the above technology in an embodiment of the present application;

[0055] Fig.12 A schematic diagram of the physical structure of an electronic device provided by an embodiment of the present invention. DETAILED DESCRIPTION

[0056] The following will be combined with the drawings in the embodiments of the present application to clearly and completely describe the technical solutions in the embodiments of the present application. Obviously, the described embodiments are only part of the embodiments of the present application, not all of the embodiments. Based on the embodiments in the present application, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of this application.

[0057] The terms "first", "second", "third", "fourth", etc. (if any) in the specification and claims of the present invention and the above-mentioned drawings are used to distinguish similar objects, and are not necessarily used to describe a specific order or sequence. It should be understood that the data used in this way can be interchangeable where appropriate, so that the embodiments of the present invention described herein can be implemented in an order other than those illustrated or described herein, for example. In addition, the terms "including" and "having" and any variations thereof are intended to cover non-exclusive inclusions. For example, a process, method, system, product, or device that includes a series of steps or units is not necessarily limited to those steps or units that are clearly listed, but may include other steps or units that are not clearly listed or inherent to these processes, methods, products, or devices.

[0058] The technical solution of the present invention is described in detail with specific embodiments below. The following specific embodiments can be combined with each other, and the same or similar concepts or processes may not be described in detail in some embodiments.

[0059] The embodiment of the present application provides a data processing method, which is applied to a data analysis platform, calls a query engine service, generates a query request using the link information and parameter information corresponding to the data output request issued by the user, queries the corresponding data source corresponding to the link information, analyzes and outputs the data to the user. This solution can improve data processing efficiency.

[0060] The data analysis platform in the embodiment of the present application can be composed of one device or multiple devices. For the data import system, data storage system, metadata management platform, query engine system, UC service, approval flow service and other systems involved in the data analysis platform, if they are developed separately for the data analysis platform, they can be used as part of the data analysis platform. If there is a part on the related system that serves the data analysis platform separately, the part that serves the data analysis platform separately belongs to the data analysis platform; if the data platform only calls the information of related systems, such as the authority system, approval flow, UC, etc., these are not included in the data analysis platform, and only the relevant information needs to be called.

[0061] See also Figure 1 , Figure 1 This is a schematic diagram of a data analysis platform architecture provided in an embodiment of the present application. Figure 1 The data analysis platform includes a data processing system, a QE system (query engine system), a data import system, and an indicator system. The metadata management platform may or may not exist and is used to integrate user uploaded data into the Hive cluster for big data analysis. The indicator system stores data files uploaded by users through DTP. In the specific implementation process, the indicator system may include local data files or databases corresponding to remote data sources.

[0062] The data analysis platform also relies on some external systems, such as scheduling systems, authority systems, approval flows, UC and other systems.

[0063] The various systems on the data analysis platform and the various systems called by the data analysis platform work together. The implementation method of batch authorization in a collective manner can help users allocate resources more flexibly and conveniently. In the existing data analysis platform, the data demand output work and data presentation are all facing different groups. How to effectively allocate the current data resources for collaborative work and present the description of collaborative work to different users has become our problem. Here we provide this implementation method of batch authorization in a collective manner.

[0064] This method of implementing batch authorization in a collaborative working mode in a collective manner can help users be more flexible when authorizing their own resources to work groups working on different systems, and the allocation method is more flexible:

[0065] First, we provide users with a way to create resource collections on the data analysis platform. Users provide a way to create resources and collections. First, we need to create a table to store the mapping relationship between the user's resources and the user's newly created resources. When used for the first time, a collection of all resources will be created for the resources currently provided by the user, which is equivalent to the user authorizing all resources. In addition, users can configure additional resource collections and authorize the collection to the collaborative workgroup they define.

[0066] Secondly, the entire collection is authorized to the collaborative workgroup; after the user defines it, the unique identification id of this collection is generated, and the relationship between the id and the collaborative workgroup is authorized to the permission system.

[0067] Finally, when obtaining permissions, you can find the authorized resources based on the workgroup the current accessor is in; when a user obtains a resource, it will determine whether the user is in the workgroup, and at the same time, it can determine whether the collection contains the resources the user wants.

[0068] Figure 1 This is only a schematic diagram of the data analysis platform architecture. The specific implementation of this application is not limited to Figure 1 The architecture diagram of the data analysis platform shown is shown.

[0069] In specific implementation, the data analysis platform can also call some file classification tables, metadata processing tables, model building systems, etc. according to actual applications.

[0070] The data import system in the embodiment of the present application can be implemented using DTP; the indicator system is used to store data files imported by users into the data analysis platform, as well as connection information of each data source as a database.

[0071] The diversity of data sources in the embodiments of the present application may be data uploaded to the data analysis platform, i.e., data files in a local database, or data in a remote database, or data in a remote database and on a data platform, where the remote here refers to data on devices outside the data platform.

[0072] The two types of data sources are associated with this data analysis platform in different ways, as follows:

[0073] For data files, such as excel files, csv files, etc.:

[0074] The data file is uploaded through the data import service DTP and stored in the hive cluster.

[0075] When uploading, first upload to the cloud disk, and then write to the self-storage hive cluster through scheduling. You can also synchronize the data file to the hive cluster of the metadata management platform at the same time for big data analysis and other processing.

[0076] In the embodiment of the present application, when a user uploads a file using the data analysis platform, table partition information of the data uploaded by the user is generated, and the table partition information is maintained according to the data upload result. When obtaining the table partition information of the data, the solution no longer needs to obtain limited table partition information from the bottom layer of the data cluster, but can obtain the table partition information of the data when the data is uploaded, and set the fields corresponding to the table partition information to be obtained as needed. Therefore, the solution can obtain the table partition information of the data simply, efficiently and comprehensively.

[0077] The specific process of maintaining table partition information is as follows:

[0078] The first step is to record the table partition information corresponding to the data according to the update type of the uploaded data when the user uploads the data.

[0079] The data update types in the embodiment of the present application include: new addition type, partial replacement type, full replacement type and append type.

[0080] When users upload data, the update type will be set. The data analysis platform determines the data update type based on the user's settings, and also sets the replaced data version for partial replacement types and full replacement types.

[0081] The data table partition information includes at least the following fields but is not limited to the following fields. Other fields can be added according to actual needs:

[0082] Table identifier, used to identify different table partition information;

[0083] Data version, also known as table partition, is used to record the version number of uploaded data. The version number can be named according to preset rules, such as time naming;

[0084] Update type, used to identify the type of data currently being uploaded;

[0085] Data import status, used to identify the current state of data import, including: uploading, failure, success, where failure and success indicate the upload result. In specific implementation, specific character identifiers such as 0, 1, etc. can be used;

[0086] Partition status is used to indicate whether the table partition information of the current data is valid, including: valid or invalid.

[0087] For all replacement types, a field for the source data version needs to be added to identify the data version replaced by the currently uploaded data.

[0088] The second step is to maintain the table partition information according to the data upload result.

[0089] The data upload results are divided into failure and success. Failed upload results need to be rolled back accordingly.

[0090] The table partition information of the data in the embodiment of the present application can be maintained in the business layer MySQL library.

[0091] In the above scheme, by recording the table partition information of the data when uploading the data and maintaining the table partition information according to the data uploading result, the table partition information of the data can be obtained conveniently, quickly and comprehensively as needed.

[0092] The following is the maintenance process of data table partition information for different update types.

[0093] The specific steps for maintaining table partition information when the update type is a new type are:

[0094] Step 1: When the user uploads data and determines that the update type is a new type, a new data table partition information is added.

[0095] The table partition information includes table identifier, data version, update type, data import status and partition status;

[0096] The table identifier is given according to the table identifier naming rules for table partitions on the data analysis platform, such as ascending order, the data version can be the current period, the update type is set to the new type, and the partition status is not set first;

[0097] During the data uploading process, the data import status is set to uploading.

[0098] Step 2: When the data uploading is completed, determine the uploading result of the data.

[0099] Step 3: When the upload result is successful, the partition import status is updated to success, and the partition status is set to valid, and the process ends.

[0100] Step 4: When the upload result is failure, the partition import status is updated to failure, and the partition status is set to invalid.

[0101] At this point, the table partition information of the newly added data type has been updated.

[0102] The specific steps for maintaining table partition information when the update type is partial replacement are:

[0103] In the first step, when the user uploads data and determines that the update type is a partial replacement type, the data import status of the table partition information of the data corresponding to the replaced data is updated to uploading, and the update type is updated to a partial replacement type.

[0104] Step 2: If there are fields included in the uploaded data that are not included in the fields included in the table partition information of the replaced data, then the fields are added to the table partition information and the newly added fields are designated and marked.

[0105] Step 3: When the data uploading is completed, determine the uploading result of the data.

[0106] Step 4: When the upload result is successful, the partition import status is updated to success, and the specified mark of the newly added field is deleted, and this process ends.

[0107] Step 5: When the upload result is failure, delete the newly added field and update the upload status to failure.

[0108] At this point, the table partition information of the data of the partial replacement type is updated.

[0109] The specific steps for maintaining table partition information when the update type is the full replacement type are:

[0110] Step 1: When the user uploads data and determines that the update type is a complete replacement type, the partition status in the table partition information corresponding to the replaced data is updated to invalid, and a new data table partition information is added.

[0111] For the newly added table partition information, set the partition import status to uploading, record the data version and source data version, and set the update type to replace all.

[0112] The fields included for table partition information also include:

[0113] Table ID, filled in according to the naming rules of the ID of each table partition information, such as ascending sequence.

[0114] The partition status is not set yet;

[0115] The data version records the version of the currently uploaded data; the source data version records the data version of the data that is completely replaced.

[0116] Step 2: If there are new fields in the table partition information relative to the fields of the data to be completely replaced, the new fields are added to the new table partition information.

[0117] Step 3: When the data uploading is completed, determine the uploading result of the data.

[0118] Step 4: When the upload result is successful, the partition import status in the table partition information of the newly added data is updated to success, and the partition status is set to valid, and this process ends.

[0119] Step 5: When the upload result is failure, the partition import status in the newly added table partition information is updated to failure, the partition status is set to valid, and the partition status in the table partition information corresponding to the replaced data is updated to valid.

[0120] The table partition information corresponding to the replaced data is determined according to the source data version in the current table partition information.

[0121] Step 6: If there are new fields in the newly added table partition information relative to the table partition information of the data to be completely replaced, the new fields are deleted.

[0122] At this point, the table partition information of the data of the complete replacement type is updated.

[0123] The specific steps for maintaining table partition information when the update type is append type are:

[0124] In the first step, when the user uploads data and determines that the update type is an append type, a new data table partition information is added; wherein, during the data uploading process, the data import status is uploading.

[0125] The table partition information includes: data version, data import status, partition status, and newly added fields; the partition status information is not set for the time being;

[0126] The table partition information also includes a table identifier.

[0127] Step 2: When the data uploading is completed, determine the uploading result of the data.

[0128] Step 3: When the upload result is successful, the partition import status in the table partition information of the newly added data is updated to success, and the partition status is set to valid, and this process ends.

[0129] Step 4: When the upload result is failure, the partition import status in the table partition information of the newly added data is updated to failure, the newly added field is deleted, and the partition status is set to invalid.

[0130] At this point, the table partition information of the appended data is updated. The main difference between uploading this type of data and uploading the new type of data is that a new field is added. The process of maintaining the table partition information is similar.

[0131] This embodiment records the table partition information of the data in the business layer MySQL database according to the data update type when uploading data, and maintains the table partition information according to the upload result when uploading data. The field information in the table partition information can be added as needed to expand the relevant information of the data operation. This solution realizes the business of the underlying centralized information by demarcating the table partition information of the data at the business layer, facilitating business expansion and reducing the query intensity of dependence on cluster partition information.

[0132] This solution completes the maintenance of table partition information for user uploaded data, and at the same time realizes the expansion of maintenance information at the business layer through the expansion of a single table. At the same time, the maintenance of various states of table partition information realizes the traceability rollback of table partition information.

[0133] Since the services on the current data analysis platform will perform double-write operations on user data, different success types can be identified by marking the status field of the table partition information.

[0134] Based on the same inventive concept, the embodiment of the present application also provides a data table partition information maintenance device, the device comprising: an upload unit and a maintenance unit;

[0135] The uploading unit is used for users to upload data;

[0136] The maintenance unit is used to record the table partition information corresponding to the data according to the update type of the uploaded data when the uploading unit uploads data by the user; and maintain the table partition information according to the uploading result of the data uploaded by the uploading unit.

[0137] Preferably,

[0138] The maintenance unit is specifically used to add a new piece of table partition information of data when the update type is a new type, and record the data version, update type, data import status and partition status in the table partition information; wherein, during the data upload process, the data import status is uploading; when the upload result of the data uploaded by the upload unit is successful, the partition import status is updated to success, and the partition status is set to valid; when the upload result of the data uploaded by the upload unit is failure, the partition import status is updated to failure, and the partition status is set to invalid.

[0139] Preferably,

[0140] The maintenance unit is specifically used to update the data import status of the table partition information corresponding to the replaced data to uploading when the update type is a partial replacement type, and update the update type to a partial replacement type; if the fields included in the uploaded data are not included in the fields included in the table partition information of the replaced data, then add the fields to the table partition information and specify the newly added fields; when the upload result of the data uploaded by the upload unit is successful, update the partition import status to success and delete the specified mark of the newly added fields; when the upload result of the data uploaded by the upload unit is failure, delete the newly added fields and update the upload status to failure.

[0141] Preferably,

[0142] The maintenance unit is specifically used to update the partition status in the table partition information corresponding to the replaced data to invalid when the update type is complete replacement, add a new piece of table partition information of the data, set the partition import status to uploading, record the data version and the source data version, and set the update type to complete replacement type; if there are new fields in the table partition information of the replaced data, add the new fields in the newly added table partition information; when the upload result of the data uploaded by the upload unit is successful, update the partition import status in the table partition information of the newly added data to success, and set the partition status to valid; when the upload result of the data uploaded by the upload unit is failed, update the partition import status in the newly added table partition information to failure, set the partition status to valid, and update the partition status in the table partition information corresponding to the replaced data to valid; if there are new fields in the newly added table partition information relative to the table partition information of the replaced data, delete the new fields.

[0143] Preferably,

[0144] The maintenance unit is specifically used to add a new piece of table partition information of data when the update type is an append type, and record the data version, data import status, partition status, and newly added fields in the table partition information; wherein, during the data upload process, the data import status is uploading; when the upload result of the data uploaded by the upload unit is successful, the partition import status in the table partition information of the newly added data is updated to success, and the partition status is set to valid; when the upload result of the data uploaded by the upload unit is failure, the partition import status in the table partition information of the newly added data is updated to failure, the newly added field is deleted, and the partition status is set to invalid.

[0145] Preferably, the device further comprises: a setting unit;

[0146] The setting unit is used to set the update type of the uploaded data when the user uploads data;

[0147] The maintenance unit is further used to determine the update type corresponding to the current uploaded data according to the update type set by the setting unit.

[0148] The units in the above-mentioned embodiments may be integrated into one body or deployed separately; they may be combined into one unit or further divided into multiple sub-units.

[0149] In another embodiment, an electronic device is provided, including a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein when the processor executes the program, the steps of the method for maintaining table partition information of the data are implemented.

[0150] In another embodiment, a computer-readable storage medium is provided, on which a computer program is stored, and when the program is executed by a processor, the steps of the method for maintaining table partition information of data are implemented.

[0151] For databases such as Mysql, Presto, etc.:

[0152] You only need to configure the link information of the corresponding database on the data analysis platform, that is, the query path of the database, and then you can directly query the data when you need to.

[0153] When preparing a data source, a data connection will be established, and the name of the data source is defined by the user. When the data source is prepared, the structure of the data source and link information will be stored locally for the name of the data connection, so as to operate the data connection and query the data source.

[0154] Through the above method, the data source is prepared. For each user, the data source and the data analysis platform can be associated in advance, or the data source and the data analysis platform can be associated when data processing is needed.

[0155] The process of implementing data processing in the embodiments of the present application is described in detail below with reference to the accompanying drawings.

[0156] See also Figure 2 , Figure 2 This is a schematic diagram of the data processing flow in the embodiment of this application. The specific steps are:

[0157] Step 201: when a data output request generated by a user operating a page is obtained, link information related to the data output request is obtained.

[0158] The data output request generated in the embodiment of the present application refers to a request to obtain relevant data from the data source for analysis and output. For some local operation requests, a direct response can be made.

[0159] When a user operates a parameter in a structure of a data source corresponding to a data connection, if the operation generates a data output request, then link information corresponding to the data connection, ie, link information related to the data output request, is acquired.

[0160] The connection information obtained here can be one or more, that is, the user can operate one data connection or multiple data connections at the same time.

[0161] Step 202: Generate a query request by using the link information and parameter information corresponding to the data output request through a query engine service, and perform data query on a data source corresponding to the link information through the query request.

[0162] In this step, the query engine service uses the link information and the parameter information corresponding to the data output request to generate a query request, including:

[0163] Convert the parameter information corresponding to the data output request into parameter information in an input parameter format that meets the requirements of the query engine;

[0164] The query request is generated by using the link information and the parameter information in the input parameter format that meets the query engine requirements through the query engine service.

[0165] In specific implementation, the query engine service can be implemented using the QE service.

[0166] QueryEngine (QE) is the glue of the data pyramid. It is a highly available SQL-like query service that aims to provide users with an advanced data query language so that users can express their data query and processing logic efficiently, simply and flexibly.

[0167] The process of generating a query request here is to generate an SQL statement using the link information and the parameter information after the conversion format, and perform data query on the data source corresponding to the link information.

[0168] When the corresponding data source is found through link information, the parameter information is used as a matching item to query the data matching the parameter information; if there are multiple link information, then in the query request, the parameter information corresponding to each link information is also required so that the corresponding data can be queried in the corresponding data source.

[0169] Step 203: parse and output the queried data.

[0170] When the present application is implemented specifically, before a data output request is made or before data output is made, the method further includes:

[0171] Get the data output mode of the user input;

[0172] Users will input the desired data output method, such as bar charts, general icons, etc., through the operation interface of the data analysis platform.

[0173] This step parses and outputs the queried data, including:

[0174] The queried data is parsed and outputted in accordance with the acquired data output mode.

[0175] The current data analysis platform will initiate a data query to the address of the data source specified in the data connection used by the user-configured report. Therefore, the complexity of the user's data source is usually only known to the user; however, if the data query times out due to the complexity of the user-defined data source, it will eventually be fed back to the data analysis platform.

[0176] Currently, data query status can only be fed back to the user, who can then initiate data query again after processing by the user himself or waiting for the query engine to release enough resources. This data query method is inefficient and has a poor user experience.

[0177] A data query method is provided in an embodiment of the present application and is applied to a data analysis platform. The data analysis platform has its own self-storage system, referred to as a data cluster hereinafter, and a hive cluster that can call data.

[0178] For some preset data tables, such as those corresponding to user query timeouts and / or some specific reports, the data tables are synchronized from the hive cluster to the data cluster of the data analysis platform, and these data tables are marked and the corresponding cluster addresses (storage addresses on the data analysis platform) are added. If the query of these data tables is involved when querying data, the newly added cluster address is used for query. This implementation solution can help users speed up their queries without the users noticing, thereby improving the efficiency of data queries, improving the user experience on the data analysis platform, ensuring the service efficiency and stability of the data analysis platform, and reducing the pressure on the hive cluster.

[0179] The specific steps of the data query process in the embodiment of the present application are:

[0180] The first step is to obtain the preset data table in the Hive cluster and synchronize it to the data cluster of the data analysis platform.

[0181] The way to obtain the preset data table in the Hive cluster is as follows:

[0182] The data table corresponding to the specific data report set;

[0183] and / or,

[0184] Get the data table where the query request failed.

[0185] The data tables corresponding to the set specific data report may be all data tables or part of data tables corresponding to the report set as the focus, and the implementer may set it according to rules or experience;

[0186] The data table of failed query requests is a data table of failed query requests collected on the data analysis platform when the technical solution provided by the present application is not used.

[0187] When obtaining the preset data table in the hive cluster, the data table is synchronized to the data cluster of the data analysis platform, that is, the data cluster of the data analysis platform itself; the synchronization here means copying a copy to the data cluster of the data analysis platform, and the data table in the hive cluster does not change; that is to say, the storage method, structure, content, etc. of the hive cluster data will not be changed.

[0188] In the embodiment of the present application, it is considered that the preset data table may change, which requires setting a synchronization time; the synchronization time here can be set to be periodic, or a specific synchronization time can be given according to actual needs.

[0189] When the synchronization time arrives, the preset data table is obtained and synchronized.

[0190] You can also obtain the preset data table when the synchronization time arrives, and determine whether the currently obtained preset data table has changed. If it has changed, update the synchronized data table; if it is determined that the currently obtained preset data table has not changed, the data table will not be synchronized when the synchronization time arrives.

[0191] The second step is to mark the data table and record the query address corresponding to the data table.

[0192] The query address is the cluster address where the data table is stored on the data analysis platform.

[0193] Mark the synchronized data table, such as the table ID corresponding to the synchronization ID, indicating that the data table does not need to be queried using the cluster address of the default data cluster, but needs to be queried using the cluster address of the data analysis platform corresponding to the data table;

[0194] And the storage address of the corresponding data table is given for the synchronized data table to facilitate data query. Here, a corresponding query address can be given for each data table, or a query address for all data tables can be specified, and then the query data table can be distinguished by the identifier of the data table.

[0195] If data table 1, data table 2 and data table 3 are synchronized, data table 1 and data table 3 are marked and corresponding query addresses are given. In specific implementation, a synchronization table can also be used to represent the relevant content corresponding to the synchronized data tables, such as Table 1.

[0196] Table Identification Query address Data Table 1 Query address 1 Data Table 2 Query address 2 Data Table 3 Query address 3

[0197] Table 1

[0198] In Table 1, data table 1, data table 2 and data table 3 are marked by being added to the synchronized data table. The query address 1, query address 2 and query address 3 in Table 1 may be the same or different.

[0199] Step 3: When a data query request is received, if it is determined that there is a marked data table in the data table corresponding to the queried data, the query address is used to query the marked data table.

[0200] Use the default Hive cluster address to query the data table that does not contain the mark in the data table corresponding to the queried data.

[0201] The specific processing steps when receiving a data query request are as follows:

[0202] The first step is to obtain the data table identifier queried by the data query request upon receiving the data query request.

[0203] Step 2: For the marked data table, use the query address corresponding to the data table to query, and return the query result to end this process.

[0204] Step 3: For unlabeled data tables, use the default hive cluster address to query and return the query results.

[0205] This concludes the processing of the data query request.

[0206] To summarize, this application obtains preset data tables by analyzing the data corresponding to the report configured by the user, and isolates these preset data tables from the data in the hive cluster and stores them in the cluster of the data analysis platform. In this way, when the user queries the data, the cluster address of the data table stored on the data platform is used to query the preset data tables, which can quickly query the data and reduce the pressure on the underlying hive cluster query, thereby improving the data query efficiency and improving the user experience.

[0207] Based on the same inventive concept, an embodiment of the present application further provides a data query device, which is applied to a data analysis platform. The device includes: an acquisition unit, a synchronization unit, a receiving unit, and a query unit;

[0208] The acquisition unit is used to acquire a preset data table in the hive cluster;

[0209] The synchronization unit is used to synchronize the data table acquired by the acquisition unit to the data cluster of the data analysis platform; mark the data table and record the query address corresponding to the data table;

[0210] Preferably,

[0211] The query address is the cluster address where the data table is stored on the data analysis platform;

[0212] The receiving unit is used to receive a data query request;

[0213] The query unit is used for querying the data table using the corresponding query address when the receiving unit receives the data query request and if it is determined that there is a marked data table in the data table corresponding to the queried data.

[0214] Preferably,

[0215] The preset data table is: a data table corresponding to a set specific data report; and / or a data table for which a query request fails.

[0216] Preferably,

[0217] The synchronization unit is further used to set a synchronization time; when the synchronization time arrives, synchronize the preset data table currently acquired by the acquisition unit;

[0218] The acquisition unit is further configured to acquire a preset data table when the synchronization time set by the synchronization unit arrives.

[0219] Preferably,

[0220] The query unit is further used to query using a default hive cluster address for a data table corresponding to the queried data and for which no mark exists.

[0221] The units in the above-mentioned embodiments may be integrated into one body or deployed separately; they may be combined into one unit or further divided into multiple sub-units.

[0222] In another embodiment, an electronic device is provided, including a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor implements the steps of the data query method when executing the program.

[0223] In another embodiment, a computer-readable storage medium is provided, on which a computer program is stored, and when the program is executed by a processor, the steps of the data query method are implemented.

[0224] Preferably, in order to ensure the security of data permissions in the embodiment of the present application, before performing a data query to the data source corresponding to the link information through the query request, the method further includes:

[0225] The authority information corresponding to the user and the authority information of the data corresponding to the data output request are obtained, and data query is performed to the data source corresponding to the link information through the query request according to the obtained authority information.

[0226] The obtained permission information is obtained from the permission system by calling the permission service. The permission information here includes browsing permissions, data rows and permissions, etc.

[0227] Before describing the implementation process of row-level rights setting, let's first clarify the relationship between the following names:

[0228] 1. The relationship between data reports, data tables and data sources:

[0229] A user can configure any data report;

[0230] The data report contains n charts, and the data presented in each chart is derived from a data table in a unique data source, and each data table includes one or more data fields.

[0231] 2. The relationship between visitor attributes and chart attributes:

[0232] A visitor has many attributes in the user center, called visitor attributes, including the city, department, rank, etc.

[0233] The data produced by a chart comes from multiple fields in a data table, which are called data table fields.

[0234] The data corresponding to a data field in a chart is called accessed data.

[0235] The specific process of setting row-level permissions is given below:

[0236] The embodiments of the present application provide methods for setting permissions for data fields, including: the first method, and / or the second method. The process of setting each permission is given below:

[0237] The first one:

[0238] Provide configurable visitor attributes for the data fields corresponding to the data report. The specific steps are as follows:

[0239] The first step is to obtain a visitor attribute identifier and a data field carried in the first permission setting request when it is determined to obtain the first permission setting request.

[0240] In the embodiment of the present application, an interface for setting permissions is provided to the user, so that the user can input a permission setting request through the interface.

[0241] In the embodiment of the present application, taking the first permission setting request input by a user as an example, the implementation of judging whether the permission setting request input by the user and obtained by the data analysis platform is the first permission setting request is specifically as follows:

[0242] When the obtained permission setting request carries the visitor attribute identifier and the data field, it is determined that the permission setting request is the first permission setting request.

[0243] In a specific implementation, a specific field may be carried in the permission setting request input by the user to identify the current request as the first permission setting request.

[0244] See also Figure 3 , Figure 3 A schematic diagram of a permission configuration method in an embodiment of the present application.

[0245] Figure 3 When selecting the general configuration permission configuration interface, users are provided with selectable data source tables, data fields, and visitor attributes;

[0246] like Figure 3 The data source table selected is Demonstration Flow-Sheet1, the selected data field is "a", and the selected visitor attribute is city code;

[0247] After completing the above information selection, simply click Add Association to enter the first permission setting request.

[0248] Figure 3 This is just an example of permission configuration. The specific implementation is not limited to Figure 3 The input of a first permission settings request is shown.

[0249] Step 2: Generate and store a first permission record; wherein the first permission record includes a permission field and a controlled field; the permission field is the visitor attribute identifier, and the controlled field is the data field.

[0250] The permission record generated by the received first permission setting request will be marked and recorded as the first permission record.

[0251] Mark different permission records so that you can obtain the corresponding controlled scope later.

[0252] The permission record indicates that the controlled field (a) is controlled by the permission field. The effect after setting the permission is: if the city where user A, who is a visitor, is located is Beijing, then the data corresponding to the data field (controlled field) a that is accessed is only the content corresponding to Beijing.

[0253] The generated permission records can be saved locally or in the permission system.

[0254] When storing the corresponding permission record, a permission query identifier is generated for the permission record, specifically:

[0255] Generate an authority query identifier using an identifier of a data source table where the data field is located, an identifier of a data source corresponding to the data source table, and an identifier of a data report corresponding to the data source;

[0256] The authority query identifier is used to identify the authority record corresponding to the data field.

[0257] The above scheme realizes the permission setting that the access of the corresponding data field is controlled by the set visitor attributes by setting the visitor attributes corresponding to the data fields one by one. This scheme can actively, conveniently and flexibly set access permissions according to the user's own needs, thereby improving the efficiency of permission control settings and user experience.

[0258] Second type:

[0259] Pre-set visitor attribute templates for data fields; used to determine the personalized customization of visitor attribute controls;

[0260] For example, if the user's visitor attribute city is Beijing, the user can access the data content of Beijing, Shanghai and other places in the data field city.

[0261] The specific steps for providing the corresponding visitor attribute template for the data fields corresponding to the data report are as follows:

[0262] The first step is to obtain a data field and a visitor attribute template identifier carried in the second permission setting request when determining to obtain the second permission setting request.

[0263] In the embodiment of the present application, an interface for setting permissions is provided to the user, so that the user can input a permission setting request through the interface.

[0264] In the embodiment of the present application, taking the second permission setting request input by the user as an example, the implementation of judging whether the permission setting request input by the user and obtained by the data analysis platform is the second permission setting request is specifically as follows:

[0265] When the obtained permission setting request carries the visitor attribute template identifier and the data field, it is determined that the permission setting request is a second permission setting request.

[0266] In a specific implementation, a specific field may be carried in the permission setting request input by the user to identify the current request as the second permission setting request.

[0267] See also Figure 4 , Figure 4 A schematic diagram of another permission configuration method in an embodiment of the present application.

[0268] Figure 4 When selecting the template configuration permission configuration interface, the user is provided with selectable data source tables and data fields; although the template display box is displayed on the interface, it is pre-set and cannot be selected here. It is only displayed for users to browse, such as province-city row-level permissions (code); you can also select province-city row-level permissions (name) and store owners. You can set them according to actual needs during specific implementation, not limited to Figure 4 The given template content.

[0269] like Figure 4 The data source table selected is Demonstration Flow-Sheet1, and the data field selected is "city-code". The permission template logo configured for this field is automatically displayed, and the user cannot select it.

[0270] After completing the above information selection, simply click the OK button to enter the second permission setting request.

[0271] Figure 4 This is just an example of permission configuration. The specific implementation is not limited to Figure 4 The input of a second permission setting request is shown.

[0272] Step 2: Generate and store a second permission record; wherein the second permission record includes a permission field and a controlled field; the permission field is the visitor attribute template identifier, and the controlled field is the data field.

[0273] The permission record generated by the received second permission setting request will be marked and recorded as the second permission record.

[0274] Mark different permission records so that you can obtain the corresponding controlled scope later.

[0275] The permission record represents the controlled fields that the permission field can access. For example, the controlled field is a city code (Beijing), and the permission field is a city code (Beijing and Shanghai). The effect of setting permissions is: if the city where user A, a visitor, is located is Beijing, then user A can access data in the data fields (controlled fields) of Beijing and Shanghai.

[0276] The generated permission records can be saved locally or in the permission system.

[0277] When storing the corresponding permission record, a permission query identifier is generated for the permission record, specifically:

[0278] Generate an authority query identifier using an identifier of a data source table where the data field is located, an identifier of a data source corresponding to the data source table, and an identifier of a data report corresponding to the data source;

[0279] The authority query identifier is used to identify the authority record corresponding to the data field.

[0280] At the same time, for this method, that is, the permission record generated by receiving the second permission setting request is set with a second mark, so as to subsequently obtain the corresponding controlled range.

[0281] In the above technical solution, users can set permissions for corresponding data fields in the permission setting interface by means of templates, which can more conveniently meet the user's personalized settings and more comprehensively implement the setting of row-level permissions. This solution can actively, conveniently and flexibly set access permissions according to the user's own needs, improve the efficiency of permission control settings, and user experience.

[0282] In the above scheme, permissions are set for the data source corresponding to the data report and the fields in the data source table corresponding to the data source in two ways. That is to say, the data report with data field permissions set needs to be marked so that users can control it through the set permissions when accessing the data report.

[0283] The following is a detailed description of the steps for reporting user access data:

[0284] The first step is to, when a user's request to access a data report is received, if it is determined that the data table corresponding to the data report has permission settings, generate a permission query identifier corresponding to the data table.

[0285] When this application is implemented, it is also necessary to mark the data reports for which permissions are set, and the data tables in the data reports for which permissions are set, so that when querying the corresponding data report, it is first determined whether the data report and the corresponding data table have permissions set.

[0286] If no permission is set, all visitors on the data platform can access it. If permission is set, only visitors with corresponding permissions can access the data in the corresponding data field.

[0287] The second step is to use the permission query identifier to query the permission record, and obtain the controlled field corresponding to the data field in the data table according to the queried permission record.

[0288] If the permission record is saved locally, the corresponding controlled fields are directly obtained from the local location;

[0289] If the permission record is stored in the permission system, the corresponding controlled fields are obtained from the permission system.

[0290] The third step is to parse the controlled field and obtain the controlled range.

[0291] In this step, the controlled field is parsed to obtain the controlled range, including:

[0292] If it is determined that the permission record is the first permission record, the content corresponding to the controlled field is the visitor attribute identifier;

[0293] If it is determined that the permission record is the second permission record, the content corresponding to the controlled field is the visitor attribute template identifier.

[0294] The controlled range is encapsulated in the visitor's access report request, and a subsequent query process is performed to determine whether the visitor can access the data corresponding to the data field.

[0295] Step 4: Determine whether to output the data corresponding to the data field for the user according to whether the visitor attribute of the user belongs to the controlled range.

[0296] In summary, in the above embodiment of the present application, when a visitor accesses a data report with permission settings, it is necessary to query the controlled range corresponding to the controlled field, and then determine whether the visitor attribute of the visitor belongs to the controlled range to determine whether the visitor can access the corresponding data. This solution implements the control capability of row-level permissions, can improve the efficiency of access permission control, and improve user experience.

[0297] The embodiment of the present application cleverly avoids the defect that permissions cannot be mapped to corresponding data reports due to the limitations of the permission system itself, and more flexibly associates the controlled charts on the data report configuration page, greatly facilitating the flexibility and free configuration of users' permission control over their own reports.

[0298] As the number of visitors increases and data reports become more diverse, various personalized custom templates can be added to improve permission configuration and make permission configuration more diversified.

[0299] Based on the same inventive concept, an access rights control device is also provided in an embodiment of the present application, the device comprising: an acquisition unit, a determination unit, a generation unit and a storage unit;

[0300] The acquisition unit is configured to acquire a permission setting request; when the determination unit determines that the acquired permission setting request is a first permission setting request, acquire a visitor attribute identifier and a data field carried in the first permission setting request;

[0301] The determining unit is configured to determine whether the permission setting request obtained by the obtaining unit is a first permission setting request;

[0302] The generating unit is used to generate a first permission record; wherein the first permission record includes a permission field and a controlled field; the permission field is the visitor attribute identifier, and the controlled field is the data field;

[0303] The storage unit is used to store the first authority record generated by the generation unit.

[0304] Preferably, the device further comprises: a setting unit;

[0305] The setting unit is used to pre-set a visitor attribute template for a data field;

[0306] The determining unit is further configured to determine whether the permission setting request obtained by the obtaining unit is a second permission setting request;

[0307] The acquiring unit is further configured to acquire a data field and a visitor attribute template identifier carried in the second permission setting request when the determining unit determines that the acquired permission setting request is a second permission setting request;

[0308] The generating unit is further used to generate a second permission record; wherein the second permission record includes a permission field and a controlled field; the permission field is the visitor attribute template identifier, and the controlled field is the data field;

[0309] The storage unit is further used to store the second authority record generated by the generation unit.

[0310] Preferably,

[0311] The generation unit is further used to generate an authority query identifier using the identifier of the data source table where the data field is located, the identifier of the data source corresponding to the data source table, and the identifier of the data report corresponding to the data source; and use the authority query identifier to identify the authority record corresponding to the data field.

[0312] Preferably,

[0313] The acquisition unit is further configured to receive a user's request to access the data report;

[0314] The determination unit is further configured to determine whether the data source table corresponding to the data report has been set with permissions when the acquisition unit receives a request from the user to access the data report; and determine whether to output the data corresponding to the data field for the user according to whether the visitor attribute of the user belongs to the controlled range determined by the storage unit;

[0315] The generating unit is further configured to generate a permission query identifier corresponding to the data source table if the determining unit determines that the data source table corresponding to the data report has permission settings;

[0316] The storage unit is further used to query the corresponding permission record using the permission query identifier generated by the generation unit; obtain the controlled field corresponding to the data field in the data source table according to the queried permission record; and parse the controlled field to obtain the controlled range.

[0317] Preferably,

[0318] The storage unit is specifically used for, if it is determined that the permission record is the first permission record, then the content corresponding to the controlled field is the visitor attribute identifier; if it is determined that the permission record is the second permission record, then the content corresponding to the controlled field is the visitor attribute template identifier.

[0319] In another embodiment, an electronic device is provided, including a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor implements the steps of the access permission control method when executing the program.

[0320] In another embodiment, a computer-readable storage medium is provided, on which a computer program is stored, and when the program is executed by a processor, the steps of the access permission control method are implemented.

[0321] The units in the above-mentioned embodiments may be integrated into one body or deployed separately; they may be combined into one unit or further divided into multiple sub-units.

[0322] The above technical solution provided by this application uploads data files through the data import service DTP, or directly connects to the remote database by configuring the connection, which can simply realize the association of data sources, and then realize the diversity of data sources; when the user issues a data output request, the query engine service is called, and the query request is generated using the link information and parameter information corresponding to the data output request issued by the user, and the corresponding data is queried and analyzed from the data source corresponding to the link information, and finally output to the user according to the data method configured by the user. This solution greatly improves the diversity and convenience of data source access, and does not rely on Kylin. The entire data acquisition, analysis and output process is simple, thereby improving data processing efficiency.

[0323] In conjunction with the accompanying drawings, the following describes in detail the process of uploading and processing data on the data analysis platform by taking an Excel file as the data source and the data output request as outputting a report as an example.

[0324] First, the process of uploading Excel files to the data analysis platform is given:

[0325] In the first step, when the user clicks the establish connection button through the operation interface, the data management platform outputs a prompt message for the user to select the data source type.

[0326] like Figure 5 , Figure 5 A schematic diagram of a data source type selection provided in an embodiment of the present application. Figure 5 In the example, three optional data source types are taken, such as excel (file), csv (file), and MySQL (database).

[0327] When uploading an Excel file, the user can directly select the data source type as Excel (file).

[0328] Step 2: When the data management platform receives the data source type selected by the user and it is an Excel file, it provides the user with an upload interface.

[0329] The user can directly drag the Excel file to be uploaded to the specified location on the upload interface, or use the click-to-upload method to upload the Excel file. The DTP service is used to upload the Excel file.

[0330] Step 3: After receiving the save instruction from the user, the relevant information of the Excel file is displayed to the user.

[0331] See also Figure 6 , Figure 6 A schematic diagram of information related to a data connection established in an embodiment of the present application. Figure 6There is the data connection name of the data connection, the data file name (ceshi444.xlsx), the header settings, that is, the architecture information of the data in the data connection. For example, the Excel file in this embodiment includes two Sheets, Sheet1 and Sheet2, and the dimension information of the Excel file. Figure 6 Due to space constraints, it is not shown.

[0332] Step 4: After uploading the Excel file through the data import service DTP, store it in the hive cluster.

[0333] When uploading, first upload to the cloud disk, and then write to the self-storage hive cluster through scheduling. You can also synchronize the data file to the hive cluster of the metadata management platform at the same time for big data analysis and other processing.

[0334] Step 5. When the upload of the Excel file is completed, the data connection is created.

[0335] The Excel file is stored locally, and the link information of the Excel file is automatically generated. The link information and file structure information corresponding to the created data connection (demonstration process) are stored so that the link information can be associated with the data query later.

[0336] Secondly, the process of creating a report (a form of data output) based on the above Excel file is as follows:

[0337] The first step is to display relevant information of the corresponding data connection to the user when receiving an operation of the user selecting an established data connection on the operation page of the data analysis platform.

[0338] See also Figure 7 , Figure 7 It is the content displayed after selecting a data connection in the embodiment of the present application. Figure 7 The selected data connection is given as a demonstration process. This embodiment takes selecting one data connection as an example. In a specific application, multiple data connections can also be selected.

[0339] Step 2: When the user selects sheet1, relevant information corresponding to sheet1 is displayed to the user.

[0340] The third step is to set the data output method (you can provide users with output method options).

[0341] See also Figure 8 , Figure 8 This is a schematic diagram of selecting a chart type given in an embodiment of the present application. Figure 8The available types are: general chart, comparative trend chart, indicator card, dashboard, pie chart, funnel chart, and Sanchi chart. Figure 8 These are just a few chart options provided in the embodiments of the present application. In actual applications, the selection of chart types can be increased or decreased according to actual applications.

[0342] Step 4: After receiving the chart type selected by the user, record it.

[0343] After setting the chart type, users can implement operation requests by dragging parameters.

[0344] See also Fig. 9 , Fig. 9 This is a schematic diagram of an interface for issuing a data output request in an embodiment of the present application. Fig. 9 The user can trigger the issuance of data output request by dragging the two parameters "Dimension" and "Measurement" to the canvas.

[0345] Step 5. When the data analysis platform receives a data output request triggered by the user dragging a parameter, it obtains the link information corresponding to the data connection (demonstration process) and parameter information (dimension, metric).

[0346] Step 6: Convert the parameter information into the format required by the query engine, and generate a query request (such as an SQL statement) by using the converted parameter information and link information through the query engine service.

[0347] Step 7: Call the permission service to obtain the permission information corresponding to the user and the permission information of the data corresponding to the data output request, and perform data query to the data source corresponding to the link information through the query request based on the obtained permission information.

[0348] Step 8: Obtain dimension and metric related data from the data source corresponding to the link information through the query engine service.

[0349] Step 9. Parse the acquired data and display it using the output method selected by the user.

[0350] See also Fig.10 , Fig.10 This is a schematic diagram of displaying output data in a general chart format in an embodiment of the present application. Fig.10 A chart with the x-axis as latitude information and the y-axis as measurement information.

[0351] This completes the creation of a report (data report).

[0352] During the specific implementation, the report can also be saved for direct viewing later.

[0353] Currently, the report creation and basic information of the report on the data analysis platform are generally the same. When the user creates a report, there is no need to pop up a prompt box and then enter the report editing page. All the system needs to create a default report name for the report, and the created report name globally uniquely identifies a report.

[0354] In the embodiment of the present application, a naming method for data reports is provided. According to the preset suffix naming rule and the suffix of the data reports named on the platform, a suffix is ​​generated for the currently generated data report. The suffix determined for the currently generated data report is the suffix that does not exist in the acquired suffixes and ranks first according to the preset suffix naming rule. If it is in the suffix of the named data report, that is, if the suffix is ​​missing, the suffix needs to be filled. Through this solution, when naming the data report, the suffix of the data report can be automatically filled, which enhances the readability of the data report name.

[0355] In the embodiment of the present application, when a data report is generated, the data report is named, and the name given to the data report is the name of the data report.

[0356] The name of the data report in the embodiment of the present application is composed of a prefix + a suffix, which correspond to the naming rules of the prefix and the suffix respectively. When naming, the prefix is ​​named according to the prefix naming rule, and the suffix is ​​named according to the suffix naming rule, and the prefix and suffix are combined as the name of a data report.

[0357] The specific implementation process of data report naming in the embodiment of the present application is as follows:

[0358] The first step is to obtain the suffix of the named data report in the data analysis platform when generating the data report.

[0359] In the embodiment of the present application, the suffix of the named data report in the data analysis platform is obtained. The specific steps are:

[0360] Step A: Obtain the name of the named data report in the data analysis platform.

[0361] In the specific implementation of this step, the list stream, that is, the names of all data reports, is obtained through the stream method; then each value in the list, that is, the name of each data report, is obtained through the forEach method;

[0362] Step B: verifying the legality of the prefix of the name of the data report according to the prefix naming rule.

[0363] The prefix naming rule in the embodiment of the present application may be to preset a fixed prefix, such as data report, etc.;

[0364] Verifying the legitimacy of a prefix is ​​to determine whether the prefix of the data report complies with the prefix naming rules, such as the comparison method, or whether the string length is equal.

[0365] The implementation of this step is to determine that the data report name contains a prefix named according to the prefix naming rule, so as to avoid the name of the acquired data report being illegal or not belonging to the scope of the data report naming performed in the embodiment of the present application.

[0366] Step C: Obtain the name of the data report with a legal prefix.

[0367] Step D: remove the prefix of the name of the data report and obtain the suffix of the name of the data report.

[0368] In the embodiment of the present application, the suffix of the name of the data report can be implemented by numbers, so after the prefix is ​​cut off, the suffix of the name of the acquired data report is a data set.

[0369] The second step is to determine a suffix for naming the data report currently being generated according to a preset suffix naming rule and the obtained suffix.

[0370] The suffix determined for the currently generated data report is a suffix that does not exist in the acquired suffixes and is ranked first according to the preset suffix naming rule.

[0371] If the preset suffix naming rule is a rule that starts from 1 and increases by 1, if the obtained suffixes are 1, 3, and 5, then the suffix that does not exist in the obtained suffixes and ranks first according to the preset suffix naming rule is 2;

[0372] For example, if the preset suffix naming rule is a rule of decreasing by 1 starting from 100, if the obtained suffixes are 100, 99, and 95, the suffix 98 that does not exist in the obtained suffixes and ranks first according to the preset rear axle naming rule is 98.

[0373] In the embodiment of the present application, according to the preset suffix naming rule, and the suffix obtained is the currently generated data report, there are two implementation methods for determining the suffix used for naming, but not limited to the following two implementation methods, specifically:

[0374] The first one:

[0375] In the first embodiment of the present application, the steps of determining the suffix used for naming the currently generated data report are as follows:

[0376] Step A: Generate a first suffix according to the preset suffix naming rule, and compare the generated first suffix with the obtained suffix.

[0377] The first suffix is ​​the first suffix among the suffixes generated according to the preset suffix naming rule; that is, the suffix initially generated according to the preset rule is the first suffix here.

[0378] Step B: If there is a suffix in the acquired suffixes that is the same as the first suffix, then continue to generate the next suffix according to the preset suffix naming rule and compare again.

[0379] Step C: until there is no suffix generated according to the preset suffix naming rule among the acquired suffixes, the generated suffix is ​​determined as the suffix of the currently generated data report.

[0380] If the preset suffix naming rule is a naming rule that increases by 1, and the first suffix generated according to the preset suffix naming rule is 1, 1 is used to compare with the obtained suffix;

[0381] Assuming that the set of acquired suffixes is {1, 2, 4, 6}, if there is a suffix in the acquired suffixes that is the same as the first suffix generated, then continue to generate the next suffix (the second suffix) 2.

[0382] Compare 2 with the obtained suffix and determine that 2 exists in the obtained suffix, then generate the next suffix (the third suffix) 3 according to the preset suffix naming rule, compare 3 with the obtained suffix and determine that 3 does not exist in the obtained suffix, then use suffix 3 as the suffix used for naming the currently generated report.

[0383] The preset suffix naming rule here is incremental, and increases by 1. In specific implementation, the preset suffix naming rule is not limited to this, as long as the report name named using the preset suffix naming rule is readable. As can be seen from the above, if a data report is deleted in the data reports named in sequence according to the preset suffix rule, it will be named in place when the data report is generated later, which will make the data report name more readable.

[0384] Second type:

[0385] The second specific steps of determining the suffix used for naming the currently generated data report in the embodiment of the present application are:

[0386] Step A: determining the last-ranked suffix among the acquired suffixes according to a preset suffix naming rule, and determining the suffix to be the Mth suffix generated according to the preset suffix naming rule.

[0387] Wherein, M is an integer greater than 0.

[0388] Step B, determine whether M is equal to N, if so, execute step C; otherwise, execute step E.

[0389] N is the total number of suffixes obtained, an integer greater than 0.

[0390] Step C: Generate the N+1th suffix according to the preset suffix naming rule, and determine it as the suffix of the currently generated data report, and end this process.

[0391] Step D: Generate a first suffix according to the preset suffix naming rule, and compare the generated first suffix with the obtained suffix.

[0392] The first suffix is ​​the suffix that ranks first among the suffixes generated according to the preset suffix naming rule.

[0393] Step E: If there is a suffix in the acquired suffixes that is the same as the first suffix, continue to generate the next suffix according to the preset suffix naming rule and compare again; until there is no suffix generated according to the preset suffix naming rule in the acquired suffixes, determine the generated suffix as the suffix of the currently generated data report.

[0394] If the preset suffix naming rule is a naming rule that increases by 1, assuming that the set of suffixes obtained is {1, 2, 3}, it is determined that the last suffix in the ranking is 3, and according to the naming rule that increases by 1, it is determined that the third suffix is ​​generated, and it is determined that the total number of suffixes obtained is 3, then it is further determined that 3=3. Therefore, according to the preset suffix rule, the 3+1th suffix 4 is generated as the suffix used for naming the currently generated data report.

[0395] The preset suffix naming rule here is incremental, and increases by 1. In specific implementation, the preset suffix naming rule is not limited to this, as long as the report name named using the preset suffix naming rule is readable. As can be seen from the above, if a data report is deleted in the data reports named in sequence according to the preset suffix rule, it will be named in place when the data report is generated later, which will make the data report name more readable.

[0396] In the embodiment of the present application, it is first determined whether there is a suffix among the acquired suffixes that needs to be filled. If not, the next suffix is ​​directly generated according to the preset suffix naming rules. If there is a suffix that needs to be filled, that is, if there is a deleted data report, the suffix filling and naming are performed according to the above-mentioned one-to-one comparison method.

[0397] Step 3: Use the determined suffix to name the currently generated data report.

[0398] The data report in the embodiment of the present application is named as a prefix named according to the prefix naming rule + a suffix named according to the preset suffix naming rule;

[0399] If the preset prefix naming rule is the default naming data report, the name of the currently generated data report is data report + the determined suffix.

[0400] Based on the same inventive concept, an embodiment of the present application also provides a data report naming device, which is applied to a data analysis platform, and the device includes: a first generating unit, an acquiring unit, a first determining unit and a naming unit;

[0401] The first generating unit is used to generate a data report;

[0402] The acquisition unit is configured to acquire a suffix of a named data report in the data analysis platform when the first generation unit generates a data report;

[0403] The first determining unit is used to determine a suffix for naming the data report currently generated according to a preset suffix naming rule and the suffix acquired by the acquiring unit; wherein the suffix for naming the data report currently generated is a suffix that does not exist in the acquired suffixes and ranks first according to the preset suffix naming rule;

[0404] The naming unit is used to name the currently generated data report using the suffix determined by the first determining unit.

[0405] Preferably, the acquisition unit comprises: a first acquisition subunit, a verification unit, a second acquisition subunit, a interception unit and a third acquisition subunit;

[0406] The first acquisition subunit is used to acquire the name of the named data report in the data analysis platform;

[0407] The verification unit is used to verify the legitimacy of the prefix of the name of the data report obtained by the first obtaining subunit according to the prefix naming rule;

[0408] The second acquisition subunit is used to obtain the name of the data report verified by the verification unit;

[0409] The interception unit is used to intercept the prefix of the name of the data report acquired by the second acquisition subunit;

[0410] The third acquisition subunit is used to obtain the suffix of the name of the data report after the prefix of the name of the data report is cut off by the cutting unit.

[0411] Preferably, the first determining unit comprises: a second generating unit, a comparing unit and a second determining unit;

[0412] The second generating unit is used to generate a first suffix according to the preset suffix naming rule; wherein the first suffix is ​​the suffix that ranks first among the suffixes generated according to the preset suffix naming rule;

[0413] The comparison unit is used to compare the first suffix generated by the second generation unit with the acquired suffix; if there is a suffix in the acquired suffix that is the same as the first suffix, the second generation unit is triggered to continue to generate the next suffix according to the preset suffix naming rule, and compare again;

[0414] The second determining unit is configured to determine the generated suffix as the suffix of the currently generated data report until there is no suffix generated according to the preset suffix naming rule among the acquired suffixes.

[0415] Preferably, the first determining unit includes: a second determining unit, a judging unit, a second generating unit and a third determining unit;

[0416] The second determining unit is used to determine the last-ranked suffix among the acquired suffixes according to a preset suffix naming rule, and determine that the suffix is ​​the Mth suffix generated according to the preset suffix naming rule;

[0417] The judging unit is used to determine whether M is equal to N; wherein N is the total number of the suffixes obtained;

[0418] The second generating unit is used to generate the N+1th suffix according to the preset suffix naming rule when the judging unit determines that M is equal to N; generate the first suffix according to the preset suffix naming rule when it is determined that M is not equal to N; and trigger the judging unit to use the generated first suffix to compare with the acquired suffix; if there is a suffix in the acquired suffix that is the same as the first suffix, continue to generate the next suffix according to the preset suffix naming rule, and trigger the judging unit to compare again;

[0419] The third determination unit is used to determine the N+1th suffix as the suffix used for naming the currently generated data report when the second generation unit generates the N+1th suffix; until there is no suffix generated according to the preset suffix naming rule among the acquired suffixes, and then determine the generated suffix as the suffix of the currently generated data report; wherein N is the total number of the suffixes acquired, and the first suffix is ​​the first-ranked suffix among the suffixes generated according to the preset suffix naming rule.

[0420] The units in the above-mentioned embodiments may be integrated into one body or deployed separately; they may be combined into one unit or further divided into multiple sub-units.

[0421] In another embodiment, an electronic device is provided, including a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor implements the steps of the data report naming method when executing the program.

[0422] In another embodiment, a computer-readable storage medium is provided, on which a computer program is stored, and when the program is executed by a processor, the steps of the data report naming method are implemented.

[0423] In summary, the present application obtains the suffix of the name of the named data report, determines to fill in the name for the currently generated data report, and generates a continuous suffix that meets the preset suffix naming rules, so that the name of the data report is readable.

[0424] And when obtaining the suffix of the name of the named data report, the prefix of the name of the data report is legally verified, and the illegal data report names are filtered out, thereby improving the efficiency and accuracy of data report naming.

[0425] Based on the same inventive concept, the present application also provides a data processing device in an embodiment, which is applied to the above data analysis platform. Fig.11 , Fig.11 This is a schematic diagram of the structure of a device applied to the above technology in an embodiment of the present application. The device includes: an acquisition unit 1101, a query engine unit 1102 and an output unit 1103;

[0426] An acquiring unit 1101 is configured to acquire link information related to a data output request when acquiring a data output request generated by a user operating a page;

[0427] The query engine unit 1102 is used to generate a query request using the link information obtained by the query engine service obtaining unit 1101 and the parameter information corresponding to the data output request, and perform data query to the data source corresponding to the link information through the query request;

[0428] The output unit 1103 is used to parse and output the data queried by the query engine unit 1102.

[0429] Preferably,

[0430] The acquisition unit 1101 is further used to convert the parameter information corresponding to the data output request into parameter information in an input parameter format that meets the requirements of the query engine;

[0431] The query engine unit 1102 is specifically used to generate a query request by using the link information and the parameter information corresponding to the data output request through the query engine service, and to generate the query request by using the link information and the parameter information in the input parameter format that meets the query engine requirements after being converted by the query engine service 1101.

[0432] Preferably,

[0433] The data source is a data file in a local database and / or a remote database.

[0434] Preferably, the device further comprises: an uploading unit 1104 and a configuration unit 1105;

[0435] The uploading unit 1104 is used for uploading the data file through the data import service dynamic relay protocol DTP and storing it in the local database when the data source is a data file stored in the local database;

[0436] The configuration unit 1105 is used to configure the link information of the database when the data source is a remote database.

[0437] Preferably,

[0438] An acquisition unit 1101 is used to acquire the data output mode input by the user;

[0439] The output unit 1103 is further used to parse the queried data and output it in accordance with the data output method obtained by the obtaining unit.

[0440] Preferably,

[0441] The acquisition unit 1101 is further used to obtain the permission information corresponding to the user and the permission information of the data corresponding to the data output request before the query engine unit 1102 performs a data query on the data source corresponding to the link information through the query request, and trigger the query engine unit 1102 to perform a data query on the data source corresponding to the link information through the query request according to the obtained permission information.

[0442] The units in the above-mentioned embodiments may be integrated into one body or deployed separately; they may be combined into one unit or further divided into multiple sub-units.

[0443] In another embodiment, an electronic device is provided, including a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor implements the steps of the data processing method when executing the program.

[0444] In another embodiment, a computer-readable storage medium is provided, on which computer instructions are stored. When the instructions are executed by a processor, the steps of the data processing method can be implemented.

[0445] Fig.12 Schematic diagram of the physical structure of an electronic device provided by an embodiment of the present invention. Fig.12 As shown, the electronic device may include: a processor 1210, a communication interface 1220, a memory 1230 and a communication bus 1240, wherein the processor 1210, the communication interface 1220, and the memory 1230 communicate with each other through the communication bus 1240. The processor 1210 may call the logic instructions in the memory 1230 to execute the following method:

[0446] When a data output request generated by a user operation page is obtained, link information related to the data output request is obtained;

[0447] Generate a query request by using the link information and parameter information corresponding to the data output request through a query engine service, and perform data query on a data source corresponding to the link information through the query request;

[0448] Parse and output the queried data.

[0449] In addition, the logic instructions in the above-mentioned memory 1230 can be implemented in the form of a software functional unit and can be stored in a computer-readable storage medium when it is sold or used as an independent product. Based on such an understanding, the technical solution of the present invention can be essentially or partly embodied in the form of a software product that contributes to the prior art. The computer software product is stored in a storage medium, including several instructions for a computer device (which can be a personal computer, a server, or a network device, etc.) to perform all or part of the steps of the method described in each embodiment of the present invention. The aforementioned storage medium includes: various media that can store program codes, such as a USB flash drive, a mobile hard disk, a read-only memory (ROM), a random access memory (RAM), a magnetic disk or an optical disk.

[0450] The device embodiments described above are merely illustrative, wherein the units described as separate components may or may not be physically separated, and the components displayed as units may or may not be physical units, that is, they may be located in one place, or they may be distributed on multiple network units. Some or all of the modules may be selected according to actual needs to achieve the purpose of the scheme of this embodiment. Ordinary technicians in this field can understand and implement it without paying creative labor.

[0451] Through the description of the above implementation methods, those skilled in the art can clearly understand that each implementation method can be implemented by means of software plus a necessary general hardware platform, and of course, it can also be implemented by hardware. Based on this understanding, the above technical solution is essentially or the part that contributes to the prior art can be embodied in the form of a software product, and the computer software product can be stored in a computer-readable storage medium, such as ROM / RAM, a disk, an optical disk, etc., including a number of instructions for a computer device (which can be a personal computer, a server, or a network device, etc.) to execute the methods described in each embodiment or some parts of the embodiments.

[0452] The above description is only a preferred embodiment of the present invention and is not intended to limit the present invention. Any modifications, equivalent substitutions, improvements, etc. made within the spirit and principles of the present invention should be included in the scope of protection of the present invention.

Claims

1. A data processing method, characterized in that: The method comprises: When a data output request generated by a user operating a page is obtained, link information related to the data output request is obtained, wherein the data output request is generated when the user operates a parameter on the page; Generate a query request by using the link information and the parameter information corresponding to the data output request through a query engine service, and perform data query on the data source corresponding to the link information through the query request, and when the corresponding data source is found through the link information, use the parameter information as a matching item to query data matching the parameter information; Parse and output the queried data; The data source is a data file in a local database and / or a remote database, and the link information is a query path of the database; If there are multiple link information in the query request, then in the query request, each link information is respectively matched with the parameter information, so as to search for data matching the parameter information in the data source corresponding to each link information.

2. The method according to claim 1, characterized in that: The step of generating a query request by using the link information and parameter information corresponding to the data output request through a query engine service includes: Convert the parameter information corresponding to the data output request into parameter information in an input parameter format that meets the requirements of the query engine; The query request is generated by the query engine service using the link information and the parameter information in the input parameter format that meets the query engine requirements.

3. The method according to claim 1, characterized in that When the data source is a data file stored in a local database, the data file is uploaded via a data import service dynamic relay protocol DTP and stored in the local database; When the data source is a remote database, the method further includes: configuring link information of the database.

4. The method according to claim 1, characterized in that: The method further comprises: Get the data output mode of the user input; The step of parsing and outputting the queried data includes: The queried data is parsed and outputted in accordance with the acquired data output mode.

5. The method according to any one of claims 1 to 4, characterized in that Before performing data query to the data source corresponding to the link information through the query request, the method further comprises: The authority information corresponding to the user and the authority information of the data corresponding to the data output request are obtained, and data query is performed to the data source corresponding to the link information through the query request according to the obtained authority information.

6. A data processing device, characterized in that: The device comprises: an acquisition unit, a query engine unit and an output unit; The acquisition unit is used to acquire link information related to the data output request when acquiring the data output request generated by the user operating the page; wherein the data output request is generated when the user operates on the parameters on the page; The query engine unit is used to generate a query request through a query engine service using the link information obtained by the acquisition unit and the parameter information corresponding to the data output request, and to perform a data query to the data source corresponding to the link information through the query request, and when the corresponding data source is found through the link information, use the parameter information as a matching item to query data matching the parameter information; The output unit is used to parse and output the data queried by the query engine unit; The data source is a data file in a local database and / or a remote database, and the link information is a query path of the database; If there are multiple link information in the query request, then in the query request, parameter information is respectively matched to each link information, so as to search for data matching the parameter information in the data source corresponding to each link information.

7. The device according to claim 6, characterized in that The acquisition unit is further used to convert the parameter information corresponding to the data output request into parameter information in an input parameter format that meets the requirements of the query engine; The query engine unit is specifically used to generate a query request through the query engine service using the link information and the parameter information corresponding to the data output request, and to generate the query request through the query engine service using the link information and the parameter information in the input parameter format that meets the query engine requirements after conversion by the acquisition unit.

8. An electronic device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that: When the processor executes the program, the method according to any one of claims 1 to 5 is implemented.

9. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the program is executed by a processor, the method described in any one of claims 1 to 5 is implemented.

Images