Data processing method and device, server and computer readable storage medium

Abstract

The application provides a data processing method and device, a server and a computer readable storage medium, and belongs to the technical field of data processing. The target data is acquired by acquiring at least one index corresponding to an original index from an index system according to data requirement information, generating data collection code and data processing code specially used for acquiring data corresponding to the original index, and then outputting the acquired target data. Since the names of the indexes in the index system and the association relationship between the indexes are pre-set, at least one index corresponding to the original index can be directly acquired from the index system, without the need for technical personnel to communicate for the indexes to be acquired, thereby avoiding the misalignment of terms, directly acquiring data through the code, avoiding the misunderstanding of the technical personnel for the association relationship of the indexes, leading to the acquired data not being the actual required data, improving the accuracy of the acquired data, and further improving the data processing accuracy.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of data processing, in particular to a data processing method and device, a server and a computer readable storage medium. BACKGROUND

[0002] With the continuous development of Internet business, a large amount of data will be generated in the running process of each Internet business. With the accumulation of data, data field requirements such as data statistics, data analysis, and external data services will gradually arise. In view of the requirements of different data fields, the accumulated data can be analyzed to provide data statistics results, data analysis results, and external data services, so as to maintain and improve the running Internet business or develop new Internet business.

[0003] Currently, when data is obtained based on a certain requirement in the data field, the business engineer and the data demander mainly communicate the required data indicators, and then the data development engineer obtains the data corresponding to the data indicators from the well-organized data warehouse according to the data indicators provided by the business engineer, and calculates the data, and then feeds back the calculated data to the data demander.

[0004] In the above implementation process, since different people may have different names for the same data indicator, it is easy to cause the terms to be misaligned in the communication process, which may cause the obtained data to be incorrect, and the obtained data is not the actual required data, thereby causing the data processing accuracy to be low. SUMMARY

[0005] The embodiments of the present application provide a data processing method, device, server and computer readable storage medium, which can improve the data processing accuracy. The technical scheme is as follows:

[0006] On the one hand, a data processing method is provided, which comprises:

[0007] In response to obtaining data requirement information, at least one indicator corresponding to the original indicator in the data requirement information is obtained from an indicator system, the indicator system being used to represent the association relationship between indicators and the database information of the indicators;

[0008] According to the at least one indicator and the data requirement information, a data collection code and a data processing code are generated, the data collection code being used to obtain first data corresponding to the at least one indicator from the database, and the data processing code being used to process the first data of the at least one indicator based on the data requirement information;

[0009] The data acquisition code is used to acquire first data corresponding to the at least one index, and the data processing code is used to process the first data to obtain second data corresponding to the original index.

[0010] Based on the second data and the data requirement information, target data meeting the data requirement information is output.

[0011] In one aspect, a data processing apparatus is provided, which comprises:

[0012] An index acquisition module is configured to acquire at least one index corresponding to an original index in data requirement information from an index system in response to acquiring the data requirement information, the index system being used to represent an association relationship between indexes and database information of the indexes;

[0013] A code generation module is configured to generate data acquisition code and data processing code according to the at least one index and the data requirement information, the data acquisition code being used to acquire first data corresponding to the at least one index from a database, and the data processing code being used to process the first data of the at least one index based on the data requirement information;

[0014] A data processing module is configured to acquire the first data corresponding to the at least one index through the data acquisition code, and generate second data corresponding to the original index through the data processing code and the first data;

[0015] A data output module is configured to output target data meeting the data requirement information based on the second data and the data requirement information.

[0016] In one possible implementation, the apparatus further comprises:

[0017] An adding module is configured to add database information of the second data to a corresponding position in data storage template code to obtain data storage code for storing the second data, if the second data needs to be stored;

[0018] A storage module is configured to store the database information of the second data to the index system as the database information corresponding to the original index through the data storage code.

[0019] In a possible implementation, the data output module is configured to add the database information of the second data to a corresponding position in the data acquisition template code to obtain data acquisition code, the data acquisition code being used to acquire the second data corresponding to the original index from the database; add the interface identifier of the data query interface corresponding to the second data to a corresponding position in the data query interface template code to obtain data query interface code, the data query interface code being used to query the second data; and add the data acquisition code and the data query interface code as index data application code, the data query interface code being used to query the second data, and output the second data as target data of the data requirement information through the index data application code.

[0020] In a possible implementation, the apparatus further includes:

[0021] The first code generation module is configured to add the platform identifier of the statistical platform to a corresponding position in the first general-purpose data application template code to obtain first general-purpose data application code, the first general-purpose data application code being used to transmit the second data to the statistical platform.

[0022] The second code generation module is configured to add the file format of the character-separated value file to a corresponding position in the second general-purpose data application template code to obtain second general-purpose data application code, the second general-purpose data application template code being used to process the second data into a character-separated value file.

[0023] In an aspect, a server is provided, which includes one or more processors and one or more memories having at least one piece of program code stored therein, the program code being loaded and executed by the one or more processors to implement operations performed by the data processing method.

[0024] In an aspect, a computer-readable storage medium is provided, which has at least one piece of program code stored therein, the program code being loaded and executed by a processor to implement operations performed by the data processing method.

[0025] The scheme provided in the application obtains at least one index corresponding to the original index from the index system according to the data requirement information, generates data collection code and data processing code specially used for obtaining data corresponding to the original index, and obtains target data, and then outputs the obtained target data. Since the names of the indexes in the index system and the association relationship between the indexes are pre-set, at least one index corresponding to the original index can be obtained directly from the index system, without the need for technical personnel to communicate for the indexes to be obtained, avoiding the problem of misaligned terms, and then the data is obtained directly through the code, avoiding the misunderstanding of the technical personnel for the association relationship of the indexes, so that the obtained data is not the actual required data, improving the accuracy of obtaining data, and then improving the data processing accuracy. BRIEF DESCRIPTION OF DRAWINGS

[0026] In order to more clearly illustrate the technical solutions in the embodiments of the application, the following will briefly introduce the drawings needed to be used in the embodiment description. Obviously, the drawings in the following description are only some embodiments of the application, and for those skilled in the art, other drawings can also be obtained from these drawings without creative labor.

[0027] Figure 1 is an implementation environment schematic diagram of a data processing method provided by the embodiment of the application;

[0028] Figure 2 is a flowchart of a data processing method provided by the embodiment of the application;

[0029] Figure 3 is a flowchart of a data processing method provided by the embodiment of the application;

[0030] Figure 4 is a flowchart of generating data collection code provided by the embodiment of the application;

[0031] Figure 5 is a flowchart of generating data processing code provided by the embodiment of the application;

[0032] Figure 6 is a flowchart of generating index data application code provided by the embodiment of the application;

[0033] Figure 7 is a flowchart of generating general data application code provided by the embodiment of the application;

[0034] Figure 8 is an architecture diagram of a data processing system provided by the embodiment of the application;

[0035] Figure 9Fig. 1 is a structural schematic diagram of a data processing device provided by an embodiment of the present application;

[0036] Figure 10 Fig. 2 is a structural schematic diagram of a server provided by an embodiment of the present application. DETAILED DESCRIPTION

[0037] In order to make the purpose, technical solutions and advantages of the present application clearer, the embodiments of the present application will be further described in detail below with reference to the drawings.

[0038] First, the related technical terms and terminologies involved in the present application will be explained as follows:

[0039] Indicator: So-called indicator is often used in data analysis, which is actually a kind of measurement. The indicator is as large as used for monitoring and evaluating the status of business process, as small as measuring the situation of a functional module, or the effect of one's own activities. From the perspective of operation, a good indicator needs to have the following four characteristics:

[0040] (1) Business level is valuable;

[0041] (2) Measurable business real situation;

[0042] (3) Simple and executable;

[0043] (4) Commonly recognized.

[0044] From the technical point of view, a good indicator needs to have the following four characteristics:

[0045] (1) Easy to collect and measure quickly;

[0046] (2) High accuracy;

[0047] (3) Can be multi-dimensionally decomposed;

[0048] (4) From a single data source.

[0049] More technically, the most basic indicator is a valuable field or record in a data entity (table) (such as the amount of a transaction payment), and the next layer of indicator is another data indicator calculated based on one or more fields (such as the total income of a supermarket in a day), and the upper layer is a more advanced measure of data.

[0050] Indicator system: The indicator system is a system composed of individual indicators, which is a way for data demanders to see the overall data situation more clearly, and can also represent the hierarchy and correlation between indicators.

[0051] Data entity: an entity for storing data, and the specific storage method can be selected according to the characteristics of the entity, and a relational database or a non-relational database is used for storage. Common data entities include personnel information tables, payment flow tables, and the like.

[0052] Comma-Separated Values (CSV) file format: a commonly used file format for data applications, in which the stored data can be quickly written or read by various commonly used languages, and can also be used by users to view in table format using Excel.

[0053] Figure 1 is a schematic diagram of an implementation environment of a data processing method provided by an embodiment of the present application, referring to Figure 1 The implementation environment includes a terminal 101 and a server 102.

[0054] The terminal 101 can be a smart phone, a tablet computer, a notebook computer, a desktop computer, a smart speaker, a smart watch, and the like, but is not limited thereto. The terminal 101 and the server 102 can be directly or indirectly connected through wired or wireless communication, which is not limited in the present application. The terminal 101 can provide various functional services for users based on various Internet services provided by the server 102. Users can search, browse, chat, watch videos, and the like through the terminal 101, and in the process of searching, browsing, chatting, and watching videos, the users can generate a large amount of data. The terminal 101 can obtain the data generated by the users and send the data to the server 102.

[0055] The terminal 101 can generally refer to one of a plurality of terminals, and the terminal 101 is used as an example in the embodiment. Those skilled in the art can know that the number of the above-mentioned terminals can be more or less. For example, the above-mentioned terminals can be only one, or the above-mentioned terminals can be dozens or hundreds, or more, and the number and type of the terminal are not limited in the embodiment of the present application.

[0056] The server 102 can be a stand-alone physical server, a server cluster composed of multiple physical servers, or a distributed system, and can also be a cloud server providing cloud services, cloud databases, cloud computing, cloud functions, cloud storage, network services, cloud communication, middleware services, domain name services, security services, content delivery networks (CDN), and basic cloud computing services such as big data and artificial intelligence platforms. The server 102 and the terminal 101 can be directly or indirectly connected through wired or wireless communication, which is not limited in the present application. The server 102 can receive data sent by the terminal 101, and the server 102 can also maintain multiple databases to store the received data. The database can store data in the form of a data table, or can store data in other forms, which is not limited in the present application. The server 102 can obtain data required for data analysis from the database to output based on the data. Alternatively, the number of servers can be more or less, which is not limited in the present application. Of course, the server 102 can also include other functional servers to provide more comprehensive and diversified services.

[0057] Figure 2 is a flowchart of a data processing method provided by an embodiment of the present application, referring to Figure 2 The method comprises:

[0058] 201, the server responds to the acquisition of data requirement information, and acquires at least one index corresponding to the original index in the data requirement information from an index system, the index system being used to represent the association relationship between indexes and the database information of the indexes.

[0059] It should be noted that the data requirement information is used to represent the demand of a technical personnel for data, and the data requirement information comprises an index identifier of the original index. The server can match in the index system according to the index identifier of the original index after acquiring the data requirement information, so as to determine the original index, and then determine at least one index corresponding to the original index according to the association relationship between indexes in the index system.

[0060] Through the data requirement information, the original index to be acquired can be directly determined from the index system, and then the index corresponding to the original index can be directly acquired from the index system according to the association relationship between indexes in the index system, without the need for communication between people, avoiding the problem of inconsistent communication terms, ensuring the accuracy of the collected data, and improving the data processing accuracy.

[0061] 202. The server generates data collection code and data processing code according to the at least one index and the data requirement information, the data collection code being used to obtain first data corresponding to the at least one index from the database, and the data processing code being used to process the first data of the at least one index based on the data requirement information.

[0062] It should be noted that, by generating unique data collection code and data processing code for the data requirement information in real time according to the at least one index and the data requirement information, the data collection code and the data processing code are specially used to process the data requirement information, and then the data can be directly processed through the code without manual processing, thereby improving the data processing efficiency.

[0063] 203. The server obtains first data corresponding to the at least one index through the data collection code, and processes the first data through the data processing code to obtain second data corresponding to the original index.

[0064] It should be noted that, by obtaining the first data through the data collection code and processing the first data through the data processing code, the data does not need to be manually obtained and processed, thereby improving the data processing efficiency.

[0065] 204. The server outputs target data meeting the data requirement information based on the second data and the data requirement information.

[0066] It should be noted that, the target data can be set with a specific interface, so that the target data can be output through the interface. Alternatively, the target data can be processed into a specific file format, so that the target data in the form of a file can be directly output.

[0067] The scheme provided by the embodiment of the application obtains at least one index corresponding to an original index from an index system according to data requirement information, generates data collection code and data processing code specially used to obtain data corresponding to the original index, and then obtains target data, and outputs the obtained target data. Since the names of the indexes in the index system and the association relationship between the indexes are pre-set, at least one index corresponding to the original index can be directly obtained from the index system without the need for a technician to communicate with respect to the index to be obtained, thereby avoiding the problem of misaligned terms, and the data is directly obtained through the code, thereby avoiding the technician's misunderstanding of the association relationship of the indexes, resulting in that the obtained data is not the actual required data, improving the accuracy of obtaining data, and further improving the data processing accuracy.

[0068] Figure 3 is a flowchart of a data processing method provided by the embodiment of the application, referring to Figure 3 The method comprises the following steps.

[0069] 301、The server acquires at least one index corresponding to the original index in the data requirement information from an index system in response to acquiring the data requirement information, the index system being used to represent the association relationship between indexes and database information of the indexes.

[0070] The index system includes a plurality of indexes, database information of each index, and the association relationship between indexes. The database information of an index includes the database name, data table name, and data field name storing data corresponding to the index. Optionally, the index system and the database information can include other contents, which are not limited in the embodiments of the present application. The index system can be managed by a data demander, and a technician can query the indexes and database information in the index system. By managing the index system by the data demander, the consistency of terms can be maintained, the inconsistency of terms used by different people leading to incorrect data acquisition can be avoided, and the accuracy of the acquired data can be ensured.

[0071] It should be noted that the terminal can display the indexes in the index system on a visual interface. When data needs to be acquired, the data demander can select on the visual interface, select an index by triggering a check box corresponding to the index, the terminal can determine the triggered index as an original index in response to the triggering operation of the user, generate data requirement information based on the original index, and then send the data requirement information to the server. Alternatively, when data needs to be acquired, the user can also input his / her own data requirement into the terminal. The terminal can acquire a keyword in the data requirement input by the user, match the acquired keyword with the existing indexes in the index system, determine an index matched with the keyword as an original index, generate data requirement information based on the original index, and then send the data requirement information to the server. The embodiments of the present application do not limit which way is used.

[0072] In a possible implementation, the server can locate the original index in the index system in response to acquiring the data requirement information, locate indexes having an association relationship with the original index according to the association relationship between the indexes in the index system, and take the located indexes as the at least one index. For example, if the original index in the data requirement information is "actual payment amount", the server can locate "actual payment amount" in the index system in response to acquiring the data requirement information, locate "order amount" and "discount amount" having an association relationship with "actual payment amount" according to the association relationship between the indexes in the index system, and take the located "order amount" and "discount amount" as the at least one index.

[0073] By directly locating the original index from the data system, without communication between the data demander and the researcher, the situation of obtaining incorrect data due to inconsistent terms in communication is avoided, the accuracy of the obtained data is ensured, and by directly locating the index having a correlation relationship with the original index from the data system, it can be quickly known which basic indexes need to be processed to obtain the original index, without the need for technical personnel to determine it themselves, avoiding obtaining incorrect data, and improving the accuracy of data processing.

[0074] 302、The server obtains the database name, the data table name and the data field name corresponding to the at least one index according to the database information of the index in the index system.

[0075] It should be noted that when data collection is performed based on data demand information, there can be two ways as follows:

[0076] (1) According to the storage mode of the data corresponding to the at least one index, the required data is queried and recorded, for example, in a relational database, the database and the data table where the data corresponding to the index is located are queried, the data field where the data corresponding to the index is located in the data table is determined, and then the data corresponding to the index is obtained according to the determined data field.

[0077] (2) The data corresponding to the at least one index is configured with an automatic reporting function, and the latest data is reported to the server corresponding to the index system at regular time.

[0078] In the above first collection method, the server can directly obtain the database name, the data table name and the data field name corresponding to the at least one index according to the database information of the index in the index system. In the above second collection method, the server can obtain the database name and the data table name corresponding to the at least one index according to the database information of the index in the index system, if the data table corresponding to the data table name has been configured with an automatic reporting function, then the data field name corresponding to the at least one index is obtained according to the database information of the index in the index system, if the data table corresponding to the data table name has not been configured with an automatic reporting function, then the data table name is added to the corresponding position in the data reporting template code to obtain a data reporting code, and the data field name corresponding to the at least one index is obtained from the index system through the data reporting code, the data reporting code is used to report the data in the data table in real time. By configuring the automatic reporting function, the data field name to be obtained can be directly reported to the server through the data reporting code, without the need for the server to query itself, improving the data reporting efficiency and thus improving the data processing efficiency.

[0079] It should be noted that in addition to generating data reporting code to report the latest updated data to the server, data synchronization code can also be generated to synchronize all data in the database to the server, and the embodiments of the present application do not limit this. In addition, while generating the data reporting code, data reporting protocols can also be generated to limit the interface used when reporting data, avoid the use of other interfaces from affecting the data reporting efficiency, and improve the data processing efficiency.

[0080] By obtaining the database name, data table name and data field name corresponding to the index according to the database information in the index system, the most fundamental source of data can be determined, and the first data corresponding to at least one index can be accurately obtained, without the need for technical personnel to analyze the original index to determine the data source, ensuring the accuracy of the data source and the accuracy of the obtained data, and improving the accuracy of data processing.

[0081] 303、The server adds the database name, the data table name and the data field name to the corresponding positions in the data collection template code respectively to obtain data collection code.

[0082] By automatically generating data collection code according to the storage location of the data, subsequent data collection of the first data corresponding to at least one index can be performed by running the data collection code, without the need for technical personnel to manually collect data, improving the data collection efficiency and the data processing efficiency.

[0083] It should be noted that the processes of steps 302 to 303 above can be referred to Figure 4 , Figure 4 is a flowchart of generating data collection code provided by the embodiments of the present application. After obtaining at least one index corresponding to the original index, the server can obtain the database name of at least one index from the index system through step 401, obtain the data table name of at least one index from the index system through step 402, determine whether the data table corresponding to the data table name of at least one index has been configured with an automatic reporting function through step 403, if the data table has not been configured with an automatic reporting function, the server can generate data reporting code and data reporting protocols through step 404, obtain the data field name of at least one index from the index system through step 405, if the data table has been configured with an automatic reporting function, the server can directly obtain the data field name of at least one index from the index system through step 405, and then generate code for obtaining field data corresponding to at least one index from the reported data through step 406 to obtain the final data collection code.

[0084] 304、The server obtains the first data corresponding to the at least one index through the data collection code.

[0085] In a possible implementation, the server can run the data collection code, determine the data table corresponding to the at least one index from the database corresponding to the at least one index, and then determine the data field corresponding to the at least one index in the data table, so as to obtain the first data corresponding to the at least one index from the data field.

[0086] 305、The server adds the association between the original index and the at least one index to the corresponding position in the data processing code template respectively, and obtains the data processing code.

[0087] It should be noted that after the first data corresponding to the at least one index is collected, the first data corresponding to the at least one index often needs to be processed to obtain the second data corresponding to the original index. The association in the index system can clearly indicate the calculation rule of the second data corresponding to the original index and the first data corresponding to the at least one index, so that the server can directly process the first data corresponding to the at least one index according to the association to obtain the second data corresponding to the original index. For example, for the original index "actual payment amount" and the corresponding "order amount" and "discount amount", the index system can clearly give the association between "actual payment amount" and "order amount" and "discount amount" as:

[0088] "actual payment amount" = "order amount" - "discount amount"

[0089] Based on this, the server can clearly obtain the calculation rule between the first data corresponding to the at least one index and the second data corresponding to the original index, and then can automatically generate data processing code based on the calculation rule to process the first data corresponding to the at least one index to obtain the second data corresponding to the original index, without the data demand party and the technical personnel needing to communicate multiple times on the second data corresponding to the original index, the first data corresponding to the at least one index, and the calculation rule between the first data and the second data, avoiding any communication link error or ambiguity affecting the final data result, and improving the efficiency and accuracy of data processing.

[0090] The data processing code can further include code for storing the second data corresponding to the original index. If the second data needs to be stored, the database information of the second data is added to the corresponding position in the data storage template code to obtain data storage code. The data storage code is used to store the second data. Through the data storage code, the database information of the second data is stored in the index system as the database information corresponding to the original index. By storing the second data corresponding to the original index, the second data corresponding to the original index can be directly obtained from the database when needed in the future, without the need to process the first data corresponding to at least one index again, thereby improving the data processing efficiency.

[0091] It should be noted that the first data corresponding to at least one index has been collected in step 304, and the server can automatically generate data processing code based on the association relationship between the indexes in the index system. The generation process of the data processing code can be referred to Figure 5 , Figure 5 is a flowchart for generating data processing code provided by an embodiment of the present application. Through steps 301 to 303, the process of obtaining at least one index corresponding to the original index from the index system, obtaining the storage identifier of the at least one index in the index system, and generating the data collection code for obtaining the first data corresponding to the at least one index in steps 501 to 503 has been realized. Then, the calculation rule of the original index and the at least one index can be obtained from the index system through step 504, the code for processing the first data corresponding to the at least one index according to the calculation rule can be generated through step 505, and whether the second data corresponding to the original index needs to be stored can be judged through step 506. If the second data corresponding to the original index does not need to be stored, the generation of the data processing code ends. If the second data corresponding to the original index needs to be stored, the code for storing the second data corresponding to the original index can be generated through step 507, and the database information of the stored second data corresponding to the original index can be recorded in the index system through step 508, and the generation of the data processing code ends. When judging whether the second data corresponding to the original index needs to be stored, the use frequency of the original index can be used for judgment. When the use frequency of the original index is greater than a preset threshold, the second data corresponding to the original index is stored. By storing the second data corresponding to the original index when the use frequency of the original index is greater than the preset threshold, only the second data corresponding to the original index with high use frequency can be stored, and the storage space occupied by the data corresponding to the index with low use frequency is avoided, thereby improving the storage efficiency of the server and the processing speed of the server.

[0092] By generating the data processing code, the server can process the first data corresponding to the at least one indicator by itself to obtain the second data corresponding to the original indicator, without the need for technical personnel to manually process the data, thereby improving the efficiency of data processing and further improving the efficiency of data processing.

[0093] 306、The server processes the first data through the data processing code to obtain the second data corresponding to the original indicator.

[0094] In a possible implementation, the server can run the data processing code, perform calculation based on the first data, and obtain the second data corresponding to the original indicator.

[0095] 307、The server adds the database information of the second data to the corresponding position in the data acquisition template code to obtain data acquisition code, and the data acquisition code is used to acquire the second data corresponding to the original indicator from the database.

[0096] It should be noted that the second data corresponding to the original indicator obtained by processing can have different application scenarios, and the data output target is also different for different application scenarios. Several common data application scenarios and data output methods are as follows:

[0097] (1) Statistical scenario: the data demander needs statistical results of data, such as statistical values and statistical charts, and the data output target of this scenario is a data statistical platform or a data statistical result report.

[0098] (2) Analysis scenario: the data demander needs to use data in a certain analysis process, including but not limited to data analysis report and artificial intelligence (AI) analysis, and the data output target of this scenario is a data file required by the demander.

[0099] (3) Data opening: the data demander needs to open the data to a third party for query, including but not limited to internal operation personnel and external users, and the data output target of this scenario is a data query interface.

[0100] For the above different data application scenarios, the output of the data can be divided into two steps:

[0101] First, provide a query interface of the second data corresponding to the original indicator;

[0102] Second, obtain the second data corresponding to the original indicator from the query interface and process the second data into a required data file format.

[0103] The first step is required by all application scenarios, and the second step has different processing methods according to different application scenarios, such as calling the interface of the data statistics platform for data transmission, or processing the data into a file of a certain format for user saving. It should be noted that the processing of the above two steps is not related, that is, the second step does not need to know which index the processed data corresponds to, and only needs to provide a general data conversion method to process the data corresponding to any index. Based on this, the server can first generate index data application code based on the index system, and then generate several general data application codes with high usage frequency according to the usage frequency. The specific process can be referred to step 308, which will not be described here.

[0104] 308、The server adds the interface identifier of the data query interface corresponding to the second data to the data query interface template code in the corresponding position to obtain the data query interface code. The data acquisition code and the data query interface code are used as the index data application code, and the data query interface code is used to query the second data.

[0105] The process corresponding to steps 307 to 308 is the process of generating index data application code based on the index system. The above process can be referred to the flowchart shown in Figure 6 The flowchart shown in Figure 6 is a flowchart for generating index data application code provided by an embodiment of the present application. The server can obtain the index identifier of the original index from the index system through step 601, generate the code for obtaining the data corresponding to the original index according to the index identifier through step 602, that is, the data acquisition code, and then generate the data query interface code for the data corresponding to the original index through step 603, and finally obtain the target index data application code. Through the target index data application code, relevant personnel can query the second data corresponding to the original index through the set interface.

[0106] It should be noted that after generating the index data application code, the server can also generate several commonly used general data application codes to process the second data corresponding to the original index to obtain a data file meeting the output requirements. In one possible implementation, the server can add the platform identifier of the statistics platform to the first general data application template code in the corresponding position to obtain the first general data application code, and the first general data application code is used to transmit the second data to the statistics platform. In another possible implementation, the server can add the file format of the character-separated value file to the second general data application template code in the corresponding position to obtain the second general data application code, and the second general data application code is used to process the second data into a character-separated value file.

[0107] The process of generating the general data application code through the above two implementation manners can be seen from Figure 7 , Figure 7 is a flowchart for generating a general data application code provided by the embodiment of the present application. The server can generate, through step 701, code for transmitting the second data to the statistical platform according to the original index, that is, the first general data application code, and can also generate, through step 702, code for processing the second data into a CSV file according to the original index, that is, the second general data application code. Optionally, the server can also generate other general data application codes with higher usage frequency. The specific general data application code generated can be determined by the specific use scenario, which is not limited in the embodiment of the present application. The index data application code and the general data application code can be collectively referred to as data application code.

[0108] 309、The server outputs the second data as the target data of the data requirement information through the index data application code.

[0109] Wherein, when outputting the second data, an interface for querying the second data can be provided through only the index data application code, and the second data can be further processed through the general data application code to obtain a data file meeting the output target requirements, which is not limited in the embodiment of the present application.

[0110] It should be noted that the processes of steps 301 to 309 can be implemented by a data processing system as shown in Figure 8 , Figure 8 is an architecture diagram of a data processing system provided by the embodiment of the present application. Referring to Figure 8 , the demand side manages the index system 801 to maintain the consistency of index terms. The index can be generated to generate the code. The demand side can generate the data requirement based on the index in the index system 801, and then automatically generate the code according to the association relationship between the indexes in the index system and the database information of the indexes, to realize the data collection process 802, the data processing process 803 and the data application process 804. Specifically, the server can generate the data collection code through steps 301 to 303, generate the data processing code through step 305, generate the data application code through steps 307 to 308, to obtain the code covering the data source, data processing and data providing parts, so that the data requirement is automatically realized without manual processing, greatly improving the efficiency of data processing, reducing the error probability, improving the accuracy of data processing and improving the efficiency of development and maintenance.

[0111] It should be noted that the processes of steps 301 to 309 can be divided into code generation and code running. The above is an example of generating a code and immediately running the code to realize the processing of data, and then generating the next code. In more possible implementation manners, all kinds of codes can be generated at one time, and then the codes are run to realize the processing of data at one time. The embodiment of the application does not limit which way is used.

[0112] The scheme provided by the embodiment of the application obtains at least one index corresponding to the original index from the index system according to the data requirement information, generates the data collection code and the data processing code specially used for collecting the data corresponding to the original index, collects the target data, and then outputs the collected target data. Since the names of the indexes in the index system and the association relationship between the indexes are pre-set, at least one index corresponding to the original index can be directly obtained from the index system, without the need for the technical personnel to communicate for the indexes to be collected, so as to avoid the misalignment of the terms, directly collect the data through the code, avoid the misunderstanding of the technical personnel for the association relationship of the indexes, so that the collected data is not the actual required data, improve the accuracy of collecting the data, and then improve the data processing accuracy. The original index and the at least one index are obtained through the index system managed by the demand side, without the need for the demand side and the technical personnel to repeatedly check the database, the data table and the data field of the stored data, so as to avoid the different descriptions of the same index between the demand side and the technical personnel, cause the inconsistency of the terms in the communication, and the high communication cost. The inconsistency of the communication terms is effectively reduced, the communication cost is reduced, and the early communication efficiency is improved. Moreover, the data can be directly understood from the indexes, which can greatly reduce the development time, the maintenance cost and the error probability of the data requirement. Through the data collection code, the data processing code and the data application code, the whole process from data collection to data processing and then to data application can be realized, without the need for the technical personnel to manually process, avoid the errors that may occur in the manual processing, ensure that the generated code and the obtained data completely comply with the index system specified by the demand side, ensure the accuracy of the data processing, and greatly reduce the error rate of the data processing. In addition, the data is directly processed through the code, without the need for the technical personnel to perform a lot of repetitive labor, greatly improve the efficiency of the technical personnel, improve the data processing efficiency, and enable the use, analysis and statistics of the data to be accurately and quickly promoted.

[0113] All the optional technical solutions can be combined to form optional embodiments of the application, which will not be described here.

[0114] Figure 9 is a structural schematic diagram of a data processing device provided by the embodiment of the application, referring to Figure 9 The device comprises:

[0115] The index obtaining module 901 is configured to obtain at least one index corresponding to an original index in the data requirement information from an index system in response to obtaining the data requirement information, the index system being used to represent an association relationship between indexes and database information of the indexes.

[0116] The code generating module 902 is configured to generate data collection code and data processing code according to the at least one index and the data requirement information, the data collection code being used to obtain first data corresponding to the at least one index from a database, and the data processing code being used to process the first data of the at least one index based on the data requirement information.

[0117] The data processing module 903 is configured to obtain the first data corresponding to the at least one index through the data collection code, and generate second data corresponding to the original index through the data processing code and the first data.

[0118] The data output module 904 is configured to output target data meeting the data requirement information based on the second data and the data requirement information.

[0119] The apparatus provided by the embodiments of the present application obtains the target data by obtaining at least one index corresponding to an original index from an index system according to data requirement information, generating data collection code and data processing code specially used to obtain data corresponding to the original index, and then outputting the obtained target data. Since the names of indexes in the index system and the association relationship between the indexes are pre-set, the at least one index corresponding to the original index can be directly obtained from the index system, without the need for technical personnel to communicate with respect to the indexes to be obtained, thereby avoiding the problem of misaligned terms, and then the data is directly obtained through the code, thereby avoiding the situation that the technical personnel misunderstand the association relationship of the indexes, resulting in that the obtained data is not the actual required data, improving the accuracy of obtaining data, and further improving the data processing accuracy.

[0120] In a possible implementation, the index obtaining module 901 is configured to locate the original index in the data requirement information in the index system, locate indexes having an association relationship with the original index according to the association relationship between the indexes in the index system, and take the located indexes as the at least one index.

[0121] In a possible implementation, the code generating module 902 includes a first code generating submodule, and the first code generating submodule includes an obtaining unit and an adding unit.

[0122] The obtaining unit is configured to obtain a database name, a data table name, and a data field name corresponding to the at least one index according to the database information of the indexes in the index system.

[0123] The adding unit is configured to add the database name, the data table name and the data field name into corresponding positions in the data collection template code, so as to obtain the data collection code.

[0124] In a possible implementation, the obtaining unit is configured to obtain, according to the database information of the indicators in the index system, the database name and the data table name corresponding to the at least one indicator, and if the data table corresponding to the data table name is configured with the automatic reporting function, obtain, according to the database information of the indicators in the index system, the data field name corresponding to the at least one indicator, and if the data table corresponding to the data table name is not configured with the automatic reporting function, add the data table name into a corresponding position in a data reporting template code to obtain a data reporting code, and obtain, through the data reporting code, the data field name corresponding to the at least one indicator from the index system, where the data reporting code is used for reporting data in the data table in real time.

[0125] In a possible implementation, the code generation module 902 includes a second code generation sub-module.

[0126] The second code generation sub-module is configured to add the association relationship between the original indicator and the at least one indicator into corresponding positions in a data processing code template to obtain the data processing code.

[0127] In a possible implementation, the apparatus further includes:

[0128] The adding module is configured to, if the second data needs to be stored, add database information of the second data into a corresponding position in a data storage template code to obtain a data storage code, where the data storage code is used for storing the second data.

[0129] The storing module is configured to store, through the data storage code, the database information of the second data into the index system as the database information corresponding to the original indicator.

[0130] In a possible implementation, the data output module 904 is configured to add the database information of the second data into a corresponding position in a data obtaining template code to obtain a data obtaining code, where the data obtaining code is used for obtaining the second data corresponding to the original indicator from the database; add an interface identifier of a data query interface corresponding to the second data into a corresponding position in a data query interface template code to obtain a data query interface code, where the data query interface code is used for querying the second data; and take the data obtaining code and the data query interface code as an indicator data application code, and output, through the indicator data application code, the second data as target data of the data requirement information.

[0131] In a possible implementation, the apparatus further includes:

[0132] The first code generation module is configured to add the platform identifier of the statistical platform to the first general data application template code at a corresponding position, to obtain a first general data application code, the first general data application code being used to transmit the second data to the statistical platform.

[0133] The second code generation module is configured to add the file format of the character-separated value file to the second general data application template code at a corresponding position, to obtain a second general data application code, the second general data application code being used to process the second data into the character-separated value file.

[0134] It should be noted that the data processing apparatus provided in the above embodiments is only used as an example for the division of the above functional modules, and in actual application, the above functions can be completed by different functional modules according to needs, that is, the internal structure of the server is divided into different functional modules to complete all or part of the functions described above. In addition, the data processing apparatus and the data processing method provided in the above embodiments belong to the same concept, and the specific implementation process is described in the method embodiments, which will not be repeated here.

[0135] Figure 10 is a structural schematic diagram of a server provided by the embodiments of the present application. The server 1000 can have great differences due to different configurations or performances, and can include one or more processors (Central Processing Units, CPUs) 1001 and one or more memories 1002. The one or more memories 1002 store at least one program code, which is loaded and executed by the one or more processors 1001 to implement the method provided by each method embodiment described above. Of course, the server 1000 can also have a wired or wireless network interface, a keyboard, an input and output interface, and other components for realizing the functions of the device, and the details are not described here.

[0136] In the exemplary embodiments, a computer readable storage medium, such as a memory including program code, is also provided, and the above program code can be executed by a processor to complete the data processing method in the above embodiments. For example, the computer readable storage medium can be a Read-Only Memory (ROM), a Random Access Memory (RAM), a Compact Disc Read-Only Memory (CD-ROM), a magnetic tape, a floppy disk, and an optical data storage device, etc.

[0137] Those skilled in the art can understand that all or part of the steps of the above-mentioned embodiments can be completed by hardware, or by program code related hardware, and the program can be stored in a computer readable storage medium. The storage medium mentioned above can be a read-only memory, a magnetic disk or an optical disk, etc.

[0138] The above is only an optional embodiment of the present application, and is not used to limit the present application. Any modification, equivalent replacement, improvement, etc. made within the spirit and principles of the present application shall be included in the protection scope of the present application.

Claims

1. A data processing method, characterized by, The method comprises: in response to obtaining data requirement information, obtaining at least one index corresponding to the original index in the data requirement information from an index system, the index system being used to represent the correlation between indexes and database information of the indexes; generating data collection code and data processing code according to the at least one index and the data requirement information, the data collection code being used to obtain first data corresponding to the at least one index from a database, and the data processing code being used to process the first data of the at least one index based on the data requirement information; obtaining the first data corresponding to the at least one index through the data collection code, processing the first data through the data processing code, and obtaining second data corresponding to the original index; adding database information of the second data to a corresponding position in data acquisition template code to obtain data acquisition code, the data acquisition code being used to obtain the second data corresponding to the original index from the database; adding an interface identifier of a data query interface corresponding to the second data to a corresponding position in data query interface template code to obtain data query interface code; taking the data acquisition code and the data query interface code as index data application code, the data query interface code being used to query the second data; and outputting the second data as target data of the data requirement information through the index data application code.

2. The method of claim 1, wherein, The response to obtaining data requirement information comprises: locating an original index in the data requirement information in the index system; locating indexes having a correlation with the original index according to the correlation between indexes in the index system; taking the located indexes as the at least one index.

3. The method of claim 1, wherein, The generation process of the data collection code comprises: obtaining a database name, a data table name and a data field name corresponding to the at least one index according to database information of indexes in the index system; adding the database name, the data table name and the data field name to corresponding positions in data collection template code respectively to obtain the data collection code.

4. The method of claim 3, wherein, The obtaining of the database name, the data table name and the data field name corresponding to the at least one index according to the database information of indexes in the index system comprises: obtaining a database name and a data table name corresponding to the at least one index according to the database information of indexes in the index system; if a data table corresponding to the data table name has configured an automatic reporting function, obtaining a data field name corresponding to the at least one index according to the database information of indexes in the index system; if a data table corresponding to the data table name has not configured an automatic reporting function, adding the data table name to a corresponding position in data reporting template code to obtain data reporting code, and obtaining a data field name corresponding to the at least one index from the index system through the data reporting code, the data reporting code being used to report data in the data table in real time.

5. The method of claim 1, wherein, The generation process of the data processing code comprises: adding the association between the original index and the at least one index to corresponding positions in a data processing code template respectively to obtain the data processing code.

6. The method of claim 1, wherein, After the first data is processed by the data processing code to obtain the second data corresponding to the original index, the method further comprises: If the second data needs to be stored, database information of the second data is added to corresponding positions in a data storage code template to obtain a data storage code, the data storage code being used for storing the second data; the database information of the second data is stored to the index system as the database information corresponding to the original index by the data storage code.

7. The method of claim 1, wherein, After the interface identifier of the data query interface corresponding to the second data is added to corresponding positions in a data query interface template code to obtain a data query interface code, the method further comprises at least one of the following: a platform identifier of a statistical platform is added to corresponding positions in a first general data application template code to obtain a first general data application code, the first general data application code being used for transmitting the second data to the statistical platform; a file format of a character-separated values file is added to corresponding positions in a second general data application template code to obtain a second general data application code, the second general data application code being used for processing the second data into a character-separated values file.

8. A data processing apparatus, characterized by, The apparatus comprises: an index acquisition module, configured to acquire at least one index corresponding to an original index in data requirement information from an index system in response to acquiring the data requirement information, the index system being used for representing an association between indexes and database information of the indexes; a code generation module, configured to generate data collection code and data processing code according to the at least one index and the data requirement information, the data collection code being used for acquiring first data corresponding to the at least one index from a database, and the data processing code being used for processing the first data of the at least one index based on the data requirement information; a data processing module, configured to acquire the first data corresponding to the at least one index by the data collection code, and generate second data corresponding to the original index by the data processing code and the first data; a data output module, configured to add database information of the second data to corresponding positions in a data acquisition template code to obtain data acquisition code, the data acquisition code being used for acquiring the second data corresponding to the original index from the database; add an interface identifier of a data query interface corresponding to the second data to corresponding positions in a data query interface template code to obtain data query interface code; and take the data acquisition code and the data query interface code as index data application code, the data query interface code being used for querying the second data; and output the second data as target data of the data requirement information by the index data application code.

9. The apparatus of claim 8, wherein, The index acquisition module is configured to locate an original index in the data requirement information in the index system, locate indexes having a correlation relationship with the original index according to the correlation relationship between the indexes in the index system, and take the located indexes as the at least one index.

10. The apparatus of claim 8, wherein, The code generation module comprises a first code generation submodule, which comprises an acquisition unit and an adding unit. The acquisition unit is configured to acquire a database name, a data table name, and a data field name corresponding to the at least one index according to database information of the indexes in the index system. The adding unit is configured to add the database name, the data table name, and the data field name to corresponding positions in data collection template code respectively to obtain the data collection code.

11. The apparatus of claim 10, wherein, The acquisition unit is configured to acquire a database name and a data table name corresponding to the at least one index according to database information of the indexes in the index system, acquire a data field name corresponding to the at least one index according to the database information of the indexes in the index system if the data table corresponding to the data table name has been configured with an automatic reporting function, add the data table name to a corresponding position in data reporting template code to obtain data reporting code if the data table corresponding to the data table name has not been configured with the automatic reporting function, acquire the data field name corresponding to the at least one index from the index system through the data reporting code, and use the data reporting code to report data in the data table in real time.

12. The apparatus of claim 8, wherein, The code generation module comprises a second code generation submodule. The second code generation submodule is configured to add the correlation relationship between the original index and the at least one index to corresponding positions in data processing code template respectively to obtain the data processing code.

13. A server, characterized by The server comprises one or more processors and one or more memories, and at least one program code is stored in the one or more memories. The program code is loaded and executed by the one or more processors to implement operations performed by the data processing method according to any one of claims 1 to 7.

14. A computer-readable storage medium, characterized in that, At least one program code is stored in the computer readable storage medium. The program code is loaded and executed by the processor to implement operations performed by the data processing method according to any one of claims 1 to 7.

Images