Data processing method and apparatus, device, and storage medium

Abstract

The application provides a data processing method, device and equipment and a storage medium. The method comprises the following steps: in response to a data access request, obtaining target business data associated with the data access request; querying a target data dictionary corresponding to the target business data, and configuring a target access file of the target business data according to the target data dictionary; calling a preset scheduling task module to perform data detection on the target business data, obtaining target check data, accessing the target check data to the target access file, and generating a data cache file; obtaining a target data table corresponding to the target check data, and loading the data cache file to the target data table. The target business data is automatically accessed through the data dictionary and the scheduling task module, the batch data access efficiency is improved, and the access error rate is reduced.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of computers, in particular to a data processing method and device, equipment and a storage medium. BACKGROUND

[0002] At present, with the rapid development of big data operation and analysis, existing financial, banking and other service enterprises need to apply big data for analysis and processing when processing related businesses. Before big data analysis is adopted, data access is needed, and the existing data access method needs manual configuration of a large number of files, resulting in a large amount of manpower and time being consumed for batch access of a large number of tables, and manual configuration of jobs is prone to errors, which may cause significant business risks and affect downstream development progress and project delivery. SUMMARY

[0003] The embodiments of the present application provide a data processing method, device, equipment and storage medium, aiming at solving the technical problems of low access efficiency and high error rate in the data docking process in the prior art.

[0004] In one aspect, the embodiments of the present application provide a data processing method, which comprises the following steps:

[0005] In response to a data access request, target business data associated with the data access request is obtained;

[0006] A target data dictionary corresponding to the target business data is queried, and a target access file of the target business data is configured according to the target data dictionary;

[0007] A preset scheduling task module is called to perform data detection on the target business data, target verification data is obtained, the target verification data is accessed to the target access file, and a data cache file is generated;

[0008] A target data table corresponding to the target verification data is obtained, and the data cache file is loaded to the target data table.

[0009] In one possible implementation of the present application, the target data dictionary corresponding to the target business data is queried, and the target access file of the target business data is configured according to the target data dictionary, which comprises:

[0010] A preset data dictionary database is queried to obtain the target data dictionary corresponding to the target business data;

[0011] Data structure fields and storage rule fields in the target data dictionary are obtained, and a target table creation statement of the target business data is configured according to the data structure fields and the storage rule fields;

[0012] Input the target table building statement into a preset template file, and configure a target access file of the target business data.

[0013] In a possible implementation manner of the present application, the preset dispatch task module is called to perform data detection on the target business data to obtain target verification data, the target verification data is accessed to the target access file, and a data cache file is generated, including:

[0014] The dispatch task module is obtained to obtain a data detection submodule and a detection log submodule in the dispatch task module.

[0015] The data detection submodule is called to perform data detection on the target business data to obtain target verification data, and the detection log submodule is called to output data detection logs.

[0016] The target verification data and the data detection logs are recorded in the target access file to obtain a data cache file.

[0017] In a possible implementation manner of the present application, the data detection submodule is called to perform data detection on the target business data to obtain target verification data, and the detection log submodule is called to output data detection logs, including:

[0018] The business detection identifier and the type detection identifier in the data detection submodule are obtained.

[0019] The business detection identifier is compared with a business identifier of the target business data, and the type detection identifier is compared with a data type of the target business data.

[0020] The target business data with the same business identifier and the same data type is set as target verification data.

[0021] The target verification data is recorded in the detection log submodule to obtain data detection logs.

[0022] In a possible implementation manner of the present application, the target data table corresponding to the target verification data is obtained, and the target verification data in the data cache file is loaded to the target data table, including:

[0023] The business type and the data type of the target verification data are obtained.

[0024] The business type and the data type are matched with a preset data table in a preset data cluster to obtain a target data table.

[0025] The preset synchronization module is called to transmit the data cache file into the target data cluster, and target check data in the data cache file is loaded into a target data table in the target data cluster according to a preset access configuration.

[0026] In a possible implementation manner of the application, the preset scheduling task module is called to perform data detection on the target business data to obtain target check data, and the target check data is accessed to the target access file to generate a data cache file, including:

[0027] A preset release configuration file is acquired, and the target access file is processed according to the release configuration file to obtain a target release file.

[0028] The preset scheduling task module is called to perform data detection on the target business data to obtain target check data, and the target check data is accessed to the target release file to generate a data cache file.

[0029] In a possible implementation manner of the application, after the target data dictionary corresponding to the target business data is queried and the target access file of the target business data is configured according to the target data dictionary, the method further includes:

[0030] An update data dictionary associated with the configuration update request is acquired in response to the configuration update request.

[0031] An update structure field and an update rule field in the update data dictionary are acquired, and an update table statement is configured according to the update structure field and the update rule field.

[0032] The target access file is updated and configured according to the update table statement to obtain an updated target access file.

[0033] On the other hand, the application provides a data processing apparatus, including:

[0034] The data acquisition module is configured to acquire target business data associated with a data access request in response to the data access request.

[0035] The file configuration module is configured to query a target data dictionary corresponding to the target business data, and configure a target access file of the target business data according to the target data dictionary.

[0036] The data detection module is configured to call a preset scheduling task module to perform data detection on the target business data to obtain target check data, and access the target check data to the target access file to generate a data cache file.

[0037] a data loading module configured to obtain a target data table corresponding to the target verification data, and load the data cache file into the target data table.

[0038] In another aspect, the present application also provides a data processing device, which comprises:

[0039] one or more processors;

[0040] a memory; and

[0041] one or more application programs, wherein the one or more application programs are stored in the memory and configured to be executed by the processor to implement the steps of the data processing method.

[0042] In another aspect, the present application also provides a computer readable storage medium having a computer program stored thereon, wherein the computer program is loaded by a processor to execute the steps of the data processing method.

[0043] In the present application, target business data associated with a data access request is obtained in response to the data access request; a target data dictionary corresponding to the target business data is queried, and a target access file of the target business data is configured according to the target data dictionary; a preset scheduling task module is called to perform data detection on the target business data to obtain target verification data, the target verification data is accessed to the target access file to generate a data cache file; a target data table corresponding to the target verification data is obtained, and the data cache file is loaded into the target data table. The target business data is automatically accessed through the data dictionary and the scheduling task module, the batch data access efficiency is improved, and the access error rate is reduced. BRIEF DESCRIPTION OF DRAWINGS

[0044] In order to more clearly illustrate the technical solutions in the embodiments of the present application, the drawings needed in the embodiment description will be briefly introduced. Obviously, the drawings in the following description are only some embodiments of the present application, and other drawings can be obtained by those skilled in the art without creative labor.

[0045] Figure 1 A scene schematic diagram of the data processing method of the embodiments of the present application;

[0046] Figure 2 A flowchart of one embodiment of the data processing method in the embodiments of the present application;

[0047] Figure 3 A flowchart of one embodiment of the data processing method in the embodiments of the present application;

[0048] Figure 4 a flowchart of one embodiment of a data processing method provided by an embodiment of the present application for configuring a target access file;

[0049] Figure 5 a structural schematic diagram of one embodiment of a data processing device provided by an embodiment of the present application;

[0050] Figure 6 a structural schematic diagram of one embodiment of a data processing device provided by an embodiment of the present application. DETAILED DESCRIPTION

[0051] The technical solutions in the embodiments of the present application will be clearly and completely described in conjunction with the drawings in the embodiments of the present application. Obviously, the described embodiments are only part of the embodiments of the present application, rather than all the embodiments. Based on the embodiments in the present application, all other embodiments obtained by those skilled in the art without creative work fall within the scope of protection of the present application.

[0052] In the description of the present application, it should be understood that the terms "center", "longitudinal", "lateral", "length", "width", "thickness", "upper", "lower", "front", "rear", "left", "right", "vertical", "horizontal", "top", "bottom", "inner", "outer" and the like indicate the orientation or positional relationship based on the orientation or positional relationship shown in the drawings, and are only for the convenience of describing the present application and simplifying the description, and do not indicate or imply that the device or element referred to must have a particular orientation, be constructed and operated in a particular orientation, and therefore cannot be understood as a limitation on the present application. In addition, the terms "first", "second" are only for the purpose of description, and cannot be understood as indicating or implying relative importance or implicitly indicating the number of the technical features indicated. Therefore, the features limited by "first", "second" can explicitly or implicitly include one or more of the features. In the description of the present application, the meaning of "multiple" is two or more, unless otherwise specifically limited.

[0053] In the present application, the word "exemplary" is used to mean "serving as an example, instance, or illustration." Any implementation described as "exemplary" in the present application is not necessarily to be construed as preferred or advantageous over other implementations. The following description is presented to enable any person skilled in the art to make and use the application. In the following description, for purposes of explanation, specific details are set forth. It will be apparent to those skilled in the art that the present application can be practiced without using these specific details. In other instances, well-known structures and processes are not described in detail in order to avoid obscuring the description of the present application. Thus, the present application is not intended to be limited to the embodiments shown, but is to be accorded the widest scope consistent with the principles and features presented herein.

[0054] Currently, with the rapid development of big data computing and analysis, existing financial, banking, and other service enterprises need to apply big data for analysis and processing when handling related business. However, before adopting big data analysis, data access is required. Existing data access methods require manual configuration of a large number of files, resulting in a significant consumption of manpower and time for batch access of a large number of tables. Furthermore, manual configuration is prone to errors, which may cause significant business risks and affect downstream development progress and project delivery.

[0055] Based on this, this application proposes a data processing method, apparatus, device, and computer-readable storage medium to solve the technical problems of low access efficiency and high error rate in the data docking process in the prior art.

[0056] The data processing method in this embodiment of the invention is applied to a data processing device, which is disposed in a data processing equipment. The data processing equipment is provided with one or more processors, a memory, and one or more application programs, wherein the one or more application programs are stored in the memory and configured to be executed by the processor to implement the data processing method. The data processing equipment can be a smart terminal, such as a mobile phone, tablet computer, network device, and smart computer. Optionally, the data processing equipment can also be a server or a service cluster composed of multiple servers.

[0057] like Figure 1 As shown, Figure 1 This is a schematic diagram of a data processing method according to an embodiment of the present application. The data processing scenario in this embodiment includes a data processing device 100 (which integrates a data processing unit) and a target business terminal 200. The data processing device 100 has a computer-readable storage medium corresponding to the data processing method running to execute the steps of the data processing method. The target business terminal 200 is a business terminal that accesses data from the data processing device 100.

[0058] Understandable Figure 1 The data processing device in the data processing method scenario shown, or the device included in the data processing device, does not constitute a limitation on the embodiments of the present invention. That is, the number or type of data processing device in the data processing method scenario, or the number or type of device included in each device, does not affect the overall implementation of the technical solution in the embodiments of the present invention, and can all be considered as equivalent substitutions or derivatives of the technical solutions claimed in the embodiments of the present invention.

[0059] In this embodiment of the invention, the data processing device 100 is mainly used for: responding to a data access request and obtaining the target business data associated with the data access request;

[0060] Query the target data dictionary corresponding to the target service data, and configure the target access file of the target service data according to the target data dictionary;

[0061] The preset scheduling task module is invoked to perform data detection on the target business data, obtain target verification data, and then the target verification data is connected to the target access file to generate a data cache file.

[0062] Obtain the target data table corresponding to the target verification data, and load the data cache file into the target data table.

[0063] The data processing device 100 in this embodiment of the invention can be an independent data processing device, such as a mobile phone, tablet computer, network device, server and smart computer, or a data processing network or data processing cluster composed of multiple data processing devices.

[0064] This application provides a data processing method, apparatus, device, and computer-readable storage medium, which will be described in detail below.

[0065] It will be understood by those skilled in the art that Figure 1 The application environment shown is only one application scenario related to the solution of this application and does not constitute a limitation on the application scenario of this application. Other application environments may include more than one application scenario. Figure 1 The number of data processing devices shown, or the data processing network connections, for example Figure 1 Only one data processing device is shown in the document. It is understood that the scenario of this data processing method may also include one or more data processing devices, which are not specifically limited here. The data processing device may also include a memory for storing target business data and other data.

[0066] It should be noted that, Figure 1 The schematic diagram of the data processing method shown is merely an example. The scenarios of the data processing method described in the embodiments of the present invention are intended to more clearly illustrate the technical solutions of the embodiments of the present invention and do not constitute a limitation on the technical solutions provided in the embodiments of the present invention.

[0067] Based on the scenarios described above, various embodiments of the data processing method disclosed in this invention are proposed.

[0068] like Figure 2 As shown, Figure 2 This is a flowchart illustrating one embodiment of the data processing method in this application. The data processing method includes the following steps 201 to 204:

[0069] 201、in response to the data access request, obtaining target business data associated with the data access request;

[0070] The data processing method in the embodiment is applied to a data processing device, and the type and number of the data processing device are not specifically limited, that is, the data processing device can be one or more intelligent terminals or servers. In a specific embodiment, the data processing device is an intelligent computer.

[0071] Specifically, the data processing device is configured to obtain target business data associated with a data access request in response to the data access request, and perform data access on the target business data and load the target business data into a target data table corresponding to the target business data.

[0072] Specifically, the data processing device receives a data access request during operation, wherein the data access request is an operation instruction for driving the data processing device to communicate with a target business terminal, obtaining target business data of the target business terminal, and performing data access processing on the target business data. Optionally, the triggering mode of the data access request is not specifically limited here, that is, the data access request can be triggered by a user, for example, a user who is a big data analyst actively triggers a data access request for specific target business data by clicking a data access button in the data processing device. In addition, the data access request can also be triggered automatically by the data processing device, for example, the data processing device is preconfigured with a timing access process to automatically trigger a data access request at a specified time period.

[0073] Specifically, after the data processing device obtains the data access request, the data processing device obtains each target business terminal corresponding to the data access request in response to the data access request, communicates with the target business terminal, and calls a preset big data collection script to collect target business data in each target business terminal. The target business data is business data generated by each target business terminal performing corresponding business. Optionally, the target business data can be at least one of structured data, unstructured data and semi-structured data.

[0074] After the data processing device obtains the target business data associated with the data access request, the data processing device also performs data access processing on the target business data, thereby integrating the target business data into a big data platform in the data processing device, so that the data processing device can perform big data analysis according to the target business data in the subsequent process.

[0075] 202、querying a target data dictionary corresponding to the target business data, and configuring a target access file of the target business data according to the target data dictionary;

[0076] Specifically, the data processing device, after collecting the target service data of each target service terminal, further queries a target data dictionary corresponding to the target service data, and configures a target access file of the target service data according to the target data dictionary.

[0077] Specifically, the data processing device pre-acquires, from an upstream terminal, an access data dictionary corresponding to a target data field of each service data, and stores the access data dictionary in a preset data dictionary database. After collecting each target service data, the data processing device acquires a terminal identifier and a service identifier of each target service data, inputs the terminal identifier and the service identifier into the data dictionary database for matching, and acquires a target data dictionary matched with the terminal identifier and the service data. The target data dictionary is a rule set for defining and describing data items, data structures, and data processing logic of the target service data.

[0078] After acquiring the target data dictionary corresponding to the target service data, the data processing device inserts the target data dictionary into a mysql library, and performs data access processing on the target service data according to the target data dictionary. Specifically, the data processing device acquires a data structure field and a storage rule field in the target data dictionary, and configures a target table creation statement of the target service data through the data structure field and the storage rule field. The target table creation statement is a data table configuration statement for configuring a target access file corresponding to the target service data.

[0079] After generating each target table creation statement, the data processing device inputs the target table creation statement into a preset template file for data table configuration, so as to configure a target access file associated with the target service data. The target access file is a data table file for temporarily storing the target service data.

[0080] Optionally, the data processing device, after generating the target access file associated with the target service data, performs release processing on the target access file.

[0081] 203、calling a preset scheduling task module to perform data detection on the target service data, obtaining target verification data, and accessing the target verification data to the target access file to generate a data cache file;

[0082] Specifically, the data processing device, after collecting each target service data, further calls a preset scheduling task module to perform data detection on the target service data, so as to obtain target verification data, and access the target verification data to the target access file to obtain a data cache file.

[0083] Specifically, the data processing device is pre-configured with a scheduling task module, and the target service data collected is subjected to data detection through the scheduling task module. Optionally, in an embodiment, the scheduling task module is a linkdo scheduling task module.

[0084] Specifically, the data processing device acquires a data detection submodule and a detection log submodule in the scheduling task module. The data detection submodule is invoked to perform data detection on the target service data, so as to obtain target verification data, and the detection log submodule is invoked to output a data detection log associated with the target verification data.

[0085] Specifically, after the data processing device performs data detection on the target service data and obtains target verification data and a data detection log, the target verification data and the data detection log are recorded in the target access file, so as to obtain a data cache file.

[0086] 204、acquire a target data table corresponding to the target verification data, and load the data cache file into the target data table.

[0087] Specifically, after the data processing device obtains the target verification data, a file access task is triggered, a target data table corresponding to the target verification data is acquired, and the target verification data in the data cache data is loaded into the target data table.

[0088] Specifically, the data processing device acquires a business type and a data type of the target verification data, and matches the business type and the data type with a preset data table in a preset data cluster, so as to obtain a target data table of the target verification data. Optionally, in an embodiment, the preset data cluster is a hadoop cluster.

[0089] Specifically, the data processing device compares the business type of the target verification data with a business field of a preset data table in the preset data table, and compares the data type of the target verification data with a data item field of the preset data table in the preset data table. If the business type of the target verification data is the same as the business field and the data type is the same as the data item field, the data processing device sets the preset data table as the target data table of the target verification data.

[0090] After the data processing device acquires the target data table of the target verification data, a preset synchronization module is invoked to transmit the data cache file into the target data cluster, and the target verification data in the data cache file is loaded into the target data table in the target data cluster according to a preset access configuration. Optionally, in an embodiment, the preset synchronization module is a filesync2.0 synchronization tool.

[0091] In this embodiment, the data processing device obtains target service data associated with a data access request in response to the data access request, queries a target data dictionary corresponding to the target service data, configures a target access file of the target service data according to the target data dictionary, calls a preset scheduling task module to perform data detection on the target service data to obtain target verification data, accesses the target verification data to the target access file to generate a data cache file, obtains a target data table corresponding to the target verification data, and loads the data cache file to the target data table. The target service data is automatically accessed through the data dictionary and the scheduling task module, the batch data access efficiency is improved, and the access error rate is reduced.

[0092] As shown in Figure 3 , Figure 3 FIG. 1 is a flowchart of an embodiment of a data processing method provided by the present application for verifying target service data, and specifically includes steps 301-304.

[0093] 301, obtaining a service detection identifier and a type detection identifier in the data detection submodule;

[0094] 302, comparing the service detection identifier with a service identifier of the target service data and comparing the type detection identifier with a data type of the target service data;

[0095] 303, setting the target service data with the same service identifier and the same data type as the service detection identifier and the type detection identifier as target verification data;

[0096] 304, entering the target verification data into the detection log submodule to obtain a data detection log.

[0097] Based on the above embodiment, in this embodiment, the data processing device further calls a preset scheduling task module to perform data detection on the target service data after collecting each target service data, and then obtains target verification data.

[0098] Specifically, the data processing device obtains a service detection identifier and a type detection identifier in the data detection submodule, compares the service detection identifier with a service identifier in the target service data, and compares the type detection identifier with a data type of the target service data, to obtain a data detection result.

[0099] Optionally, if the service identifier of the target service data is different from the service detection identifier or the data type of the target service data is different from the type detection identifier, the data processing device determines that the target service data is non-access data and filters the non-access data.

[0100] Optionally, if the service identifier of the target service data is the same as the service detection identifier and the data type of the target service data is the same as the type detection identifier, then the data processing device determines that the target service data is target verification data.

[0101] After acquiring the target verification data, the data processing device records the target verification data into the detection log submodule, and outputs the data detection log carrying the target verification data through the detection log submodule.

[0102] After obtaining the target verification data and data detection log, the data processing device records the target verification data and data detection log into the target access file to obtain the data cache file.

[0103] Optionally, in other embodiments, after generating the target access file, the data processing device also obtains a preset release configuration file and performs release processing on the target access file according to the release configuration file to obtain a target release file. After obtaining the target release file, the data processing device inserts the target verification data after data verification into the target release file to obtain a data cache file.

[0104] After generating the data cache file, the data processing device also calls the preset synchronization module to transfer the data cache file to the target data cluster. According to the preset access configuration, the target verification data in the data cache file is loaded into the target data table in the target data cluster, thereby realizing the data access operation on the target verification data.

[0105] In this embodiment, the data processing device acquires the service detection identifier and type detection identifier from the data detection submodule; compares the service detection identifier with the service identifier of the target service data, and compares the type detection identifier with the data type of the target service data; sets the target service data whose service identifier and service detection identifier are the same, and whose data type is the same as the type detection identifier, as the target verification data; and records the target verification data into the detection log submodule to obtain the data detection log. This enables the detection of service data before data access, reducing the data access error rate.

[0106] like Figure 4 As shown, Figure 4 A flowchart illustrating an embodiment of the data processing method for configuring a target access file provided in this application, specifically including steps 401 to 403:

[0107] 401. Respond to the configuration update request and obtain the update data dictionary associated with the configuration update request;

[0108] 402. Obtain the update structure field and update rule field from the update data dictionary, and configure the update table statement according to the update structure field and the update rule field;

[0109] 403. Update the target access file according to the update table statement to obtain the updated target access file.

[0110] Based on the above embodiments, in this embodiment, after configuring the target access file of the target service data according to the target data dictionary, the data processing device also responds to the configuration update request and updates the configuration of the target access file.

[0111] Specifically, during operation, the data processing device responds to configuration update requests and parses these requests to obtain the updated data dictionary associated with the configuration update request. This updated data dictionary is the one updated by the upstream business terminal based on actual business needs. Optionally, in other embodiments, the data processing device also pre-configures a timed polling process to periodically poll the MySQL database within a preset time period to obtain the updated data dictionary.

[0112] After acquiring the updated data dictionary, the data processing device reads the update structure field and update rule field from the updated data dictionary, and configures the update table statement for the target business data according to the update structure field and update rule field. The update table statement is a set of data table configuration rules for updating the target access file.

[0113] Specifically, after generating the update table statement, the data processing device updates the configuration of the target access file according to the update table statement, thereby obtaining the updated target access file.

[0114] After obtaining the updated target access file, the data processing device imports the obtained target business file into the target access file to obtain the target cache file. It then calls the preset synchronization module to pass the data cache file into the target data cluster. According to the preset access configuration, the target verification data in the data cache file is loaded into the target data table in the target data cluster, thereby realizing the data access operation on the target verification data.

[0115] In this embodiment, the data processing device, in response to a configuration update request, obtains the update data dictionary associated with the configuration update request; obtains the update structure field and update rule field from the update data dictionary, configures update table statements according to the update structure field and update rule field; and updates the configuration of the target access file according to the update table statements to obtain the updated target access file. This improves the scalability of the data access process.

[0116] In order to better implement the data processing method in the embodiments of the present application, on the basis of the data processing method, the embodiments of the present application further provide a data processing device, as shown in Figure 5 Figure 5 The data processing device provided in the embodiments of the present application has the structure shown in the figure, and specifically, the data processing device 500 comprises:

[0117] The data acquisition module 501 is configured to acquire target service data associated with a data access request in response to the data access request;

[0118] The file configuration module 502 is configured to query a target data dictionary corresponding to the target service data, and configure a target access file of the target service data according to the target data dictionary;

[0119] The data detection module 503 is configured to call a preset scheduling task module to perform data detection on the target service data, obtain target verification data, access the target verification data to the target access file, and generate a data cache file;

[0120] The data loading module 504 is configured to acquire a target data table corresponding to the target verification data, and load the data cache file to the target data table.

[0121] In a possible implementation manner of the present embodiment, querying a target data dictionary corresponding to the target service data and configuring a target access file of the target service data according to the target data dictionary comprises:

[0122] Querying a preset data dictionary database to obtain a target data dictionary corresponding to the target service data;

[0123] Acquiring a data structure field and a storage rule field in the target data dictionary, and configuring a target table creation statement of the target service data according to the data structure field and the storage rule field;

[0124] Inputting the target table creation statement into a preset template file to configure a target access file of the target service data.

[0125] In a possible implementation manner of the present embodiment, calling a preset scheduling task module to perform data detection on the target service data, obtaining target verification data, accessing the target verification data to the target access file, and generating a data cache file comprise:

[0126] Acquiring the scheduling task module to obtain a data detection submodule and a detection log submodule in the scheduling task module;

[0127] ​The data detection submodule is called to perform data detection on the target service data to obtain target verification data, and the detection log submodule is called to output data detection logs;

[0128] The target verification data and the data detection logs are recorded in the target access file to obtain a data cache file.

[0129] In a possible implementation manner of the embodiment, the data detection submodule is called to perform data detection on the target service data to obtain target verification data, and the detection log submodule is called to output data detection logs, including:

[0130] The service detection identifier and the type detection identifier in the data detection submodule are obtained;

[0131] The service detection identifier and the service identifier of the target service data are compared, and the type detection identifier and the data type of the target service data are compared;

[0132] The target service data with the same service identifier and the same data type as the service detection identifier and the type detection identifier are set as target verification data;

[0133] The target verification data is recorded in the detection log submodule to obtain data detection logs.

[0134] In a possible implementation manner of the embodiment, a target data table corresponding to the target verification data is obtained, and the target verification data in the data cache file is loaded into the target data table, including:

[0135] The service type and the data type of the target verification data are obtained;

[0136] The service type and the data type are matched with a preset data table in a preset data cluster to obtain a target data table;

[0137] A preset synchronization module is called to transmit the data cache file into the target data cluster, and the target verification data in the data cache file is loaded into the target data table in the target data cluster according to a preset access configuration.

[0138] In a possible implementation manner of the embodiment, a preset scheduling task module is called to perform data detection on the target service data to obtain target verification data, and the target verification data is accessed to the target access file to generate a data cache file, including:

[0139] A preset release configuration file is obtained, and the target access file is processed according to the release configuration file to obtain a target release file;

[0140] The preset scheduling task module is called to perform data detection on the target service data, target verification data is obtained, and the target verification data is accessed to the target release file to generate a data cache file.

[0141] In a possible implementation of the embodiment, after the target data dictionary corresponding to the target service data is queried and the target access file of the target service data is configured according to the target data dictionary, the following steps are further included:

[0142] In response to a configuration update request, an update data dictionary associated with the configuration update request is obtained.

[0143] An update structure field and an update rule field in the update data dictionary are obtained, and an update table statement is configured according to the update structure field and the update rule field.

[0144] The target access file is updated and configured according to the update table statement to obtain an updated target access file.

[0145] In the embodiment, the data processing apparatus obtains target service data associated with a data access request in response to the data access request, queries a target data dictionary corresponding to the target service data, configures a target access file of the target service data according to the target data dictionary, calls a preset scheduling task module to perform data detection on the target service data to obtain target verification data, accesses the target verification data to the target access file to generate a data cache file, and obtains a target data table corresponding to the target verification data and loads the data cache file to the target data table. The target service data is automatically accessed through the data dictionary and the scheduling task module, the efficiency of batch data access is improved, and the error rate of access is reduced.

[0146] The embodiment of the present application further provides a data processing device, as shown in Figure 6 The embodiment of the present application further provides a data processing device, as shown in Figure 6 The embodiment of the present application further provides a data processing device, as shown in

[0147] The data processing device integrates any one of the data processing apparatuses provided in the embodiments of the present application, and the data processing device includes:

[0148] One or more processors;

[0149] A memory; and

[0150] One or more application programs, wherein the one or more application programs are stored in the memory and are configured to perform the steps of the data processing method in any one of the data processing method embodiments by the processor.

[0151] In particular, the data processing device can include a processor 601 having one or more processing cores, a memory 602 having one or more computer-readable storage media, a power supply 603, and an input unit 604, etc. Those skilled in the art can understand that Figure 6 The data processing device structure shown in the figure is not a limitation on the data processing device, and can include more or fewer components than shown, or combine certain components, or different component arrangements. Among them:

[0152] The processor 601 is the control center of the data processing device, which connects various parts of the entire data processing device through various interfaces and lines, and performs various functions and processes data of the data processing device by running or executing software programs and / or modules stored in the memory 602 and calling data stored in the memory 602, thereby overall monitoring the data processing device. Optionally, the processor 601 can include one or more processing cores; preferably, the processor 601 can integrate an application processor and a modem processor, wherein the application processor mainly processes the operating system, user interface and application program, etc., and the modem processor mainly processes wireless communication. It can be understood that the above-mentioned modem processor can also not be integrated into the processor 601.

[0153] The memory 602 can be used to store software programs and modules, and the processor 601 executes various functions and data processing by running the software programs and modules stored in the memory 602. The memory 602 can mainly include a program storage area and a data storage area, wherein the program storage area can store the operating system, at least one application program required by the function (such as sound playing function, image playing function, etc.), etc.; the data storage area can store data created according to the use of the data processing device, etc. In addition, the memory 602 can include a high-speed random access memory, and can also include a non-volatile memory, such as at least one magnetic disk storage device, a flash memory device, or other volatile solid-state memory device. Accordingly, the memory 602 can also include a memory controller to provide access for the processor 601 to the memory 602.

[0154] The data processing device also includes a power supply 603 for powering various components, and preferably the power supply 603 can be logically connected to the processor 601 through a power management system, so as to realize the functions of managing charging, discharging, and power consumption management, etc. through the power management system. The power supply 603 can also include one or more direct current or alternating current power supplies, a recharging system, a power supply fault detection circuit, a power supply converter or inverter, a power supply state indicator, etc. any component.

[0155] The data processing device can also include an input unit 604, which can be used to receive inputted digital or character information, and generate keyboard, mouse, joystick, optical or trackball signal inputs related to user settings and function controls.

[0156] Although not shown, the data processing device can also include a display unit, etc., which will not be described here. In particular in the present embodiment, the processor 601 in the data processing device will load the executable file corresponding to the process of one or more application programs into the memory 602 according to the following instructions, and run the application program stored in the memory 602 by the processor 601, thereby realizing various functions, such as the following:

[0157] In response to a data access request, obtaining target business data associated with the data access request;

[0158] Querying a target data dictionary corresponding to the target business data, and configuring a target access file of the target business data according to the target data dictionary;

[0159] Calling a preset scheduling task module to perform data detection on the target business data, obtaining target verification data, accessing the target verification data to the target access file, and generating a data cache file;

[0160] Obtaining a target data table corresponding to the target verification data, and loading the data cache file to the target data table.

[0161] To this end, the embodiment of the present application provides a computer readable storage medium, which can include a read-only memory (ROM), a random access memory (RAM), a magnetic disk or an optical disk, etc. A computer program is stored thereon, and the computer program is loaded by a processor to execute the steps in any of the data processing methods provided by the embodiment of the present application. For example, the computer program loaded by the processor can execute the following steps:

[0162] In response to a data access request, obtaining target business data associated with the data access request;

[0163] Querying a target data dictionary corresponding to the target business data, and configuring a target access file of the target business data according to the target data dictionary;

[0164] Calling a preset scheduling task module to perform data detection on the target business data, obtaining target verification data, accessing the target verification data to the target access file, and generating a data cache file;

[0165] Obtain a target data table corresponding to the target check data, and load the data cache file into the target data table.

[0166] In the above embodiments, the description of each embodiment has its own focus, and the parts not described in detail in a certain embodiment can be referred to the detailed description of other embodiments above, which will not be repeated here.

[0167] In the implementation, each unit or structure above can be implemented as an independent entity, or can be combined as the same or several entities, and the specific implementation of each unit or structure can be referred to the method embodiments above, which will not be repeated here.

[0168] The specific implementation of each operation above can be referred to the embodiments above, which will not be repeated here.

[0169] The data processing method provided by the embodiments of the present application is described in detail above, and the principles and implementation manners of the present application are described by applying specific embodiments; the above embodiment descriptions are only used to help understand the method and core idea of the present application; meanwhile, for those skilled in the art, the specific implementation manners and application ranges will be changed according to the idea of the present application, and the above description should not be understood as the limitation of the present application.

Claims

1. A data processing method, characterized in that, The data processing method includes: Respond to the data access request and obtain the target business data associated with the data access request; Query the target data dictionary corresponding to the target service data, and configure the target access file of the target service data according to the target data dictionary; The preset scheduling task module is invoked to perform data detection on the target business data, obtain target verification data, and then the target verification data is connected to the target access file to generate a data cache file. Obtain the target data table corresponding to the target verification data, and load the data cache file into the target data table; The process of querying the target data dictionary corresponding to the target service data and configuring the target access file for the target service data according to the target data dictionary includes: Query the preset data dictionary database to obtain the target data dictionary corresponding to the target business data; Obtain the data structure fields and storage rule fields from the target data dictionary, and configure the target table creation statement for the target business data based on the data structure fields and storage rule fields; Input the target table creation statement into a preset template file to configure the target access file for the target business data.

2. The data processing method as described in claim 1, characterized in that, The process of calling a preset scheduling task module to perform data detection on the target business data, obtaining target verification data, and then inserting the target verification data into the target access file to generate a data cache file includes: Obtain the scheduling task module, and then obtain the data detection submodule and the detection log submodule within the scheduling task module; The data detection submodule is invoked to perform data detection on the target business data to obtain target verification data, and the detection log submodule is invoked to output data detection log. The target verification data and the data detection log are entered into the target access file to obtain a data cache file.

3. The data processing method as described in claim 2, characterized in that, The process involves calling the data detection submodule to perform data detection on the target business data, obtaining target verification data, and calling the detection log submodule to output data detection logs, including: Obtain the business detection identifier and type detection identifier from the data detection submodule; The service detection identifier is compared with the service identifier of the target service data, and the type detection identifier is compared with the data type of the target service data; Set the target business data that has the same business identifier and business detection identifier and the same data type as the type detection identifier as the target verification data; The target verification data is entered into the detection log submodule to obtain the data detection log.

4. The data processing method as described in claim 1, characterized in that, The step of obtaining the target data table corresponding to the target verification data and loading the data cache file into the target data table includes: Obtain the business type and data type of the target verification data; The target data table is obtained by matching the business type and data type with a preset data table in a preset data cluster. The preset synchronization module is invoked to transmit the data cache file to the target data cluster, and the target verification data in the data cache file is loaded into the target data table in the target data cluster according to the preset access configuration.

5. The data processing method as described in claim 1, characterized in that, The process of calling a preset scheduling task module to perform data detection on the target business data, obtaining target verification data, and then inserting the target verification data into the target access file to generate a data cache file includes: Obtain a preset release configuration file, and perform release processing on the target access file according to the release configuration file to obtain the target release file; The preset scheduling task module is invoked to perform data detection on the target business data, obtain target verification data, and then the target verification data is connected to the target release file to generate a data cache file.

6. The data processing method according to any one of claims 1-5, characterized in that, After querying the target data dictionary corresponding to the target service data and configuring the target access file of the target service data according to the target data dictionary, the process further includes: In response to a configuration update request, obtain the update data dictionary associated with the configuration update request; Obtain the update structure field and update rule field from the update data dictionary, and configure the update table statement according to the update structure field and update rule field; The target access file is updated by updating the configuration according to the update table statement to obtain the updated target access file.

7. A data processing apparatus, characterized in that, The data processing device includes: The data acquisition module is configured to respond to a data access request and acquire the target business data associated with the data access request; The file configuration module is configured to query a preset data dictionary database to obtain the target data dictionary corresponding to the target business data; obtain the data structure fields and storage rule fields in the target data dictionary; configure the target table creation statement for the target business data according to the data structure fields and storage rule fields; input the target table creation statement into a preset template file to obtain the target access file for the target business data. The data detection module is configured to call a preset scheduling task module to perform data detection on the target business data, obtain target verification data, and connect the target verification data to the target access file to generate a data cache file; The data loading module is configured to obtain the target data table corresponding to the target verification data and load the data cache file into the target data table.

8. A data processing device, characterized in that, The data processing device includes: One or more processors; Memory; and One or more applications, wherein the one or more applications are stored in the memory and configured to be executed by the processor to implement the steps of the data processing method according to any one of claims 1 to 6.

9. A computer-readable storage medium, characterized in that, It stores a computer program, which is loaded by a processor to execute the steps of the data processing method according to any one of claims 1 to 6.

Images