Data processing method and device, computer equipment, readable storage medium and program product

Abstract

The invention relates to a data processing method and device, computer equipment, a readable storage medium and a program product, and is applied to the technical field of big data. The method comprises the following steps: acquiring data source information of a preset data source and historical data processing information of a data system; generating data interaction configuration information of the data system according to the data source information of the preset data source, and controlling the data system to perform data interaction configuration according to the data interaction configuration information; according to the historical data processing information, target data processing information of the data system is predicted, and the target data processing information is used for representing at least one of a data synchronization scheme and a data cleaning scheme; and controlling the data system and the preset data source to perform data interaction processing according to the target data processing information. By adopting the method, the data processing efficiency of the data system can be improved.

Description

Technical Field

[0001] The present application relates to the field of big data technology, and in particular to a data processing method, apparatus, computer equipment, computer-readable storage medium, and computer program product. Background Art

[0002] With the rapid development of information technology, especially the popularization and application of emerging technologies such as the Internet, the Internet of Things, and cloud computing, the amount of data generated by enterprises and organizations is growing at an exponential rate. These data are not only huge in quantity, but also diverse in type, including structured data (such as table data in databases), unstructured data (such as text files, pictures, videos), and semi-structured data (such as XML, JSON). Therefore, each type of data is usually stored in different data systems. Faced with such massive and complex data, how to effectively collect, store, interact with, manage, and analyze data through various data systems has become an urgent problem to be solved.

[0003] At present, the system functions of various data systems are different, so the data stored in each data system are isolated from each other. Therefore, there are data islands between the data systems, which makes each data system prone to problems such as information lag, duplication of work and low decision-making efficiency, resulting in low data processing efficiency of the data system. Summary of the invention

[0004] Based on this, it is necessary to provide a data processing method, apparatus, computer equipment, computer-readable storage medium and computer program product that can improve the data processing efficiency of a data system in response to the above technical problems.

[0005] In a first aspect, the present application provides a data processing method, comprising:

[0006] Obtain data source information of a preset data source and historical data processing information of a data system;

[0007] Generate data interaction configuration information of the data system according to the data source information of the preset data source, and control the data system to perform data interaction configuration according to the data interaction configuration information;

[0008] Predicting target data processing information of the data system based on the historical data processing information, wherein the target data processing information is used to characterize at least one of a data synchronization solution and a data cleaning solution;

[0009] According to the target data processing information, the data system is controlled to perform data interaction processing with the preset data source.

[0010] In a second aspect, the present application further provides a data processing device, comprising:

[0011] An acquisition module, used to acquire data source information of a preset data source and historical data processing information of a data system;

[0012] A generating module, used to generate data interaction configuration information of the data system according to the data source information of the preset data source, and control the data system to perform data interaction configuration according to the data interaction configuration information;

[0013] A prediction module, configured to predict target data processing information of the data system based on the historical data processing information, wherein the target data processing information is used to characterize at least one of a data synchronization solution and a data cleaning solution;

[0014] A control module is used to control the data system to perform data interaction processing with the preset data source according to the target data processing information.

[0015] In a third aspect, the present application further provides a computer device, including a memory and a processor, wherein the memory stores a computer program, and when the processor executes the computer program, the following steps are implemented:

[0016] Obtain data source information of a preset data source and historical data processing information of a data system;

[0017] Generate data interaction configuration information of the data system according to the data source information of the preset data source, and control the data system to perform data interaction configuration according to the data interaction configuration information;

[0018] Predicting target data processing information of the data system based on the historical data processing information, wherein the target data processing information is used to characterize at least one of a data synchronization solution and a data cleaning solution;

[0019] According to the target data processing information, the data system is controlled to perform data interaction processing with the preset data source.

[0020] In a fourth aspect, the present application further provides a computer-readable storage medium having a computer program stored thereon, wherein when the computer program is executed by a processor, the following steps are implemented:

[0021] Obtain data source information of a preset data source and historical data processing information of a data system;

[0022] Generate data interaction configuration information of the data system according to the data source information of the preset data source, and control the data system to perform data interaction configuration according to the data interaction configuration information;

[0023] Predicting target data processing information of the data system based on the historical data processing information, wherein the target data processing information is used to characterize at least one of a data synchronization solution and a data cleaning solution;

[0024] According to the target data processing information, the data system is controlled to perform data interaction processing with the preset data source.

[0025] In a fifth aspect, the present application further provides a computer program product, including a computer program, which implements the following steps when executed by a processor:

[0026] Obtain data source information of a preset data source and historical data processing information of a data system;

[0027] Generate data interaction configuration information of the data system according to the data source information of the preset data source, and control the data system to perform data interaction configuration according to the data interaction configuration information;

[0028] Predicting target data processing information of the data system based on the historical data processing information, wherein the target data processing information is used to characterize at least one of a data synchronization solution and a data cleaning solution;

[0029] According to the target data processing information, the data system is controlled to perform data interaction processing with the preset data source.

[0030] The above-mentioned data processing method, apparatus, computer equipment, computer-readable storage medium and computer program product obtain data source information of a preset data source and historical data processing information of a data system; generate data interaction configuration information of the data system according to the data source information of the preset data source, and control the data system to perform data interaction configuration according to the data interaction configuration information; predict target data processing information of the data system according to the historical data processing information, wherein the target data processing information is used to characterize at least one of a data synchronization scheme and a data cleaning scheme; and control the data system to perform data interaction processing with the preset data source according to the target data processing information. Based on the data source information of the preset data source, the data interaction configuration information of the data system is generated, and the data system is configured based on the data interaction configuration information to ensure that the data interaction configuration of the data system matches the preset data source, thereby ensuring the data interaction efficiency between the data system and the preset data source. The historical data processing information is used as the prediction basis for the target data processing information, so that the target data processing information is summarized from the previous data processing schemes of the data system, which improves the rationality of the target data processing information. Therefore, the process of controlling the data system to perform data interaction processing with the preset data source based on the target data processing information is smooth and time-saving. In summary, the data processing efficiency of the data system is improved. BRIEF DESCRIPTION OF THE DRAWINGS

[0031] In order to more clearly illustrate the technical solutions in the embodiments of the present application or related technologies, the drawings required for use in the embodiments of the present application or related technical descriptions will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present application. For ordinary technicians in this field, other related drawings can be obtained based on these drawings without paying creative work.

[0032] Figure 1 An application environment diagram of a data processing method in an embodiment;

[0033] Figure 2 is an application environment diagram of a data processing method in another embodiment;

[0034] Figure 3 is a flow chart of a data processing method in one embodiment;

[0035] Figure 4 A schematic diagram of a flow chart of a step of predicting target data processing information of a data system according to historical data processing information in an embodiment;

[0036] Figure 5 is a structural block diagram of a data processing device in one embodiment;

[0037] Figure 6 FIG. 4 is a diagram showing the internal structure of a computer device in one embodiment. DETAILED DESCRIPTION

[0038] In order to make the purpose, technical solution and advantages of the present application more clearly understood, the present application is further described in detail below in conjunction with the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are only used to explain the present application and are not used to limit the present application.

[0039] It should be noted that the information (for example, data source information, historical data processing information, system operation information, and user behavior information, etc.) and data (including but not limited to data used for analysis, stored data, displayed data, etc.) involved in this application are all information and data authorized by the user or fully authorized by all parties, and the acquisition, transmission, storage, use and processing of relevant data are in compliance with the relevant provisions of national laws and regulations. For content pushed to users (for example, data interaction configuration information and target data processing information, etc.), users can refuse or conveniently refuse content push, etc. In the embodiments of this application, certain software, components, models, and other existing solutions in the industry may be mentioned, and they should be considered as exemplary. Their purpose is only to illustrate the feasibility of the implementation of the technical solution of this application, but it does not mean that the applicant has or will necessarily use the solution.

[0040] The data processing method provided in the embodiment of the present application can be applied to Figure 1In the application environment shown. Among them, the data system 102, the preset data source 104 and the terminal 106 communicate with the server 108 through the network respectively. The data storage system can store the data that the server 108 needs to process. The data storage system can be integrated on the server 108, or it can be placed on the cloud or other network servers. The server 108 obtains the data source information sent by the preset data source 104, and obtains the historical data processing information sent by the data system 102; according to the data source information sent by the preset data source 104, the data interaction configuration information of the data system 102 is generated, and according to the data interaction configuration information, the data system 102 is controlled to perform data interaction configuration; according to the historical data processing information, the target data processing information of the data system 102 is predicted, wherein the target data processing information is used to characterize the data synchronization scheme, and at least one of the data cleaning schemes; according to the target data processing information, the data system 102 is controlled to perform data interaction processing with the preset data source 104. The server 108 may also push at least one of the data interaction configuration information and the target data processing information to the terminal 106, wherein the terminal 106 may be, but is not limited to, various personal computers, laptops, smart phones, tablet computers, IoT devices, and portable wearable devices, and the IoT devices may be smart speakers, smart TVs, smart air conditioners, smart car devices, projection devices, etc. Portable wearable devices may be smart watches, smart bracelets, head-mounted devices, etc. The head-mounted devices may be virtual reality (VR) devices, augmented reality (AR) devices, smart glasses, etc. The server 108 may be an independent physical server, a server cluster or distributed system consisting of multiple physical servers, or a cloud server that provides cloud computing services.

[0041] The data processing method provided in the embodiment of the present application can also be applied to Figure 2 In the application environment shown. The data system 102 includes a communication unit 21, a storage unit 22 and a control unit 23. The communication unit 21 is used to communicate with the preset data source 104 and obtain the data source information sent by the preset data source 104. The storage unit 22 is used to store the data source information sent by the preset data source 104 and the historical data processing information of the data system 102. The control unit 23 is used to generate data interaction configuration information of the data system 102 according to the data source information of the preset data source 104 stored in the storage unit 22, and control the data system 102 to perform data interaction configuration according to the data interaction configuration information; predict the target data processing information of the data system according to the historical data processing information, wherein the target data processing information is used to characterize at least one of the data synchronization scheme and the data cleaning scheme; and control the data system 102 to perform data interaction processing with the preset data source 104 according to the target data processing information.

[0042] Furthermore, the data system 102 may also include a display unit, and the display unit is used to display at least one of the data interaction configuration information and the target data processing information.

[0043] In an exemplary embodiment, Figure 3 As shown, a data processing method is provided, which is applied to Figure 1 The server 108 in the example is used as an example to illustrate the method, which includes the following steps 202 to 208. Among them:

[0044] Step 202: Acquire data source information of a preset data source and historical data processing information of a data system.

[0045] Among them, the preset data source in step 202 can be a preset database, for example, MySQL (an open source relational database management system) database, PostgreSQL (an object-relational database management system of free software with very complete features) database and MongoDB (a database based on distributed file storage), etc., or it can be other systems that communicate with the data system, such as other business systems or other data systems. The preset data source can be single or multiple. The data source information is used to characterize the data scale of the data stored in the data source, and the data source information includes at least one of the data quantity and the data size. The historical data processing information is used to characterize the data processing status recorded after the data system performs data processing, and the historical data processing information includes at least one of the historical data processing scheme and the historical system operation information after the data system adopts the historical data processing scheme.

[0046] Step 204: Generate data interaction configuration information of the data system according to the data source information of the preset data source, and control the data system to perform data interaction configuration according to the data interaction configuration information.

[0047] Among them, the data interaction configuration information in step 204 is used to characterize the interaction configuration status of the data system. The data interaction configuration information includes performance configuration information and basic interaction information. The basic interaction information includes at least one of the data source type, data source name, host name of the data source, data source IP address and data source port number; the performance configuration information includes at least one of the connection pool information, API (Application Programming Interface) information, interaction timeout limit and interaction retry limit. The connection pool information is used to characterize the connection pool size, and the API information is used to characterize the API flow limit size. The interaction timeout limit is the maximum interaction time allowed when the data system interacts with the preset data source, and the interaction retry limit is the maximum number of times the data system can retry the interaction when the data system interacts with the preset data source.

[0048] As one embodiment, data interaction configuration information of a data system is generated according to data source information of a preset data source, including: evaluating the data scale of the preset data source according to the data source information of the preset data source to obtain data scale information; generating data interaction configuration information of the data system according to the data scale information, wherein the more data in the data source information, the larger the data scale represented by the evaluated data scale information; the larger the data size in the data source information, the larger the data scale represented by the evaluated data scale information; the larger the data scale represented by the evaluated data scale information, the larger the connection pool size represented by the connection pool information in the data interaction configuration information; the larger the data scale represented by the evaluated data scale information, the smaller the API current limiting size represented by the API information in the data interaction configuration information.

[0049] As another embodiment, data interaction configuration information of a data system is generated according to data source information of a preset data source, including: obtaining user behavior information corresponding to the data system; and generating data interaction configuration information of the data system according to the user behavior information and the data source information of the preset data source.

[0050] Furthermore, data interaction configuration information of the data system is generated based on the user behavior information and the data source information of the preset data source, including: based on the user behavior information, screening the target data source that meets the preset access conditions from the various data sources that interact with the data system; generating the data interaction configuration information of the data system based on the data source information of the target data source.

[0051] As an embodiment, based on user behavior information, a target data source that meets a preset access condition is screened from various data sources that interact with a data system, including: based on the user behavior information, identifying the user's access status to each data source that interacts with the data system, and obtaining the user access frequency corresponding to each data source; and selecting a data source whose user access frequency meets a preset frequency condition from each data source as the target data source that meets the preset access condition, wherein the preset frequency condition can be greater than a preset frequency threshold, or can be a maximum value among all user access frequencies.

[0052] As another embodiment, based on user behavior information, target data sources that meet preset access conditions are screened from various data sources that interact with the data system, including: based on the user behavior information, identifying the user's access status to each data source that interacts with the data system, and obtaining the user access frequency corresponding to each data source; obtaining a preset execution project, and obtaining the association between each data source and the preset execution project; generating a screening priority corresponding to each data source based on the user access frequency and association corresponding to each data source; based on the screening priority of each data source, screening the target data source that meets the preset access condition from various data sources that interact with the data system, wherein the preset execution project can be a business project or an access project, etc., which is not limited here.

[0053] The user behavior information includes at least one of the number of times the user accesses each data source, the access duration, and the access time.

[0054] In this way, considering that the data system is used to execute projects, it is necessary to interact with the data sources associated with the preset execution projects, and the user access frequency can characterize the data source with which the user expects to interact. Therefore, the above two factors are used together as the basis for generating the screening priority of each data source. In addition, the screening priority is the basis for screening the target data source, so the screening accuracy of the target data source is improved.

[0055] As an embodiment, after screening the target data source that meets the preset access conditions from various data sources that interact with the data system based on user behavior information, it also includes: screening the data source information of the target data source from the data source information of the preset data source, identifying at least one of the data source type, data source name, data source host name, data source IP address and data source port number from the data source information of the target data source; or, sending a basic interaction information query request to the target data source, and receiving basic interaction information sent by the target data source in response to the basic interaction information query request.

[0056] As an embodiment, controlling the data system to perform data interaction configuration according to the data interaction configuration information includes: sending the data interaction configuration information to the data system, and controlling the data system to perform data interaction configuration according to the data interaction configuration information.

[0057] Step 206: predict target data processing information of the data system based on the historical data processing information, wherein the target data processing information is used to characterize at least one of the data synchronization solution and the data cleaning solution.

[0058] The target data processing information in step 206 includes at least one of a target data synchronization solution and a target data cleaning solution.

[0059] Exemplarily, step 206 includes: identifying the historical data processing status of the data system based on the historical data processing information, obtaining the historical data processing scheme of the data system and the data processing effect corresponding to the historical data processing scheme, and predicting the target data processing information of the data system based on the historical data processing scheme of the data system and the data processing effect corresponding to the historical data processing scheme.

[0060] Step 208: Control the data system to perform data interaction processing with the preset data source according to the target data processing information.

[0061] Exemplarily, step 208 includes: controlling the data system to synchronize the data of the target data source in the preset data source to the data system according to the target data synchronization scheme in the target data processing information; and performing data cleaning on the synchronized data in the data system according to the target data cleaning scheme in the target data processing information.

[0062] In the above data processing method, by obtaining the data source information of the preset data source and the historical data processing information of the data system; generating the data interaction configuration information of the data system according to the data source information of the preset data source, and controlling the data system to perform data interaction configuration according to the data interaction configuration information; predicting the target data processing information of the data system according to the historical data processing information, wherein the target data processing information is used to characterize at least one of the data synchronization scheme and the data cleaning scheme; controlling the data system to perform data interaction processing with the preset data source according to the target data processing information, generating the data interaction configuration information of the data system based on the data source information of the preset data source, and configuring the data system based on the data interaction configuration information, ensuring that the data interaction configuration of the data system matches the preset data source, thereby ensuring the data interaction efficiency between the data system and the preset data source, and using the historical data processing information as the prediction basis of the target data processing information, so that the target data processing information is summarized from the previous data processing scheme of the data system, thereby improving the rationality of the target data processing information, and therefore, making the process of controlling the data system to perform data interaction processing with the preset data source based on the target data processing information smooth and time-saving, and in summary, improving the data processing efficiency of the data system.

[0063] In an exemplary embodiment, Figure 4 As shown, a method for accurately generating target data processing information is provided, the historical data processing information includes historical data synchronization information, the target data processing information includes target data synchronization information and target data cleaning information; according to the historical data processing information, the target data processing information of the prediction data system includes steps 302 to 308. Among them:

[0064] Step 302: Acquire current system operation information of the data system.

[0065] Among them, the current system operation information in step 302 is used to characterize the current operation status of the data system. The current system operation information includes at least one of the current server load information, the current network information and the current storage information. The current server load information is used to characterize the current server load occupancy status of the data system, the current network information is used to characterize the current network smoothness or congestion of the data system, and the current storage information is used to characterize the current storage occupancy status of the data system.

[0066] Step 304: Determine the historical synchronization scheme adopted by the data system and the historical synchronization effect corresponding to the historical synchronization scheme according to the historical data synchronization information.

[0067] Among them, the historical data synchronization information in step 304 includes at least one of historical synchronization time information, historical synchronization type and historical system operation information. The historical synchronization time information includes the execution time of each data synchronization; the historical synchronization type includes at least one of incremental synchronization type and full synchronization type; the historical system operation information includes the operating status of the data system at each time when the data system performed data synchronization in the past, and the historical system operation information includes at least one of historical server load information, historical network information and historical storage information.

[0068] As an embodiment, step 304 includes: determining the historical synchronization scheme adopted by the data system based on the historical synchronization time information and the historical synchronization type in the historical data synchronization information; and determining the historical synchronization effect corresponding to the historical synchronization scheme based on the historical system operation information.

[0069] Among them, the more the current server load of the data system represented by the historical server load information in the historical system operation information is occupied, the worse the historical synchronization effect of the corresponding historical synchronization scheme is; the higher the current network congestion of the data system represented by the current network information in the historical system operation information is, the worse the historical synchronization effect of the corresponding historical synchronization scheme is; the more the current storage of the data system represented by the current storage information in the historical system operation information is occupied, the worse the historical synchronization effect of the corresponding historical synchronization scheme is.

[0070] In this way, the synchronization effect of the historical synchronization plan is quantified through the historical system operation information. Therefore, when the target data synchronization information is predicted based on the historical synchronization plan, the synchronization effect can be used as an auxiliary judgment basis to improve the prediction accuracy of the target data synchronization information.

[0071] Step 306: predict target data synchronization information of the data system based on historical synchronization schemes, historical synchronization effects, and current system operation information.

[0072] The target data processing information in step 306 includes a target data processing scheme, and the target data processing scheme includes at least one of target synchronization time information and a target synchronization type for each data waiting to be synchronized.

[0073] As an embodiment, step 306 includes: based on the historical synchronization effect, selecting a target historical synchronization scheme from the historical synchronization schemes; based on the current system operation information, predicting the synchronization effect when the data system is synchronized with the target historical synchronization scheme to obtain a predicted synchronization effect; and constructing a target data processing scheme for the data system based on the predicted synchronization effect and the target historical synchronization scheme.

[0074] Furthermore, based on the historical synchronization effect, a target historical synchronization scheme is screened from the historical synchronization schemes, including: screening the target historical synchronization scheme from the historical synchronization schemes whose historical synchronization effect meets a preset effect condition, wherein the preset effect condition may be greater than a preset effect threshold, or may be the maximum value among the historical synchronization effects of all historical synchronization schemes.

[0075] As an embodiment, based on the current system operation information, the synchronization effect of the data system when synchronized with the target historical synchronization scheme is predicted to obtain the predicted synchronization effect, including: based on the current system operation information, simulating the process of synchronizing the data system with the target historical synchronization scheme to obtain the predicted system operation information when the data system is synchronized with the target historical synchronization scheme; based on the predicted system operation information, evaluating the synchronization effect of the data system when synchronized with the target historical synchronization scheme to obtain the predicted synchronization effect, wherein the predicted system operation information includes at least one of predicted server load information, predicted network information and predicted storage information.

[0076] Optionally, based on the predicted system operation information, the synchronization effect when the data system is synchronized with the target historical synchronization scheme is evaluated to obtain a specific implementation method of the predicted synchronization effect. The specific implementation steps of determining the historical synchronization effect corresponding to the historical synchronization scheme based on the historical system operation information can be referred to above, and will not be repeated here.

[0077] As an embodiment, a target data processing scheme of a data system is constructed according to a predicted synchronization effect and a target historical synchronization scheme, including: if there is a scheme in the target historical synchronization scheme whose predicted synchronization effect satisfies a preset effect condition, the scheme is determined as the target data processing scheme of the data system; if there is no scheme in the target historical synchronization scheme whose predicted synchronization effect satisfies the preset effect condition, target synchronization time information whose predicted system operation information satisfies the preset operation condition is selected from the synchronization time information of the target historical synchronization scheme, and the target synchronization time information in the target historical synchronization scheme is updated according to the predicted system operation information to obtain the target data processing scheme of the data system, wherein the preset operation condition can be that the degree of operation abnormality of the characterized data system is greater than The preset operation abnormality degree threshold may also be the operation abnormality degree of the represented data system, which is the maximum value among the operation abnormality degrees of the data system represented under all synchronization time information. Specifically, the more the predicted server load occupancy of the data system represented by the predicted server load information in the predicted system operation information, the higher the operation abnormality degree of the data system represented by the predicted system operation information; the higher the predicted network congestion degree of the data system represented by the predicted network information in the predicted system operation information, the higher the operation abnormality degree of the data system represented by the predicted system operation information; the more the predicted storage occupancy of the data system represented by the predicted storage information in the predicted system operation information, the higher the operation abnormality degree of the data system represented by the predicted system operation information.

[0078] Step 308: predicting target data cleaning information of the data system based on the target data synchronization information.

[0079] As an embodiment, step 308 includes: evaluating the quality of synchronization data corresponding to the data system according to the target data synchronization information to obtain synchronization data quality information; and predicting target data cleaning information of the data system according to the synchronization data quality information.

[0080] Furthermore, based on the synchronization data quality information, target data cleaning information of the data system is predicted, including: based on the synchronization data quality information, screening out first data whose synchronization data quality does not meet preset quality conditions from the synchronization data corresponding to the data system, wherein the preset quality condition is greater than a synchronization data quality threshold; based on the data importance corresponding to the first data, screening out second data from the first data; based on the synchronization time corresponding to the second data, predicting a data cleaning time corresponding to the second data; and based on the data cleaning time corresponding to the second data, determining the target data cleaning information of the data system.

[0081] Among them, the synchronization data quality information includes at least one of the synchronization data repetition degree, the synchronization data vacancy degree, the synchronization data format error degree and the synchronization data error degree. The synchronization data repetition degree includes the proportion of the repeated part in the synchronization data to the whole part of the synchronization data, the synchronization data vacancy degree includes the proportion of the vacancy part in the synchronization data to the whole part of the synchronization data, the synchronization data format error degree includes the proportion of the part of the synchronization data with format errors to the whole part of the synchronization data, and the synchronization data error degree includes the proportion of the part of the synchronization data with data content errors to the whole part of the synchronization data; specifically, the higher the synchronization data repetition degree, the worse the synchronization data quality; the higher the synchronization data vacancy degree, the worse the synchronization data quality; the higher the synchronization data format error degree, the worse the synchronization data quality; the higher the synchronization data error degree, the worse the synchronization data quality.

[0082] Optionally, based on the data importance corresponding to the first data, before filtering the second data from the first data, the method also includes: obtaining the usage time when the first data will be used, and the criticality of the first data for data usage, and generating the data importance corresponding to the first data based on the usage time and the criticality of the first data, wherein the earlier the usage time, the higher the importance of the generated data, and the higher the criticality, the higher the importance of the generated data.

[0083] In this way, considering that the earlier the data is to be used, the more it is necessary to ensure the accuracy of the data, and the higher the criticality of the data to the use of the data, the greater the impact of inaccurate data on the use of the data, so it is more necessary to ensure the accuracy of the data. Therefore, considering the two factors of data usage time and criticality, the accuracy of the assessment of the data importance of the first data is improved.

[0084] As an embodiment, filtering the second data from the first data according to the data importance corresponding to the first data includes: filtering the second data whose corresponding data importance is lower than a preset importance threshold from the first data.

[0085] Optionally, after screening out first data whose synchronization data quality does not meet preset quality conditions from the synchronization data corresponding to the data system based on the synchronization data quality information, the method also includes: screening out target synchronization type and target synchronization time information corresponding to the first data from the target data synchronization scheme, and when the target synchronization type corresponding to the first data includes a full synchronization type, updating the target synchronization type corresponding to the first data to an incremental synchronization type, and / or adding synchronization time information corresponding to the first data, and updating the target synchronization time information corresponding to the first data based on the newly added synchronization time information corresponding to the first data.

[0086] In this way, for the first data whose synchronization data quality does not meet the preset quality conditions, that is, the data quality obtained by synchronization is poor, by updating the synchronization type to an incremental synchronization type, and adding synchronization time information, the first data can be synchronized more frequently, thereby improving the data quality of the first data synchronized to the data system.

[0087] As an embodiment, predicting a data cleaning time corresponding to the second data according to a synchronization time corresponding to the second data includes: determining a time after the synchronization time corresponding to the second data as the data cleaning time corresponding to the second data.

[0088] As an embodiment, target data cleaning information of a data system is determined according to a data cleaning time corresponding to the second data, including: obtaining data problem information corresponding to the second data; generating a data cleaning method according to the data problem information, and constructing target data cleaning information of the data system using the data cleaning method and data cleaning time corresponding to the second data.

[0089] Optionally, the method also includes: the target data processing scheme (including but not limited to the above-mentioned target data synchronization scheme and target data cleaning scheme) includes at least one processing task, and the processing order between each processing task is determined by the task processing priority corresponding to each processing task. Specifically, the higher the task processing priority, the earlier the processing order. The task processing priority corresponding to each processing task is determined by the data priority of the processing data contained in the processing task. The data priority is determined by the time when the data is to be used and the degree of dependence between the data. The earlier the data is to be used, the higher the priority of the generated data; the earlier the data with a degree of dependence on the data greater than a preset dependence threshold is to be used, the higher the priority of the generated data.

[0090] In this way, considering that under the data synchronization plan and data cleaning plan at a large granularity, there are also tasks and processed data at a small granularity, after determining the execution time of the data synchronization plan and the data cleaning plan through the above steps, the processing order is also determined accordingly for each task and each processed data at a small granularity, thereby improving the accuracy of data processing.

[0091] Optionally, after controlling the data system to perform data interaction processing with a preset data source according to the target data processing information, the method further includes: obtaining a data processing effect (including synchronization effect and data quality information) corresponding to the data system, and updating the target data processing information according to the data processing effect.

[0092] In this way, the target data processing information can be updated according to the data processing effect to achieve optimization of the target data processing information.

[0093] Optionally, the method also includes: determining the security level of the data processing project of the data system, setting the project confidentiality information of the data processing project of the data system according to the security level of the data processing project of the data system, and controlling the data system configuration project confidentiality information, wherein the data processing project includes at least one of a data synchronization project, a data cleaning project, a data access project and a data scheduling project.

[0094] Furthermore, determining the security level of the data processing project of the data system includes: obtaining the data source richness and data interaction complexity of the data processing project of the data system, and determining the security level of the data processing project of the data system according to the data source richness and data interaction complexity of the data processing project, wherein the higher the data source richness, the higher the security level of the data processing project of the determined data system; and the higher the data interaction complexity, the higher the security level of the data processing project of the determined data system.

[0095] Among them, the security levels of data processing projects are ranked from high to low as follows: data scheduling projects, data synchronization projects, data access projects and data cleaning projects.

[0096] Among them, the project confidential information includes at least one of the project encryption information, project authentication information and project operation authority information. The higher the security level of the data processing project, the more information content the project confidential information contains and the more items the information contains.

[0097] In this way, the security of the data system during data processing is guaranteed.

[0098] In this embodiment, the current system operation information of the data system is obtained; according to the historical data synchronization information, the historical synchronization scheme adopted by the data system and the historical synchronization effect corresponding to the historical synchronization scheme are determined; according to the historical synchronization scheme, the historical synchronization effect and the current system operation information, the target data synchronization information of the data system is predicted; according to the target data synchronization information, the target data cleaning information of the data system is predicted, and the historical data synchronization information is used as a basis to determine the historical synchronization scheme adopted by the data system and the historical synchronization effect corresponding to the historical synchronization scheme, and the historical synchronization scheme, the historical synchronization effect and the current system operation information are used together as the basis for predicting the target data synchronization information, considering that the historical synchronization scheme has a certain reference significance, and the historical synchronization effect can be used as a basis for generating the degree of reference to the historical synchronization scheme, and the current system operation information can be used to evaluate the degree of adaptation between the historical synchronization scheme and the current operation status, considering the influencing factors of the above three situations, the prediction accuracy of the target data synchronization information is improved, and the target data synchronization information is the prediction basis of the target data cleaning information, so the prediction accuracy of the target data cleaning synchronization information is also improved.

[0099] As a detailed embodiment, data source information of a preset data source and historical data processing information of a data system are obtained; user behavior information corresponding to the data system is obtained; based on the user behavior information, a target data source that meets preset access conditions is screened from various data sources that interact with the data system; data interaction configuration information of the data system is generated based on the data source information of the target data source; data interaction configuration information of the data system is generated based on the data source information of the preset data source, and the data system is controlled to perform data interaction configuration according to the data interaction configuration information.

[0100] Furthermore, the current system operation information of the data system is obtained; based on the historical data synchronization information, the historical synchronization scheme adopted by the data system and the historical synchronization effect corresponding to the historical synchronization scheme are determined; based on the historical synchronization effect, the target historical synchronization scheme is screened from the historical synchronization schemes; based on the current system operation information, the synchronization effect when the data system is synchronized with the target historical synchronization scheme is predicted to obtain the predicted synchronization effect; based on the predicted synchronization effect and the target historical synchronization scheme, a target data processing scheme of the data system is constructed; based on the historical synchronization scheme, the historical synchronization effect and the current system operation information, the target data synchronization information of the data system is predicted; based on the target data synchronization information, the quality of the synchronization data corresponding to the data system is evaluated to obtain synchronization data quality information; based on the synchronization data quality information, the first data whose synchronization data quality does not meet the preset quality conditions is screened from the synchronization data corresponding to the data system; based on the data importance corresponding to the first data, the second data is screened from the first data; based on the synchronization time corresponding to the second data, the data cleaning time corresponding to the second data is predicted; based on the data cleaning time corresponding to the second data, the target data cleaning information of the data system is determined.

[0101] In this way, by obtaining the data source information of the preset data source and the historical data processing information of the data system; generating the data interaction configuration information of the data system according to the data source information of the preset data source, and controlling the data system to perform data interaction configuration according to the data interaction configuration information; predicting the target data processing information of the data system according to the historical data processing information, wherein the target data processing information is used to characterize at least one of the data synchronization scheme and the data cleaning scheme; controlling the data system to perform data interaction processing with the preset data source according to the target data processing information, generating the data interaction configuration information of the data system based on the data source information of the preset data source, and configuring the data system based on the data interaction configuration information, ensuring that the data interaction configuration of the data system matches the preset data source, thereby ensuring the data interaction efficiency between the data system and the preset data source, and using the historical data processing information as the prediction basis of the target data processing information, so that the target data processing information is summarized from the previous data processing scheme of the data system, thereby improving the rationality of the target data processing information, and therefore, making the process of controlling the data system to perform data interaction processing with the preset data source based on the target data processing information smooth and time-saving, and in summary, improving the data processing efficiency of the data system.

[0102] Furthermore, by acquiring the current system operation information of the data system; determining the historical synchronization scheme adopted by the data system and the historical synchronization effect corresponding to the historical synchronization scheme based on the historical data synchronization information; predicting the target data synchronization information of the data system based on the historical synchronization scheme, the historical synchronization effect and the current system operation information; predicting the target data cleaning information of the data system based on the target data synchronization information, based on the historical data synchronization information, determining the historical synchronization scheme adopted by the data system and the historical synchronization effect corresponding to the historical synchronization scheme, and using the historical synchronization scheme, the historical synchronization effect and the current system operation information as the basis for predicting the target data synchronization information, considering that the historical synchronization scheme has a certain reference significance, and the historical synchronization effect can be used as a basis for generating the degree of reference to the historical synchronization scheme, and the current system operation information can be used to evaluate the degree of adaptation between the historical synchronization scheme and the current operation status, considering the influencing factors of the above three situations, the prediction accuracy of the target data synchronization information is improved, and the target data synchronization information is the prediction basis for the target data cleaning information, so the prediction accuracy of the target data cleaning synchronization information is also improved.

[0103] It should be understood that, although the steps in the flowcharts involved in the above embodiments are displayed in sequence according to the indication of the arrows, these steps are not necessarily executed in sequence according to the order indicated by the arrows. Unless there is a clear explanation in this article, the execution of these steps is not strictly limited in order, and these steps can be executed in other orders. Moreover, at least a part of the steps in the flowcharts involved in the above embodiments may include multiple steps or multiple stages, and these steps or stages are not necessarily executed at the same time, but can be executed at different times, and the execution order of these steps or stages is not necessarily carried out in sequence, but can be executed in turn or alternately with other steps or at least a part of the steps or stages in other steps.

[0104] Based on the same inventive concept, the embodiment of the present application also provides a data processing device for implementing the data processing method involved above. The implementation scheme for solving the problem provided by the device is similar to the implementation scheme recorded in the above method, so the specific limitations in the one or more data processing device embodiments provided below can refer to the limitations on the data processing method above, and will not be repeated here.

[0105] In an exemplary embodiment, Figure 5 As shown, a data processing device 500 is provided, comprising: an acquisition module 502, a generation module 504, a prediction module 506 and a control module 508, wherein:

[0106] An acquisition module 502 is used to acquire data source information of a preset data source and historical data processing information of a data system;

[0107] A generating module 504 is used to generate data interaction configuration information of the data system according to the data source information of the preset data source, and control the data system to perform data interaction configuration according to the data interaction configuration information;

[0108] A prediction module 506, configured to predict target data processing information of the data system based on the historical data processing information, wherein the target data processing information is used to characterize at least one of a data synchronization solution and a data cleaning solution;

[0109] The control module 508 is used to control the data system to perform data interaction processing with the preset data source according to the target data processing information.

[0110] In one of the embodiments, historical data processing information includes historical data synchronization information, and target data processing information includes target data synchronization information and target data cleaning information; the prediction module 506 is also used to obtain current system operation information of the data system; determine the historical synchronization scheme adopted by the data system and the historical synchronization effect corresponding to the historical synchronization scheme based on the historical data synchronization information; predict the target data synchronization information of the data system based on the historical synchronization scheme, the historical synchronization effect and the current system operation information; and predict the target data cleaning information of the data system based on the target data synchronization information.

[0111] In one of the embodiments, the target data processing information includes a target data processing plan; the prediction module 506 is also used to screen the target historical synchronization plan from the historical synchronization plans based on the historical synchronization effect; based on the current system operation information, predict the synchronization effect when the data system is synchronized with the target historical synchronization plan to obtain the predicted synchronization effect; based on the predicted synchronization effect and the target historical synchronization plan, construct the target data processing plan of the data system.

[0112] In one embodiment, the prediction module 506 is further used to evaluate the quality of the synchronization data corresponding to the data system according to the target data synchronization information to obtain the synchronization data quality information; and predict the target data cleaning information of the data system according to the synchronization data quality information.

[0113] In one of the embodiments, the prediction module 506 is also used to filter out first data whose synchronization data quality does not meet preset quality conditions from the synchronization data corresponding to the data system according to the synchronization data quality information; filter out second data from the first data according to the data importance corresponding to the first data; predict the data cleaning time corresponding to the second data according to the synchronization time corresponding to the second data; and determine the target data cleaning information of the data system according to the data cleaning time corresponding to the second data.

[0114] In one embodiment, the generation module 504 is also used to obtain user behavior information corresponding to the data system; based on the user behavior information, filter the target data source that meets the preset access conditions from various data sources that interact with the data system; and generate data interaction configuration information of the data system based on the data source information of the target data source.

[0115] Each module in the above data processing device can be implemented in whole or in part by software, hardware or a combination thereof. Each module can be embedded in or independent of a processor in a computer device in the form of hardware, or can be stored in a memory in a computer device in the form of software, so that the processor can call and execute operations corresponding to each module.

[0116] In an exemplary embodiment, a computer device is provided. The computer device may be a terminal, and its internal structure diagram may be as shown in FIG. Figure 6 As shown. The computer device includes a processor, a memory, an input / output interface, a communication interface, a display unit and an input device. The processor, the memory and the input / output interface are connected through a system bus, and the communication interface, the display unit and the input device are connected to the system bus through the input / output interface. The processor of the computer device is used to provide computing and control capabilities. The memory of the computer device includes a non-volatile storage medium and an internal memory. The non-volatile storage medium stores an operating system and a computer program. The internal memory provides an environment for the operation of the operating system and the computer program in the non-volatile storage medium. The input / output interface of the computer device is used to exchange information between the processor and an external device. The communication interface of the computer device is used to communicate with an external terminal in a wired or wireless manner, and the wireless manner can be implemented through WIFI, a mobile cellular network, near field communication (NFC) or other technologies. When the computer program is executed by the processor, a data processing method is implemented. The display unit of the computer device is used to form a visually visible picture, which can be a display screen, a projection device or a virtual reality imaging device. The display screen can be a liquid crystal display screen or an electronic ink display screen, and the input device of the computer device can be a touch layer covering the display screen, or a button, trackball or touchpad set on the computer device shell, or an external keyboard, touchpad or mouse.

[0117] Those skilled in the art will understand that Figure 6 The structure shown in the figure is only a block diagram of a part of the structure related to the solution of the present application, and does not constitute a limitation on the computer device to which the solution of the present application is applied. The specific computer device may include more or fewer components than those shown in the figure, or combine certain components, or have a different arrangement of components.

[0118] In an exemplary embodiment, a computer device is provided, including a memory and a processor, wherein a computer program is stored in the memory, and the processor implements the steps in the above-mentioned method embodiments when executing the computer program.

[0119] In one embodiment, a computer-readable storage medium is provided, on which a computer program is stored. When the computer program is executed by a processor, the steps in the above-mentioned method embodiments are implemented.

[0120] In one embodiment, a computer program product is provided, including a computer program, which implements the steps in the above method embodiments when executed by a processor.

[0121] Those skilled in the art can understand that all or part of the processes in the above-mentioned embodiments can be completed by instructing the relevant hardware through a computer program, and the computer program can be stored in a non-volatile computer-readable storage medium. When the computer program is executed, it can include the processes of the embodiments of the above-mentioned methods. Among them, any reference to the memory, database or other medium used in the embodiments provided in this application can include at least one of non-volatile memory and volatile memory. Non-volatile memory can include read-only memory (ROM), magnetic tape, floppy disk, flash memory, optical memory, high-density embedded non-volatile memory, resistive random access memory (ReRAM), magnetic random access memory (MRAM), ferroelectric random access memory (FRAM), phase change memory (PCM), graphene memory, etc. Volatile memory can include random access memory (RAM) or external cache memory, etc. As an illustration and not limitation, RAM can be in various forms, such as static random access memory (SRAM) or dynamic random access memory (DRAM). The database involved in each embodiment provided in this application may include at least one of a relational database and a non-relational database. Non-relational databases may include distributed databases based on blockchains, etc., but are not limited to this. The processor involved in each embodiment provided in this application may be a general-purpose processor, a central processing unit, a graphics processor, a digital signal processor, a programmable logic device, a data processing logic device based on quantum computing, an artificial intelligence (AI) processor, etc., but are not limited to this.

[0122] The technical features of the above embodiments may be combined arbitrarily. To make the description concise, not all possible combinations of the technical features in the above embodiments are described. However, as long as there is no contradiction in the combination of these technical features, they should be considered to be within the scope of this application.

[0123] The above embodiments only express several implementation methods of the present application, and the descriptions thereof are relatively specific and detailed, but they cannot be understood as limiting the scope of the present application. It should be pointed out that, for a person of ordinary skill in the art, several variations and improvements can be made without departing from the concept of the present application, and these all belong to the protection scope of the present application. Therefore, the protection scope of the present application shall be subject to the attached claims.

Claims

1. A data processing method, characterized in that: The method comprises: Obtain data source information of a preset data source and historical data processing information of a data system; Generate data interaction configuration information of the data system according to the data source information of the preset data source, and control the data system to perform data interaction configuration according to the data interaction configuration information; Predicting target data processing information of the data system based on the historical data processing information, wherein the target data processing information is used to characterize at least one of a data synchronization solution and a data cleaning solution; According to the target data processing information, the data system is controlled to perform data interaction processing with the preset data source.

2. The method according to claim 1, characterized in that The historical data processing information includes historical data synchronization information, and the target data processing information includes target data synchronization information and target data cleaning information; The step of predicting target data processing information of the data system according to the historical data processing information includes: Acquiring current system operation information of the data system; Determining, according to the historical data synchronization information, a historical synchronization scheme adopted by the data system and a historical synchronization effect corresponding to the historical synchronization scheme; Predicting target data synchronization information of the data system according to the historical synchronization scheme, the historical synchronization effect and the current system operation information; According to the target data synchronization information, target data cleaning information of the data system is predicted.

3. The method according to claim 2, characterized in that The target data processing information includes a target data processing scheme; the target data synchronization information of the data system is predicted according to the historical synchronization scheme, the historical synchronization effect and the current system operation information, including: According to the historical synchronization effect, selecting a target historical synchronization scheme from the historical synchronization schemes; According to the current system operation information, predicting the synchronization effect of the data system when synchronizing with the target historical synchronization scheme to obtain a predicted synchronization effect; A target data processing scheme for the data system is constructed based on the predicted synchronization effect and the target historical synchronization scheme.

4. The method according to claim 2, characterized in that: The predicting target data cleaning information of the data system according to the target data synchronization information includes: According to the target data synchronization information, the quality of the synchronization data corresponding to the data system is evaluated to obtain synchronization data quality information; According to the synchronized data quality information, target data cleaning information of the data system is predicted.

5. The method according to claim 4, characterized in that The predicting target data cleaning information of the data system according to the synchronized data quality information includes: According to the synchronization data quality information, first data whose synchronization data quality does not meet a preset quality condition is screened from the synchronization data corresponding to the data system; Filtering second data from the first data according to the data importance corresponding to the first data; Predicting a data cleaning time corresponding to the second data according to a synchronization time corresponding to the second data; Target data cleaning information of the data system is determined according to the data cleaning time corresponding to the second data.

6. The method according to any one of claims 1 to 5, characterized in that The step of generating data interaction configuration information of the data system according to data source information of a data source interacting with the data system comprises: Obtaining user behavior information corresponding to the data system; According to the user behavior information, a target data source that meets a preset access condition is selected from various data sources interacting with the data system; The data interaction configuration information of the data system is generated according to the data source information of the target data source.

7. A data processing device, characterized in that: The device comprises: An acquisition module, used to acquire data source information of a preset data source and historical data processing information of a data system; A generating module, used to generate data interaction configuration information of the data system according to the data source information of the preset data source, and control the data system to perform data interaction configuration according to the data interaction configuration information; A prediction module, configured to predict target data processing information of the data system based on the historical data processing information, wherein the target data processing information is used to characterize at least one of a data synchronization solution and a data cleaning solution; A control module is used to control the data system to perform data interaction processing with the preset data source according to the target data processing information.

8. A computer device comprising a memory and a processor, wherein the memory stores a computer program, wherein: When the processor executes the computer program, the steps of the method according to any one of claims 1 to 6 are implemented.

9. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, the steps of the method according to any one of claims 1 to 6 are implemented.

10. A computer program product, comprising a computer program, characterized in that When the computer program is executed by a processor, the steps of the method according to any one of claims 1 to 6 are implemented.

Images