Data processing method, device, electronic device and readable storage medium

Abstract

The embodiment of the present invention provides a data processing method, device, electronic device and readable storage medium, wherein the method comprises: upon receiving a first input from a user, configuring a data source set according to the first input, wherein the data source set comprises a plurality of data sources; upon receiving a second input from a user, configuring a processing strategy according to the second input and the library table of each data source in the data source set; and obtaining an original data set from each data source for processing according to the processing strategy to complete data aggregation and storage. The embodiment of the present invention solves the problem that the traditional data integration approach is not convenient for data processing and analysis of complex heterogeneous data sources.

Description

Technical Field

[0001] The present invention relates to the technical field of data batch processing, and in particular to a Java-based data processing method, device, electronic device and readable storage medium. Background Art

[0002] In the process of enterprise informatization construction, due to the phased and technical nature of the construction of various business systems and the implementation of data management systems, as well as other economic and human factors, enterprises have accumulated a large amount of business data using different storage methods, which constitute the heterogeneous data sources of the enterprise.

[0003] The traditional idea of ​​data integration is to establish an enterprise data center, extract data from various systems, centralize it, and then provide unified data services. However, with the rapid increase in data volume, especially the increase in unstructured data, traditional enterprise data centers have problems such as high cost, difficulty in modeling, difficulty in data migration, high data integration cost, and difficulty in integrating unstructured data, making it inconvenient to process and analyze data from heterogeneous data sources.

[0004] When faced with complex data processing tasks, the traditional approach requires the use of multi-threading to process data in order to improve data processing efficiency. When clustering and distinguishing data from different data sources, different businesses, and different types, and processing different businesses with different needs, not only does it require adding a large amount of code, it also increases the system's computing pressure, which seriously slows down the progress of the task. Summary of the invention

[0005] In view of the above problems, the present invention is proposed to provide a Java-based data processing method, device, electronic device and readable storage medium that solve the above problems or at least partially solve the above problems.

[0006] In a first aspect, an embodiment of the present invention provides a Java-based data processing method, the method comprising:

[0007] In case of receiving a first input from a user, configuring a data source set according to the first input, the data source set including a plurality of data sources;

[0008] When receiving a second input from a user, configuring a processing strategy according to the second input and a library table of each of the data sources in the data source set;

[0009] According to the processing strategy, the original data set is obtained from each of the data sources for processing to complete data aggregation and storage.

[0010] Optionally, in the method, configuring a processing strategy according to the second input and a library table of each data source in the data source set includes:

[0011] According to the second input and the library table of each data source in the data source set, select a data service type for each data source, configure a data processing method and a data processing level configuration;

[0012] Wherein, the data service type is selected from cluster analysis processing and data update service;

[0013] The data processing method is selected from one of batch processing, time-sharing processing, real-time processing and distribution processing;

[0014] The data processing level configuration includes level information and classification conditions of each data processing level.

[0015] Optionally, in the method, obtaining original data sets from each of the data sources for processing according to the processing strategy includes:

[0016] According to the data processing level configuration, each of the data sources is processed to obtain original data sets and data objects with tags at each level, wherein the tags are used to distinguish data service types and data processing methods;

[0017] Generate a corresponding sub-thread according to each data object;

[0018] Through the thread blocking tool and each of the sub-threads, data aggregation and data storage are performed on the original data set according to the data service type and data processing method corresponding to each data object.

[0019] Optionally, in the method, the data processing level configuration is a splitting level configuration, a merging level configuration or a conversion and cleaning level configuration;

[0020] The split class level configuration is a custom class split level configuration or a system cluster analysis configuration;

[0021] The classification conditions in the system clustering analysis configuration include the number of classifications for clustering analysis using the k_means algorithm.

[0022] Optionally, in the method, the data structure of the original data set is LIST <LIST <pojo>>, the data structure of the data object POJO is MAP<STRING,OBJECT> .

[0023] In a second aspect, an embodiment of the present invention provides a Java-based data processing device, the device comprising:

[0024] A first configuration module is used to configure a data source set according to a first input received from a user, wherein the data source set includes a plurality of data sources;

[0025] A second configuration module is used to configure a processing strategy according to the second input and the library table of each data source in the data source set when receiving a second input from the user;

[0026] The processing module is used to obtain the original data set from each of the data sources for processing according to the processing strategy to complete data aggregation and storage.

[0027] Optionally, in the device, the second configuration module is specifically used to select a data service type for each data source, configure a data processing method and a data processing level configuration according to the second input and a library table of each data source in the data source set;

[0028] Wherein, the data service type is selected from cluster analysis processing and data update service;

[0029] The data processing method is selected from one of batch processing, time-sharing processing, real-time processing and distribution processing;

[0030] The data processing level configuration includes level information and classification conditions of each data processing level.

[0031] Optionally, in the device, the processing module includes:

[0032] A first processing unit, configured to process each of the data sources according to the data processing level configuration, to obtain original data sets and data objects with tags at each level, wherein the tags are used to distinguish data service types and data processing methods;

[0033] A sub-thread unit, used for generating a corresponding sub-thread according to each data object;

[0034] The second processing unit is used to aggregate the original data set by using the thread blocking tool and each of the sub-threads according to the data service type and data processing method corresponding to each data object, so as to obtain the original data set and the marked data object of the next level;

[0035] A third processing unit is used for continuing to execute the step of generating a corresponding sub-thread according to each data object when it is determined according to the data processing level configuration that the original data set of the next level needs to be further processed;

[0036] The fourth processing unit is used to store the original data set of the next level when it is determined that the data set of the next level does not need to be further processed according to the data processing level configuration.

[0037] In a third aspect, an embodiment of the present application provides an electronic device, which includes a processor, a memory, and a program or instruction stored in the memory and executable on the processor, wherein the program or instruction, when executed by the processor, implements the steps of the method described in the first aspect.

[0038] In a fourth aspect, an embodiment of the present application provides a readable storage medium, on which a program or instruction is stored, and when the program or instruction is executed by a processor, the steps of the method described in the first aspect are implemented.

[0039] In a fifth aspect, an embodiment of the present application provides a chip, comprising a processor and a communication interface, wherein the communication interface is coupled to the processor, and the processor is used to run a program or instruction to implement the method described in the first aspect.

[0040] Compared with the prior art, the embodiments of the present invention have the following advantages:

[0041] In the embodiment of the present invention, when a user's first input is received, a data source set is configured according to the first input, and the data source set includes multiple data sources; when a user's second input is received, a processing strategy is configured according to the second input and the library table of each data source in the data source set; according to the processing strategy, the original data set is obtained from each data source for processing to complete data aggregation and storage. Because the user only needs to simply configure the data source set and prepare the processing strategy, batch processing of data from heterogeneous multiple data sources can be achieved, and services can be added to meet the needs of data cleaning, data processing, and data governance by adding new processing strategies. It has high expansibility and practicality, and can automatically create tables for data storage when the library table of the data source is set, without limiting the type of database, so that data from different regions, different systems, and different services can be used and processed more conveniently, improving the efficiency of platform data query and data utilization, and reducing data management costs.

[0042] The above description is only an overview of the technical solution of the present invention. In order to more clearly understand the technical means of the present invention, it can be implemented according to the contents of the specification. In order to make the above and other purposes, features and advantages of the present invention more obvious and easy to understand, the specific implementation methods of the present invention are listed below. BRIEF DESCRIPTION OF THE DRAWINGS

[0043] Various other advantages and benefits will become apparent to those of ordinary skill in the art by reading the detailed description of the preferred embodiments below. The accompanying drawings are only for the purpose of illustrating the preferred embodiments and are not to be considered as limiting the present invention. Moreover, the same reference symbols are used throughout the accompanying drawings to represent the same components. In the accompanying drawings:

[0044] Figure 1 A flowchart showing a step of a Java-based data processing method embodiment according to the present invention is shown;

[0045] Figure 2 A schematic diagram of the creation process of a data source in an embodiment of the present invention is shown;

[0046] Figure 3 A schematic diagram of the creation process of a library table in an embodiment of the present invention is shown;

[0047] Figure 4 A schematic diagram of a policy configuration process in an embodiment of the present invention is shown;

[0048] Figure 5 A schematic diagram of an implementation principle according to an embodiment of the present invention is shown;

[0049] Figure 6 A structural block diagram of an embodiment of a Java-based data processing device according to the present invention is shown. DETAILED DESCRIPTION

[0050] The exemplary embodiments of the present disclosure will be described in more detail below with reference to the accompanying drawings. Although the exemplary embodiments of the present disclosure are shown in the accompanying drawings, it should be understood that the present disclosure can be implemented in various forms and should not be limited by the embodiments set forth herein. On the contrary, these embodiments are provided to enable a more thorough understanding of the present disclosure and to fully convey the scope of the present disclosure to those skilled in the art.

[0051] Reference Figure 1 , shows a step flow chart of an embodiment of a Java-based data processing method according to the present invention, which may specifically include steps 101 to 103.

[0052] The embodiments of the present invention are applied to a terminal, which may be a mobile terminal or a non-mobile terminal, such as a computer, a portable computer or other office equipment.

[0053] Step 101: upon receiving a first input from a user, configure a data source set according to the first input, wherein the data source set includes a plurality of data sources.

[0054] Among them, the first input is the input for the user to select the data source, that is, the system provides an interface for the user to select the data source to be connected, and the user can dynamically select the data source to be connected through this interface. Therefore, when the user's first input is received, it means that the corresponding data source needs to be connected to the system, and thus the data source sets corresponding to the first input are configured together to form the above-mentioned data source set.

[0055] The data structures, libraries and tables of different data sources are different and can be grouped. The embodiment of the present invention does not limit the data source, and data from different regions, different systems and different services can be configured into the data source set, but the system needs to have multiple database drivers built in to ensure access to different data sources.

[0056] Specifically, users dynamically configure data sources and then connect them to the system. Each data source is grouped, classified, configured, and managed. During the configuration process, the data source access status is detected and the system is connected after verification. For the specific process, please refer to Figure 2 , which shows a schematic diagram of the process of creating a data source.

[0057] like Figure 2 As shown, first customize or select grouping according to the type of data source to be introduced, then configure the data source, then match the data source, and finally process the data source.

[0058] When configuring a data source, there are two ways to access the data source. One way is to access remotely through IP, host name, port number, driver, data source account and password. The system will verify whether the above information can be connected, confirm that the target can be found, and after the remote test passes, the data source information will be stored in the database; the other way is to manually create a data source, that is, the current server data source. The current server data source can be created or imported by the system and stored in the database.

[0059] The above data source matching process means that the system will count the data sources that have been stored in the database. When a new data source enters, the system will verify whether the data source already exists. If it does, the existing data source information will be displayed. Among them, the data source access operation can be modified, replaced or canceled.

[0060] The above data source processing process refers to the system defaulting to improving the data source information after the above operations are completed, such as recording the operator, operator IP, operation time, generating a backup plan, generating a data source access memorandum, generating a data source abnormality notification plan and other additional processing services. Among them, for generating a backup plan, users can customize and modify the backup cycle.

[0061] Step 102: When receiving a second input from the user, configure a processing strategy according to the second input and the library table of each data source in the data source set.

[0062] Among them, the second input is the operation of the user to configure the processing strategy for each data source, that is, the system provides an interface for the user to configure the processing strategy for each data source, and the user can configure the processing strategy for each data source through this interface. Therefore, when the second input of the user is received, it means that the processing strategy needs to be configured for the corresponding data source; and when the processing strategy is configured for the data source, it is necessary to obtain the data structure of each data source, so it is necessary to introduce the library table of the data source, and then the corresponding processing strategy can be configured according to the second input and the library table of each data source. The above processing strategy stipulates the processing method for the data in each data source.

[0063] Among them, users can use templates to import the library tables of data sources in the system, and after using the template to import the required fields, they can import and verify them, and then import the dynamic library tables after passing the verification, and finally perform authorization processing to import the library tables of the corresponding data sources. In addition, users can also choose to import the library tables of data sources by manually adding them.

[0064] The creation process of the library table is as follows Figure 3 As shown. Among them, the library table refers to the table corresponding to the data source, which contains the creation information of the data source storage mode; when using the template to introduce the library table, the template is the data storage structure file that needs to be introduced. The template file can support txt, db, dbf, csv, html, xml, xls and xlsx, wk1, wq1, json, mdb or accdb, odbc and other formats; select the template file to import on the system page, the system will parse the template file, and display the columns included on the page. You can manually filter the fields that need to be stored in the library. After the field screening is completed, the form will be checked again to see if there are duplicate columns, special character columns, etc., and the table will be stored after passing the verification. Then, the table needs to be authorized. Authorization is the read and write operation granted to the user to ensure data security. After the authorization is passed, the library table is introduced.

[0065] Step 103: According to the processing strategy, original data sets are obtained from each of the data sources for processing to complete data aggregation and storage.

[0066] Among them, because the above processing strategy stipulates the processing method for data in each data source, the original data set can be screened and obtained from each data source according to the processing strategy, and then the original data set can be classified, aggregated and stored in the warehouse to complete the data processing.

[0067] In the embodiment of the present invention, because the user only needs to simply configure the data source set and prepare the processing strategy, batch processing of data from heterogeneous multiple data sources can be achieved, and services can be added to meet the needs of data cleaning, data processing, and data governance by adding new processing strategies. It has high scalability and practicality, and can automatically create tables for data storage when the library table of the data source is set. There is no restriction on the type of database, so that data from different regions, different systems, and different services can be used and processed more conveniently, thereby improving the efficiency of platform data query and data utilization, and reducing data management costs.

[0068] Optionally, in one implementation, the above step 102 specifically includes step 201 .

[0069] Step 201: According to the second input and the library table of each data source in the data source set, select the data service type for each data source, configure the data processing method and the data processing level configuration;

[0070] Among them, the data service type is selected from cluster analysis processing and data update service; the data processing method is selected from batch processing, time-sharing processing, real-time processing and distribution processing; the data processing level configuration includes level information and classification conditions of each data processing level.

[0071] In this implementation, by configuring the service type, data processing method and data processing level configuration of each data source of the service executed this time, dynamic configuration of the strategy is achieved, making the entire service more flexible, and reserving a secondary development interface for the service type to cope with business expansion, with high scalability.

[0072] Among them, the service type can be selected from cluster analysis processing and data update service, that is, the processing strategy for this execution can be configured by adding a new strategy, or the existing processing strategy can be selected to adaptively update the processing strategy for this execution. In the embodiment of the present invention, customized service type interfaces such as data processing and data cleaning are reserved, and the service type can be selected by oneself, so that different types of data and heterogeneous multi-data sources can dynamically configure strategies for data processing.

[0073] Among them, in terms of data processing methods, you can choose any one of batch processing, time-sharing processing, real-time processing, and distributed processing.

[0074] Among them, data processing level configuration refers to the specific configuration of dividing the data processing of the data source into multiple levels for classification and aggregation, wherein each level is marked with level information and corresponding classification conditions, and data processing is performed from low level to high level in accordance with the classification conditions of each level.

[0075] Optionally, in one implementation, the data processing level configuration is a split class level configuration, a merge class level configuration, or a transform and cleanse class level configuration.

[0076] Split-class hierarchical configuration is to split one into multiple classification condition hierarchical configurations.

[0077] Optionally, in a specific implementation, the split class hierarchy configuration is a custom class split hierarchy configuration or a system clustering analysis configuration.

[0078] The custom class split level configuration is a manual classification split level configuration, while the system clustering analysis configuration is a system automatic classification configuration.

[0079] The custom class split hierarchical configuration requires adding hierarchical information. In the first level, the specified data source specifies the specified library, the specified table, and the specified field contains data information with the search condition g1. The second level will continue to set the search condition g2 based on the obtained data to retrieve the required data. The third level and so on will be deduced by analogy until the upper limit of the level is reached, which can be ten levels.

[0080] Among them, g1 and g2 represent the classification conditions to be classified. For example, level one searches for information containing A in the name field of a specified logistics company in a table, and level two continues to search for data containing knives in the express delivery of company A based on the data obtained at level one.

[0081] The classification conditions in the system cluster analysis configuration include the number of categories for the k-means algorithm to perform cluster analysis. When using the k-means algorithm for cluster analysis, you only need to set the number of categories you need, and there is no need to set the hierarchy. After operating on large amounts of data, the k-means algorithm will make clustering more and more accurate.

[0082] The merged hierarchical configuration is an all-in-one classification condition hierarchical configuration. Its goal is to select multiple data sources or multiple databases and tables for data merging operations. Tables and fields can be created dynamically. For example, the first level sets the search condition g1, and the second level can continue to set the search condition g2 based on the search results of the first level to retrieve the required data. The third level and so on, until the upper limit of the level is reached, which can be ten levels.

[0083] The conversion and cleaning class hierarchical configuration is the one-to-one classification condition hierarchical configuration between the data source and the target data source. You can choose data update and data retrieval. Among them, data retrieval can be set to configure the data processing hierarchical configuration according to manual classification, and data update includes configurations such as replaceable, fixed modification, specified modification, blanking, and appending the content of specified fields. According to the conversion and cleaning class hierarchical configuration, after the data is acquired, the data content is specified, such as replacement, fixed modification, specified modification, blanking, and appending the content of specified fields for processing, thereby realizing one-to-one data retrieval and data processing, updating, and storage.

[0084] The policy configuration process is as follows: Figure 4 As shown, first select the service type and processing method to be configured, then determine the specific data source for which the policy needs to be configured, and then set processing parameters for each data processing level of each data source, so that a tag string can be generated for each data source to mark the data service type and data processing method of data objects at each level.

[0085] Optionally, in one implementation, the above step 103 includes steps 301 to 305 .

[0086] Step 301: Process each of the data sources according to the data processing level configuration to obtain original data sets and data objects with tags at each level, where the tags are used to distinguish data service types and data processing methods.

[0087] Among them, because the processing strategy includes the data service type, data processing method and data processing level configuration for each data source, and the data processing level configuration stipulates the specific configuration of each data processing level being divided into multiple levels for classification and aggregation, therefore, at each data processing level, the original data of the data source can be processed according to the corresponding data processing level configuration to obtain the original data set of the corresponding level after the policy hierarchical processing, as well as the data objects corresponding to each original data set, and the object has a mark for distinguishing the data service type and the data processing method.

[0088] Among them, each original data set generates several data objects through several data processing tasks, that is, a single original data set can generate multiple data objects.

[0089] Optionally, the data structure of the original data set is LIST <LIST <pojo>>, the data structure of the data object is MAP<STRING,OBJECT> .

[0090] Step 302: Generate a corresponding child thread according to each data object.

[0091] A corresponding sub-thread is generated for each data object to process the corresponding original data set.

[0092] Step 303: Through the thread blocking tool and each of the sub-threads, the original data set is aggregated according to the data service type and data processing method corresponding to each data object to obtain the next level of original data set and marked data objects.

[0093] Among them, the thread is blocked by a preset blocking tool, and then each sub-thread executes each data processing task of the current data processing level respectively until all data processing tasks of the current data processing level are completed. The preset blocking tool releases the thread and enters the next data processing level until all data processing tasks of the original data set are completed. The above processing process avoids the problems of frequent thread creation and frequent opening and closing of database resources for single processing, thereby realizing dynamic data aggregation and storage of heterogeneous multi-data sources.

[0094] Step 304: When it is determined according to the data processing level configuration that the original data set of the next level needs to be further processed, continue to execute the step of generating a corresponding sub-thread according to each data object.

[0095] Step 305: When it is determined according to the data processing level configuration that the original data set of the next level does not need to be further processed, the original data set of the next level is stored.

[0096] In steps 304 and 305, because the data processing level configuration specifies the specific configuration of each data processing level being divided into multiple levels for classification and aggregation, at each data processing level, each original data set can be analyzed according to the corresponding data processing level configuration to determine whether it needs to continue to be processed; if it needs to continue to be processed, a sub-thread is generated for each data object in the original data set, and data aggregation is continued for the original data set according to the data service type and data processing method corresponding to the data object; and when it is determined that the original data set of the current level no longer needs to be aggregated, the original data set of this level is stored in the database.

[0097] See also Figure 5 , shows a diagram of the execution principle of the Java-based data processing method provided in an embodiment of the present invention.

[0098] Figure 5 As shown, in step 501, the data source set configuration is implemented by first selecting the data source to be accessed, and then proceeding to step 502;

[0099] In step 502, dynamic policy configuration is performed according to the data processing requirements of each data source;

[0100] In step 503, the original data from each data source is processed according to the data processing level configuration in the processing strategy to obtain the original data set LIST <LIST <pojo>>, and insert a tag for distinguishing the data service type and the data processing method, and then enter step 504;

[0101] In step 504, according to each LIST <pojo>Generate a corresponding child thread, and then go to step 505;

[0102] In step 505, the original data sets of each data source and their corresponding data identifiers are obtained to perform different processing according to the data identifiers;

[0103] In step 506, for different original data sets, corresponding service types and processing methods are configured according to their corresponding identifiers;

[0104] In step 507, the original data set LIST is processed by the thread blocking tool and each thread. <LIST <pojo>>Perform data processing and data storage, and then release the thread to the thread pool;

[0105] In step 508, the system automatically obtains the server configuration to set a preset number of core threads; wherein, in order to ensure the reuse of the server's computing power, if the server is CPU-intensive, the number of core threads is set to be server core tree + 1; and if the server is IO-intensive, the number of core threads is set to be CPU core / (1-blocking coefficient), and the blocking coefficient is 0.8 to 0.9;

[0106] In step 509, the system periodically scans and releases the core threads, and retains a preset number of core threads;

[0107] In step 510 , the system continues to determine whether there is an idle thread, and if there is no idle thread, a new thread or sub-thread is created; wherein, because a thread needs to be called during the execution of steps 503 and 504 , the system also needs to execute step 510 .

[0108] In summary, the Java-based data processing method provided in the embodiment of the present invention first dynamically configures heterogeneous source data, obtains all data source configurations, and then processes the source data set, and performs tasks such as data pre-clustering, differentiation, and storage according to the strategy.

[0109] Reference Figure 6 , shows a structural block diagram of an embodiment of a Java-based data processing device according to the present invention, the Java-based data processing device 60 comprises:

[0110] A first configuration module 61 is configured to configure a data source set according to a first input received from a user, wherein the data source set includes a plurality of data sources;

[0111] A second configuration module 62 is used to configure a processing strategy according to the second input and the library table of each data source in the data source set when receiving a second input from the user;

[0112] The processing module 63 is used to obtain the original data set from each of the data sources for processing according to the processing strategy to complete data aggregation and storage.

[0113] According to a Java-based data processing device of the present invention, since users only need to simply configure the data source set and formulate the processing strategy, batch processing of data from heterogeneous multiple data sources can be realized, and services can be added to meet the needs of data cleaning, data processing, and data governance by means of new processing strategies. It has high expansibility and practicality, and can automatically create tables for data storage when the library tables of the data sources are set, without limiting the type of database, so that data from different regions, different systems, and different services can be used and processed more conveniently, thereby improving the efficiency of platform data query and data utilization, and reducing data management costs.

[0114] Optionally, in the device, the second configuration module 62 is specifically used to select a data service type for each data source, configure a data processing method and a data processing level configuration according to the second input and the library table of each data source in the data source set;

[0115] Wherein, the data service type is selected from cluster analysis processing and data update service;

[0116] The data processing method is selected from one of batch processing, time-sharing processing, real-time processing and distribution processing;

[0117] The data processing level configuration includes level information and classification conditions of each data processing level.

[0118] Optionally, in the device, the processing module 63 includes:

[0119] A first processing unit, configured to process each of the data sources according to the data processing level configuration, to obtain original data sets and data objects with tags at each level, wherein the tags are used to distinguish data service types and data processing methods;

[0120] A sub-thread unit, used for generating a corresponding sub-thread according to each data object;

[0121] The second processing unit is used to aggregate the original data set by using the thread blocking tool and each of the sub-threads according to the data service type and data processing method corresponding to each data object, so as to obtain the original data set and the marked data object of the next level;

[0122] A third processing unit is used for continuing to execute the step of generating a corresponding sub-thread according to each data object when it is determined according to the data processing level configuration that the original data set of the next level needs to be further processed;

[0123] The fourth processing unit is used to store the original data set of the next level when it is determined according to the data processing level configuration that the original data set of the next level does not need to be further processed.

[0124] As for the device embodiment, since it is basically similar to the method embodiment, the description is relatively simple, and the relevant parts can be referred to the partial description of the method embodiment.

[0125] Optionally, an embodiment of the present application also provides an electronic device, including a processor, a memory, and a program or instruction stored in the memory and executable on the processor. When the program or instruction is executed by the processor, each process of the above-mentioned Java-based data processing method embodiment is implemented, and the same technical effect can be achieved. To avoid repetition, it will not be repeated here.

[0126] It should be noted that the electronic devices in the embodiments of the present application include the mobile electronic devices and non-mobile electronic devices mentioned above.

[0127] An embodiment of the present application also provides a readable storage medium, on which a program or instruction is stored. When the program or instruction is executed by a processor, each process of the above-mentioned Java-based data processing method embodiment is implemented, and the same technical effect can be achieved. To avoid repetition, it will not be repeated here.

[0128] The processor is a processor in the electronic device described in the above embodiment. The readable storage medium includes a computer readable storage medium, such as a computer read-only memory (ROM), a random access memory (RAM), a magnetic disk or an optical disk.

[0129] An embodiment of the present application further provides a chip, which includes a processor and a communication interface, wherein the communication interface is coupled to the processor, and the processor is used to run programs or instructions to implement the various processes of the above-mentioned Java-based data processing method embodiment, and can achieve the same technical effect. To avoid repetition, it will not be repeated here.

[0130] It should be understood that the chip mentioned in the embodiments of the present application can also be called a system-level chip, a system chip, a chip system or a system-on-chip chip, etc.

[0131] The algorithm and display provided herein are not inherently related to any particular computer, virtual system or other device. Various general purpose systems can also be used together based on this teaching. According to the above description, it is obvious that the structure required for constructing such systems. In addition, the present invention is not directed to any specific programming language either. It should be understood that various programming languages ​​can be utilized to realize the content of the present invention described herein, and the above description of specific languages ​​is to disclose the best mode of the present invention.

[0132] In the description provided herein, a large number of specific details are described. However, it is understood that embodiments of the present invention can be practiced without these specific details. In some instances, well-known methods, structures and techniques are not shown in detail so as not to obscure the understanding of this description.

[0133] Similarly, it should be understood that in order to streamline the present disclosure and aid in understanding one or more of the various inventive aspects, in the above description of exemplary embodiments of the present invention, various features of the present invention are sometimes grouped together into a single embodiment, figure, or description thereof. However, this disclosed method should not be interpreted as reflecting the intention that the claimed invention requires more features than those explicitly recited in each claim. More specifically, as reflected in the claims below, inventive aspects lie in less than all the features of the individual embodiments disclosed above. Therefore, the claims that follow the specific embodiment are hereby expressly incorporated into the specific embodiment, with each claim itself serving as a separate embodiment of the present invention.

[0134] Those skilled in the art will appreciate that the modules in the devices in the embodiments may be adaptively changed and arranged in one or more devices different from the embodiments. The modules or units or components in the embodiments may be combined into one module or unit or component, and in addition they may be divided into a plurality of submodules or subunits or subcomponents. Except that at least some of such features and / or processes or units are mutually exclusive, all features disclosed in this specification (including the accompanying claims, abstracts and drawings) and all processes or units of any method or device disclosed in this manner may be combined in any combination. Unless otherwise expressly stated, each feature disclosed in this specification (including the accompanying claims, abstracts and drawings) may be replaced by an alternative feature providing the same, equivalent or similar purpose.

[0135] In addition, those skilled in the art will appreciate that, although some embodiments described herein include certain features included in other embodiments but not other features, the combination of features of different embodiments is meant to be within the scope of the present invention and form different embodiments. For example, in the claims below, any one of the claimed embodiments may be used in any combination.

[0136] The various component embodiments of the present invention may be implemented in hardware, or in software modules running on one or more processors, or in a combination thereof. It will be appreciated by those skilled in the art that a microprocessor or digital signal processor (DSP) may be used in practice to implement some or all of the functions of some or all of the components in the file download device according to an embodiment of the present invention. The present invention may also be implemented as a device or apparatus program (e.g., a computer program and a computer program product) for executing part or all of the methods described herein. Such a program implementing the present invention may be stored on a computer-readable medium, or may have the form of one or more signals. Such a signal may be downloaded from an Internet website, or provided on a carrier signal, or provided in any other form.

[0137] It should be noted that the above embodiments illustrate the present invention rather than limit it, and that those skilled in the art may devise alternative embodiments without departing from the scope of the appended claims. In the claims, any reference symbol between brackets shall not be construed as a limitation on the claims. The word "comprising" does not exclude the presence of elements or steps not listed in the claims. The word "one" or "an" preceding an element does not exclude the presence of a plurality of such elements. The present invention may be implemented by means of hardware comprising a number of different elements and by means of a suitably programmed computer. In a unit claim enumerating a number of devices, several of these devices may be embodied by the same hardware item. The use of the words first, second, and third, etc., does not indicate any order. These words may be interpreted as names.< / pojo> < / pojo> < / pojo> < / pojo> < / pojo>

Claims

1. A Java-based data processing method, characterized in that: The method comprises: In case of receiving a first input from a user, configuring a data source set according to the first input, the data source set including a plurality of data sources; When receiving a second input from a user, configuring a processing strategy according to the second input and a library table of each of the data sources in the data source set; According to the processing strategy, original data sets are obtained from each of the data sources for processing to complete data aggregation and storage; The configuring of the processing strategy according to the second input and the library table of each data source in the data source set includes: According to the second input and the library table of each data source in the data source set, select a data service type for each data source, configure a data processing method and a data processing level configuration; Wherein, the data service type is selected from cluster analysis processing and data update service; The data processing method is selected from one of batch processing, time-sharing processing, real-time processing and distribution processing; The data processing level configuration includes level information and classification conditions of each data processing level.

2. The data processing method according to claim 1, characterized in that: According to the processing strategy, original data sets are obtained from each of the data sources for processing, including: According to the data processing level configuration, each of the data sources is processed to obtain an original data set and a data object with a tag, wherein the tag is used to distinguish a data service type and a data processing method; Generate a corresponding sub-thread according to each data object; Through the thread blocking tool and each of the sub-threads, the original data set is aggregated according to the data service type and data processing method corresponding to each data object to obtain the next level of original data set and marked data objects; When it is determined according to the data processing level configuration that the original data set of the next level needs to be further processed, the step of generating a corresponding sub-thread according to each data object is continued; When it is determined according to the data processing level configuration that the original data set of the next level does not need to be further processed, the original data set of the next level is stored.

3. The data processing method according to claim 1, characterized in that: The data processing level configuration is a splitting level configuration, a merging level configuration, or a conversion and cleaning level configuration; The split class level configuration is a custom class split level configuration or a system cluster analysis configuration; The classification conditions in the system clustering analysis configuration include the number of classifications for clustering analysis using the k_means algorithm.

4. The data processing method according to claim 2, characterized in that: The data structure of the original data set is LIST <LIST <pojo> >, the data structure of the data object POJO is MAP<STRING,OBJECT> .< / pojo> 5. A Java-based data processing device, characterized in that: The device comprises: A first configuration module is used to configure a data source set according to a first input received from a user, wherein the data source set includes a plurality of data sources; A second configuration module is used to configure a processing strategy according to the second input and the library table of each data source in the data source set when receiving a second input from the user; A processing module, used to obtain the original data set from each of the data sources for processing according to the processing strategy to complete data aggregation and storage; The second configuration module is specifically used to select a data service type for each data source, configure a data processing method and a data processing level configuration according to the second input and the library table of each data source in the data source set; Wherein, the data service type is selected from cluster analysis processing and data update service; The data processing method is selected from one of batch processing, time-sharing processing, real-time processing and distribution processing; The data processing level configuration includes level information and classification conditions of each data processing level.

6. The device according to claim 5, characterized in that The second configuration module is specifically used to select a data service type for each data source, configure a data processing method and a data processing level configuration according to the second input and the library table of each data source in the data source set; Wherein, the data service type is selected from cluster analysis processing and data update service; The data processing method is selected from one of batch processing, time-sharing processing, real-time processing and distribution processing; The data processing level configuration includes level information and classification conditions of each data processing level.

7. The device according to claim 6, characterized in that The processing module comprises: A first processing unit, configured to process each of the data sources according to the data processing level configuration, to obtain original data sets and data objects with tags at each level, wherein the tags are used to distinguish data service types and data processing methods; A sub-thread unit, used for generating a corresponding sub-thread according to each data object; The second processing unit is used to aggregate the original data set by using the thread blocking tool and each of the sub-threads according to the data service type and data processing method corresponding to each data object, so as to obtain the original data set and the marked data object of the next level; A third processing unit is used for continuing to execute the step of generating a corresponding sub-thread according to each data object when it is determined according to the data processing level configuration that the original data set of the next level needs to be further processed; The fourth processing unit is used to store the original data set of the next level when it is determined according to the data processing level configuration that the original data set of the next level does not need to be further processed.

8. An electronic device, characterized in that: The method comprises a processor, a memory and a program or instruction stored in the memory and executable on the processor, wherein the program or instruction, when executed by the processor, implements the steps of the Java-based data processing method as claimed in any one of claims 1 to 4.

9. A readable storage medium, characterized in that: The readable storage medium stores a program or instruction, and when the program or instruction is executed by a processor, the steps of the Java-based data processing method according to any one of claims 1 to 4 are implemented.

Images