Multi-source dynamic data acquisition and conversion method based on underlying technology

Abstract

The invention discloses a multi-source dynamic data acquisition and conversion method based on an underlying technology, and relates to the technical field of data processing. The multi-source dynamic data acquisition and conversion method based on the underlying technology comprises the following steps: acquiring an unknown data source, and selecting a target registration mode from candidate registration modes according to data characteristics of the unknown data source; performing data source registration on the unknown data source based on the target registration mode to obtain a registered data source; performing grammar compatibility processing on the registration data source to obtain a transition data source; unifying the types of the transition data sources to obtain compatible data sources; and performing data type conversion on the compatible data source to obtain an identifiable data source. According to the scheme, different types of data sources can be acquired, and the data acquisition efficiency is improved.

Description

Technical Field

[0001] The present invention relates to the technical field of data processing, and in particular to a multi-source dynamic data acquisition and conversion method based on underlying technology. Background Art

[0002] In the process of enterprise operation and production, a large amount of relevant data is needed to support and guide the operation and production of the enterprise. Therefore, collecting a variety of relevant data from multiple databases is a prerequisite for supporting the operation and production of the enterprise.

[0003] When collecting data, the current technical solution needs to deal with various types of relational, non-relational, structured, semi-structured and unstructured data. Different types of data usually require different data collection methods, but such data collection methods are inefficient, especially when there are many types of data to be collected, the disadvantage of low collection efficiency is extremely obvious. Therefore, there is an urgent need for a data collection method that can increase the collection efficiency when collecting multiple types of data. Summary of the invention

[0004] The purpose of the present invention is to provide a multi-source dynamic data collection and conversion method based on underlying technology, which achieves the effect of collecting different types of dynamic data sources and increasing data collection efficiency by registering different types of unknown data sources, de-differentiating the registered data sources, and finally converting the data types into recognizable data source types.

[0005] In order to achieve the above object, the present invention provides the following technical solutions:

[0006] In a first aspect, the present invention provides a multi-source dynamic data collection and conversion method based on underlying technology, the method comprising:

[0007] Obtain an unknown data source, and register the unknown data source according to data characteristics of the unknown data source to obtain a registered data source;

[0008] De-differentiate the registered data source to obtain a compatible data source;

[0009] Perform data type conversion on compatible data sources to obtain recognizable data sources.

[0010] In some embodiments, de-differentiation processing is performed on the registered data source to obtain a compatible data source, including:

[0011] Perform syntax compatibility processing on the registered data source to obtain a transitional data source;

[0012] The types of transition data sources are unified to obtain compatible data sources.

[0013] In some embodiments, performing syntax compatibility processing on a registered data source to obtain a transitional data source includes:

[0014] Unify the parameters of the registered data source to obtain a unified data source;

[0015] The differences of the unified data source are shielded, and the unified data source is processed with the same syntax to obtain a transitional data source.

[0016] In some embodiments, the transition data source is unified into a type to obtain a compatible data source, including:

[0017] Get the target data source type;

[0018] Convert the data source type of the transition data source to the target data source type to obtain a compatible data source.

[0019] In some embodiments, registering the unknown data source according to the data characteristics of the unknown data source to obtain the registered data source includes:

[0020] Selecting a target registration method from candidate registration methods according to data characteristics of the unknown data source;

[0021] The unknown data source is registered based on the target registration method to obtain the registered data source.

[0022] In some embodiments, registering an unknown data source based on a target registration method to obtain a registered data source includes:

[0023] Determine a target storage device corresponding to the unknown data source according to data characteristics of the unknown data source;

[0024] Based on the target registration method, the unknown data source is stored in the target memory to obtain the registered data source.

[0025] The beneficial effects of the present invention are:

[0026] First, an unknown data source is obtained, and the unknown data source is registered according to the data characteristics of the unknown data source to obtain a registered data source; then the registered data source is dedifferentiated to obtain a compatible data source; finally, the compatible data source is converted into a recognizable data source. That is, when facing an unknown data source, the above data collection method can be converted into a recognizable data source through registration, dedifferentiation and data type conversion, thereby realizing the collection of different types of data. When there are different types of data sources, there is no need to use different methods for collection respectively. Only the above method can be used to complete the collection of all types of data sources, which greatly increases the efficiency of data collection.

[0027] The above description is only an overview of the technical solution of the present invention. In order to more clearly understand the technical means of the present invention and implement it according to the contents of the specification, the following is a detailed description of the preferred embodiments of the present invention in conjunction with the accompanying drawings. BRIEF DESCRIPTION OF THE DRAWINGS

[0028] Figure 1 A schematic diagram of a flow chart of a multi-source dynamic data collection and conversion method based on underlying technology shown in an embodiment of the present invention;

[0029] Figure 2 A schematic diagram of a process of performing de-differentiation processing on a registered data source according to an embodiment of the present invention;

[0030] Figure 3 A schematic diagram of a process of registering a data source according to an embodiment of the present invention;

[0031] Figure 4 A flowchart of another multi-source dynamic data collection and conversion method based on underlying technology shown in one embodiment of the present invention;

[0032] Figure 5 The flowchart of angle data collection in the system is shown in one embodiment of the present invention. DETAILED DESCRIPTION

[0033] The technical solution of the present invention will be described clearly and completely below in conjunction with the accompanying drawings. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of them. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0034] It should be noted that references to "one embodiment", "embodiment", "example embodiment", etc. in this specification refer to the embodiment being described which may include specific features, structures or characteristics, but not every embodiment must include these specific features, structures or characteristics. In addition, such expressions do not refer to the same embodiment. Furthermore, when describing specific features, structures or characteristics in conjunction with an embodiment, whether or not there is an explicit description, it is indicated that incorporating such features, structures or characteristics into other embodiments is within the knowledge of those skilled in the art.

[0035] In addition, the technical features involved in the different embodiments of the present invention described below can be combined with each other as long as they do not conflict with each other.

[0036] In some embodiments, Figure 1 As shown, a multi-source dynamic data collection and conversion method based on underlying technology is provided, which specifically includes:

[0037] S101, obtaining an unknown data source, selecting a target registration method from candidate registration methods according to data characteristics of the unknown data source; registering the unknown data source based on the target registration method to obtain a registered data source.

[0038] Among them, the unknown data source is an unknown data type, which may be relational data, non-relational data, structured data, semi-structured data, and unstructured data, etc.; the data feature is the data that can reflect the characteristics of the unknown data source, and the data features may include the driver type of the unknown data source, the connection protocol of the unknown data source, the product name of the unknown data source database, etc.; the registered data source refers to the data source that has been registered on the system. Compared with the unknown data source, the registered data source is a known data source.

[0039] Specifically, when data collection is required, the operation and maintenance personnel will upload the unknown data source that needs to be collected to the system, so the unknown data source uploaded by the operation and maintenance personnel can be directly obtained, where the unknown data source includes the data features of the unknown data source, and the driver type of the unknown data source, the connection protocol of the unknown data source, the database product name of the unknown data source and other data are directly decomposed from the unknown data source, and the unknown data source is registered in the system, for example, recorded as registered data source 1. When a data source with the same driver type, connection protocol and database product name as registered data source 1 is encountered again in the future, it is not necessary to register again, and it can be directly classified as registered data source 1; when the data features of the unknown data source cannot be accurately identified, the unknown data source can be directly registered in the system, for example, recorded as registered data source 2.

[0040] S102, performing syntax compatibility processing on the registered data source to obtain a transitional data source; and performing type unification on the transitional data source to obtain a compatible data source.

[0041] The de-differentiation process is to convert different unknown data sources into data sources of the same type.

[0042] Specifically, the registered data sources can be quickly integrated according to the adapter template, the differences between various databases can be shielded, different registered data sources can be operated with the same syntax, and different registered data sources can be converted into compatible data sources of the same type.

[0043] S103, converting the data type of the compatible data source to obtain an identifiable data source.

[0044] Among them, the identifiable data source is the type of data source that the system can identify. When the data is converted into an identifiable data source, since the purpose of data collection is to apply the data, data collection is completed only after the unknown data source is converted into an identifiable data source.

[0045] In the above embodiment, an unknown data source is first obtained, and the unknown data source is registered according to the data characteristics of the unknown data source to obtain a registered data source; then the registered data source is dedifferentiated to obtain a compatible data source; finally, the compatible data source is converted into a data type to obtain an identifiable data source. That is, the above data collection method, when facing an unknown data source, can be converted into an identifiable data source through registration, dedifferentiation and data type conversion, thereby realizing the collection of different types of data. When there are different types of data sources, there is no need to use different methods for collection respectively. Only the above method can be used to complete the collection of all types of data sources, which greatly increases the efficiency of data collection.

[0046] In another embodiment, Figure 2 As shown in FIG. 1 , the process of dedifferentiation of multiple registration data specifically includes:

[0047] S201, performing syntax compatibility processing on a registered data source to obtain a transitional data source.

[0048] The syntax compatibility processing is to use the same syntax to operate different registration data, and the transition data source is the registration data source after the syntax compatibility processing.

[0049] Optionally, the process of performing syntax compatibility processing on the registered data source may be: unifying the parameters of the registered data source to obtain a unified data source; shielding the differences of the unified data source, processing the unified data source with the same syntax, and obtaining a transition data source.

[0050] Exemplarily, since different registered data sources have different parameters, it is necessary to unify the parameters of each registered data source to obtain a unified data source, so as to facilitate subsequent processing of the unified data source using the same syntax to obtain a transition data source.

[0051] S202: Unify the types of transition data sources to obtain compatible data sources.

[0052] Specifically, although the transitional data source has been processed with the same language, its data types are still diverse. The system cannot identify multiple types of transitional data sources. You can call DataReader to process the compatible data source into a Java native type, a custom type, or a project custom type to obtain a compatible data source.

[0053] Optionally, the method for unifying the type of the transition data source may also be to obtain the target data source type; convert the data source type of the transition data source into the target data source type to obtain a compatible data source.

[0054] Specifically, the target data source type can be determined from the candidate data source types according to the specific situation of the system. The target data source type can be a Java native type, a custom type, or a project custom type, that is, the target data source type is adapted to the system. At this time, DataReader can be called to process the transitional data source into the target data source type to obtain a compatible data source, which is then compatible with the system.

[0055] In the above embodiment, the registered data source is firstly processed for syntax compatibility to obtain a transitional data source; then the type of the transitional data source is unified to obtain a compatible data source, so that the compatible data source finally obtained is not only syntax compatible but also of a unified type, which facilitates the subsequent processing of the data source and thereby increases data collection efficiency.

[0056] Based on the above embodiments, Figure 3 As shown in the figure, the process of registering a data source is described in detail. The specific methods include:

[0057] S301, selecting a target registration method from candidate registration methods according to data characteristics of an unknown data source.

[0058] The candidate registration methods include automatic registration and manual registration, the target registration method is one of automatic registration and manual registration, and the data characteristics also include the submission status of unknown data sources.

[0059] Specifically, if the submission status of the unknown data source is a one-time static submission, automatic registration can be selected as the target registration method; if the submission status of the unknown data source is a manual dynamic submission, manual registration can be selected as the target registration method.

[0060] S302, registering the unknown data source based on the target registration method to obtain a registered data source.

[0061] Specifically, if the target registration method is automatic registration, the configuration file can be read when the system starts, and the unknown data source list is first read according to datasource-list, and then the connection, account password and other parameters of each unknown data source are read according to the unknown data source list, and finally the DataSourceHolder.reg() method is called to register the unknown data source. If the target method is manual registration, the DataSourceHolder.reg() method can be manually called after receiving the connection, account password and other parameters of the unknown data source to register the unknown data source.

[0062] Optionally, the target registration method may also be to determine the target storage device corresponding to the unknown data source according to the data characteristics of the unknown data source; based on the target registration method, store the unknown data source in the target storage device to obtain a registered data source.

[0063] Specifically, a relationship mapping table between data features of an unknown data source and candidate memories can be created in advance. After obtaining the data features of the unknown data source, the target memory corresponding to the unknown data source can be determined by looking up the table. The unknown data source is then stored in the target memory, thereby completing the registration of the unknown data source and obtaining the registered data source.

[0064] In the above embodiment, a target registration method is first selected from candidate registration methods based on the data characteristics of the unknown data source; then the unknown data source is registered based on the target registration method to obtain a registered data source, that is, different unknown data sources correspond to different registration methods, which makes the registration method of the unknown data source more specific and increases the accuracy of registering the unknown data source.

[0065] In order to more comprehensively demonstrate this solution, this embodiment provides an optional method for multi-source dynamic data collection and conversion based on underlying technology, such as Figure 4 As shown:

[0066] S401, obtaining an unknown data source.

[0067] S402: Select a target registration method from candidate registration methods according to data characteristics of the unknown data source.

[0068] S403: Determine a target storage device corresponding to the unknown data source according to data characteristics of the unknown data source.

[0069] S404, based on the target registration method, storing the unknown data source in the target storage to obtain a registered data source.

[0070] S405, unify the parameters of the registered data source to obtain a unified data source.

[0071] S406, shielding the differences of the unified data source, processing the unified data source with the same syntax, and obtaining a transition data source.

[0072] S407, obtaining the target data source type.

[0073] S408: Convert the data source type of the transition data source into the target data source type to obtain a compatible data source.

[0074] S409: convert the data type of the compatible data source to obtain a recognizable data source.

[0075] The specific process of S401-S408 above can refer to the description of the above method embodiment, and its implementation principle and technical effect are similar, which will not be repeated here.

[0076] In another embodiment, Figure 5 As shown in the figure, the data collection process is presented from a system perspective:

[0077] Among them, service is mainly used to access input parameters and divert cached data and database data, and the data that is not cached is handed over to dao for processing; dao: mainly used to encapsulate a unified RunPrepare, because the service provides a large number of overloads for the convenience of calling, resulting in a variety of input parameters. Dao will unify the service parameters in this step and hand them over to the adapter; adapter (syntax compatibility): has been bound to a specific data source type. After receiving the previous step RunPrepare, it will generate a command (Run object) suitable for the current database according to the data source type. For example, different databases have different paging methods, and then hand the generated command to the actuator for execution; actuator: parses the command into the format required by the driver, and calls the driver to return data to the adapter; adapter (type compatibility): after receiving the data returned by the driver, it will obtain the corresponding DataReader through DataReaderFactory according to the data type and database type, and then the DataReader will process the data into Java native type or custom type or project defined type, that is, to achieve type unification. The data returned by the driver is divided into Java native types and driver-defined types. Java native types are generally not processed and returned as is, but there are special cases such as Point types. Some databases will return byte[] arrays. At this time, they generally need to be converted into Point objects for subsequent operations. Driver-defined types generally need to be converted. Common ones such as Blob and Clob are often returned by different drivers. Different custom types; DataReaderFactory is divided into SystemDataReaderFactory and UserDataReaderFactory; SystemDataReaderFactory predefines conversions between different data types. When encountering types not defined in SystemDataReaderFactory, you can register a new DataReader in a specific project through UserDataReaderFactory.

[0078] The DataReader implementation process is:

[0079] DataReaderFactory.reg(new Object[]{NClobProxyImpl.class},newDataReader(){

[0080] @Override

[0081] public Object read(Object value){

[0082] if(value instanceof NClobProxyImpl){

[0083] try{

[0084] NClobProxyImpl clob=(NClobProxyImpl)value;

[0085] long len = clob.length();

[0086] if(len>0){

[0087] return clob.getSubString(0,(int)len);

[0088] }

[0089] }catch(Exception e){

[0090] e.printStackTrace();

[0091] }

[0092] }

[0093] return value;

[0094] }

[0095] });

[0096] Obviously, those skilled in the art should understand that the above modules or steps of the present application can be implemented by a general computing device, they can be concentrated on a single computing device, or distributed on a network composed of multiple computing devices, and optionally, they can be implemented by a program code executable by a computing device, so that they can be stored in a storage device and executed by the computing device, or they can be made into individual integrated circuit modules, or multiple modules or steps therein can be made into a single integrated circuit module for implementation. In this way, the present application is not limited to any specific combination of hardware and software.

[0097] The technical features of the above embodiments may be arbitrarily integrated. To make the description concise, not all possible integrations of the technical features in the above embodiments are described. However, as long as there is no contradiction in the integration of these technical features, they should be considered to be within the scope of this specification.

[0098] The above embodiments only express several implementation methods of the present invention, and the descriptions thereof are relatively specific and detailed, but they cannot be understood as limiting the scope of the invention patent. It should be pointed out that, for those of ordinary skill in the art, several variations and improvements can be made without departing from the concept of the present invention, and these all belong to the protection scope of the present invention. Therefore, the protection scope of the patent of the present invention shall be subject to the attached claims.

Claims

1. A multi-source dynamic data acquisition and conversion method based on underlying technology, characterized in that: The method comprises: Acquire an unknown data source, and select a target registration method from candidate registration methods according to data characteristics of the unknown data source; perform data source registration on the unknown data source based on the target registration method to obtain a registered data source; Performing syntax compatibility processing on the registered data source to obtain a transitional data source; unifying the type of the transitional data source to obtain a compatible data source; The data type of the compatible data source is converted to obtain an identifiable data source.

2. The multi-source dynamic data acquisition and conversion method based on underlying technology as claimed in claim 1 is characterized in that: The registered data source is subjected to syntax compatibility processing to obtain a transitional data source, including: Unifying the parameters of the registered data sources to obtain a unified data source; The differences of the unified data source are shielded, and the unified data source is processed with the same syntax to obtain a transitional data source.

3. The multi-source dynamic data acquisition and conversion method based on underlying technology as claimed in claim 2 is characterized in that: The transition data source is unified in type to obtain a compatible data source, including: Get the target data source type; The data source type of the transition data source is converted into a target data source type to obtain a compatible data source.

4. The multi-source dynamic data acquisition and conversion method based on underlying technology as claimed in claim 1, characterized in that: The data features of the unknown data source include the driver type of the unknown data source, the connection protocol of the unknown data source, the product name of the database of the unknown data source and the submission status of the unknown data source.

5. The multi-source dynamic data acquisition and conversion method based on underlying technology as claimed in claim 1, characterized in that: The candidate registration methods include automatic registration and manual registration. According to the data characteristics of the unknown data source, a target registration method is selected from the candidate registration methods, including: If the submission status of the unknown data source is a one-time static submission, automatic registration is selected as the target registration method; If the submission status of the unknown data source is manual dynamic submission, manual registration is selected as the target registration method.

6. The multi-source dynamic data acquisition and conversion method based on underlying technology as claimed in claim 1, characterized in that: Registering the unknown data source based on the target registration method to obtain a registered data source includes: Determining a target storage device corresponding to the unknown data source according to data characteristics of the unknown data source; Based on the target registration method, the unknown data source is stored in the target memory to obtain a registered data source.

Images