Data storage and retrieval method and device for heterogeneous data sources

Abstract

The present invention discloses a data storage and data retrieval method and device for heterogeneous data sources, which relates to the technical field of heterogeneous data source processing, and can optimize the collection, storage and retrieval operations of heterogeneous data sources, reduce processing costs, improve processing efficiency, and facilitate subsequent maintenance and management. The main technical scheme of the present invention is: receiving a data collection instruction, the data collection instruction contains an identifier of a heterogeneous data source; based on the identifier of the heterogeneous data source, finding the corresponding heterogeneous data source, wherein the heterogeneous data source has an embedded annotation, and the annotation is used to configure the data collection method of the heterogeneous data source; according to the embedded annotation in the heterogeneous data source, executing the data collection operation to obtain the target data; using a preset configuration file, storing the target data in a column storage database, and then using such a column storage database to provide a retrieval operation for the heterogeneous data source.

Description

Technical Field

[0001] The present invention relates to the technical field of heterogeneous data source processing, and in particular to a data storage and data retrieval method and device for heterogeneous data sources. Background Art

[0002] Data retrieval operations can span multiple data sources. The data source is the source of the data. Different data sources contain a large amount of data generated by business activities in various industries (for example, log data). These large amounts of data from different sources may differ in data structure, storage format, etc., forming heterogeneous data sources. The amount of data they contain is huge, which can be in the tens of billions.

[0003] At present, it is usually necessary to write customized scripts for each data source to implement the functions of data analysis and collection of required data, as well as to store the collected data and provide external retrieval capabilities. However, this places extremely high demands on program performance and reliability, which is equivalent to the need for customized development, resulting in high development costs and is not conducive to subsequent maintenance and management. Summary of the invention

[0004] In view of this, the present invention provides a data storage and data retrieval method and device for heterogeneous data sources. The main purpose is to use a common method to realize the collection, storage and retrieval operations of heterogeneous data sources, optimize the relevant data processing methods for heterogeneous data sources, reduce data processing costs, improve data processing efficiency, and also facilitate subsequent maintenance and management.

[0005] In a first aspect, the present application provides a data storage method for heterogeneous data sources, the method comprising:

[0006] receiving a data collection instruction, wherein the data collection instruction includes an identifier of a heterogeneous data source;

[0007] Based on the identifier of the heterogeneous data source, a corresponding heterogeneous data source is found, wherein the heterogeneous data source has an annotation embedded therein, and the annotation is used to configure a data collection method of the heterogeneous data source;

[0008] According to the annotations embedded in the heterogeneous data sources, a data collection operation is performed to obtain target data;

[0009] The target data is stored in a column storage database using a preset configuration file.

[0010] In some modified implementations of the first aspect of the present application, performing a data collection operation to obtain target data according to the annotation embedded in the heterogeneous data source includes:

[0011] Acquire the interface and data conversion format of the heterogeneous data source from the annotation;

[0012] Collecting data from the heterogeneous data source according to the interface;

[0013] According to the data conversion format, the collected data is converted into a unified format to obtain target data.

[0014] In some modified implementations of the first aspect of the present application, storing the target data in a column storage database by using a preset configuration file includes:

[0015] Acquire multiple preset attribute fields, multiple storage thresholds, and multiple storage addresses from the preset configuration file, wherein each storage threshold is associated with a different storage address;

[0016] Acquire, from the multiple storage thresholds, a storage threshold corresponding to a data volume greater than the target data as a target storage threshold;

[0017] Determining a corresponding associated target storage address from the multiple storage addresses according to the target storage amount threshold;

[0018] Creating a column storage database in the storage space corresponding to the target storage address according to the plurality of preset attribute fields;

[0019] The target data is stored in the column storage database.

[0020] In some modified implementations of the first aspect of the present application, after creating a column storage database in the storage space corresponding to the target storage address according to the multiple preset attribute fields, the method further includes:

[0021] Acquire a target attribute field from the plurality of preset attribute fields;

[0022] Setting the data storage format corresponding to the target attribute field, so as to store data information in the target attribute field in the data storage format; and / or,

[0023] If there are multiple target attribute fields, the multiple target attribute fields are parsed to obtain the association relationship between the multiple target attribute fields, so as to preferentially store data information in the target attribute fields having the association relationship when storing the target data in the column storage database.

[0024] In some modified implementations of the first aspect of the present application, storing the target data in the column storage database includes:

[0025] In the process of storing the target data using the plurality of preset attribute fields, storing corresponding target data in the target attribute fields according to the data storage format;

[0026] If there are multiple target attribute fields, corresponding target data are stored in the multiple target attribute fields according to the association relationship between the multiple target attribute fields.

[0027] A second aspect of the present application provides a data retrieval method for heterogeneous data sources, which is applied to a column storage database obtained by the data storage method for heterogeneous data sources as described above, and the method comprises:

[0028] receiving a data retrieval instruction, wherein the data retrieval instruction carries retrieval information;

[0029] By traversing each attribute column in the column storage database, a search result matching the search information is searched.

[0030] In some modified implementations of the second aspect of the present application, searching for a search result matching the search information by traversing each attribute column in the column storage database includes:

[0031] Parsing the search conditions and search keywords from the search information;

[0032] According to the search keyword, traverse the attribute information under each attribute field in the column storage database one by one to find matching target attribute information;

[0033] Determine the target attribute field corresponding to the attribute according to the target attribute information;

[0034] Under the target attribute field, search for search results that match the search condition.

[0035] A third aspect of the present application provides a data storage device for heterogeneous data sources, the device comprising:

[0036] A receiving unit, configured to receive a data collection instruction, wherein the data collection instruction includes an identifier of a heterogeneous data source;

[0037] A search unit, configured to search for a corresponding heterogeneous data source based on an identifier of the heterogeneous data source, wherein the heterogeneous data source has an embedded annotation, and the annotation is used to configure a data collection method of the heterogeneous data source;

[0038] A collection unit, configured to perform a data collection operation to obtain target data according to the annotations embedded in the heterogeneous data source;

[0039] The storage unit is used to store the target data in a column storage database by using a preset configuration file.

[0040] In some modified implementations of the third aspect of the present application, the collection unit includes:

[0041] An acquisition module, used to acquire the interface and data conversion format of the heterogeneous data source from the annotation;

[0042] A collection module, used for collecting data from the heterogeneous data source according to the interface;

[0043] The processing module is used to convert the collected data into a unified format according to the data conversion format to obtain target data.

[0044] In some modified implementations of the third aspect of the present application, the storage unit includes:

[0045] An acquisition module, used for acquiring a plurality of preset attribute fields, a plurality of storage thresholds and a plurality of storage addresses from the preset configuration file, wherein each of the storage thresholds is associated with a different storage address;

[0046] The acquisition module is further configured to acquire, from the multiple storage thresholds, a storage threshold corresponding to a data volume greater than the target data as a target storage threshold;

[0047] A determination module, configured to determine a corresponding associated target storage address from the plurality of storage addresses according to the target storage amount threshold;

[0048] A creation module, configured to create a column storage database on a storage space corresponding to the target storage address according to the plurality of preset attribute fields;

[0049] A storage module is used to store the target data in the column storage database.

[0050] In some modified implementations of the third aspect of the present application, the storage unit further includes:

[0051] The acquisition module is further used to acquire a target attribute field from the plurality of preset attribute fields;

[0052] A setting module, used for setting a data storage format corresponding to the target attribute field, so as to store data information in the target attribute field in the data storage format;

[0053] A module is established, which is used to parse the target attribute fields to obtain the association relationship between the target attribute fields when there are multiple target attribute fields, so as to store data information in the target attribute fields with the association relationship preferentially when storing the target data in the column storage database.

[0054] In some modified implementations of the third aspect of the present application, the storage module is further specifically used for:

[0055] In the process of storing the target data using the plurality of preset attribute fields, storing corresponding target data in the target attribute fields according to the data storage format;

[0056] When there are multiple target attribute fields, corresponding target data are stored in the multiple target attribute fields according to the association relationship between the multiple target attribute fields.

[0057] A fourth aspect of the present application provides a data retrieval device for heterogeneous data sources, the device comprising:

[0058] A receiving unit, used to receive a data retrieval instruction, wherein the data retrieval instruction carries retrieval information;

[0059] The search unit is used to search for a search result matching the search information by traversing each attribute column in the column storage database.

[0060] In some modified implementations of the fourth aspect of the present application, the search unit includes:

[0061] A parsing module, used to parse the search information to obtain search conditions and search keywords;

[0062] A search module, used to traverse the attribute information under each attribute field in the column storage database one by one according to the search keyword, and search for matching target attribute information;

[0063] A determination module, used to determine a target attribute field corresponding to the attribution according to the target attribute information;

[0064] The search module is further used to search for search results matching the search condition under the target attribute field.

[0065] A fifth aspect of the present application provides a computer-readable storage medium having a computer program stored thereon. When the computer program is executed by a processor, the computer program implements the retrieval method for heterogeneous data sources as described above, and implements the data retrieval method for heterogeneous data sources as described above.

[0066] In a sixth aspect, the present application provides an electronic device, comprising: a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein when the processor executes the computer program, the retrieval method for heterogeneous data sources as described above is implemented, and the data retrieval method for heterogeneous data sources as described above is implemented.

[0067] By means of the above technical solution, the technical solution provided by the present invention has at least the following advantages:

[0068] The present invention provides a data storage and data retrieval method and device. The present invention adds annotations to the heterogeneous data source to be collected in advance, and then when receiving the data collection instruction, performs the data collection operation according to the embedded annotations of the heterogeneous data source to obtain the target data, and then stores the target data in a column storage database using a preset configuration file, thereby realizing the collection and storage operation of each heterogeneous data source in a general manner. And when receiving the retrieval instruction, the corresponding retrieval result can be fed back using the column storage database. Compared with the prior art, the problem of high cost and disadvantageous for subsequent maintenance and management caused by customized development for each heterogeneous data source to provide collection, storage and retrieval functions is solved. The present invention optimizes the collection and storage operation of the heterogeneous data source, and does not need to customize the development for each heterogeneous data source to achieve the collection and storage functions, that is, there is no need to invasively modify the code writing, but to realize the collection and storage functions in a general manner, and correspondingly also provides a general retrieval function, according to which the general processing method reduces the processing cost, is efficient, and is also conducive to subsequent maintenance and management.

[0069] The above description is only an overview of the technical solution of the present invention. In order to more clearly understand the technical means of the present invention, it can be implemented according to the contents of the specification. In order to make the above and other purposes, features and advantages of the present invention more obvious and easy to understand, the specific implementation methods of the present invention are listed below. BRIEF DESCRIPTION OF THE DRAWINGS

[0070] Various other advantages and benefits will become apparent to those of ordinary skill in the art by reading the detailed description of the preferred embodiments below. The accompanying drawings are only for the purpose of illustrating the preferred embodiments and are not to be considered as limiting the present invention. Moreover, the same reference symbols are used throughout the accompanying drawings to represent the same components. In the accompanying drawings:

[0071] Figure 1 A flow chart of a data storage method for heterogeneous data sources provided by an embodiment of the present invention;

[0072] Figure 2 A flow chart of another data storage method for heterogeneous data sources provided by an embodiment of the present invention;

[0073] Figure 3 A flow chart of a data retrieval method for heterogeneous data sources provided by an embodiment of the present invention;

[0074] Figure 4 Flowchart of another data retrieval method for heterogeneous data sources provided by an embodiment of the present invention

[0075] Figure 5A block diagram of a data storage device for heterogeneous data sources provided by an embodiment of the present invention;

[0076] Figure 6 A block diagram of another data storage device for heterogeneous data sources provided by an embodiment of the present invention;

[0077] Figure 7 A block diagram of a data retrieval device for heterogeneous data sources provided by an embodiment of the present invention;

[0078] Figure 8 A block diagram of another data retrieval device for heterogeneous data sources provided by an embodiment of the present invention. DETAILED DESCRIPTION

[0079] The exemplary embodiments of the present invention will be described in more detail below with reference to the accompanying drawings. Although the exemplary embodiments of the present invention are shown in the accompanying drawings, it should be understood that the present invention can be implemented in various forms and should not be limited by the embodiments set forth herein. On the contrary, these embodiments are provided in order to enable a more thorough understanding of the present invention and to enable the scope of the present invention to be fully communicated to those skilled in the art.

[0080] The embodiment of the present invention provides a data storage method for heterogeneous data sources, such as Figure 1 As shown, the method realizes the collection and storage functions in a universal manner for different heterogeneous data sources, and the embodiment of the present invention provides the following specific steps:

[0081] 101. Receive a data collection instruction, where the data collection instruction includes an identifier of a heterogeneous data source.

[0082] The data source refers to the origin of the data. Different data sources contain a large amount of data (for example, log data) generated by business activities in various industries. These large amounts of data from different sources may differ in data structure, storage method, etc., constituting heterogeneous data sources.

[0083] In the embodiment of the present invention, the received data collection instruction may include one or more data source identifiers, where the identifiers are used to indicate data collection objects.

[0084] 102. Based on the identifier of the heterogeneous data source, find the corresponding heterogeneous data source.

[0085] 103. According to the annotations embedded in the heterogeneous data sources, data collection operations are performed to obtain target data.

[0086] Among them, annotations are embedded in heterogeneous data sources. Annotations are used to configure the data collection method of heterogeneous data sources. Specifically, annotations can be understood as special tags in the code. These tags can be read during compilation, class loading, and runtime, and corresponding processing can be performed. Through annotations, developers can embed supplementary information in the source code without changing the original code and logic.

[0087] In the embodiment of the present invention, the annotations are universal for different heterogeneous data sources, rather than custom-developed scripts written for each data source to implement certain specified operations. After the collection object (i.e., data source) is determined, the annotations are pre-written into the data source.

[0088] Specifically, at the code level, a software development kit (SDK) can be used to write annotations to the starting end of the data source. Annotations are equivalent to special flags in the code. These flags can be read during compilation, class loading, and runtime, and corresponding processing is performed to facilitate other tools to supplement information or deploy. In the embodiment of the present invention, data collection operations are performed by running the annotations embedded in the data source to obtain target data.

[0089] 104. Use the preset configuration file to store the target data in the column storage database.

[0090] The preset configuration file at least includes preset attribute fields and storage addresses. The preset attribute fields are used to indicate which attribute fields are used to store the target data; the storage address is used to indicate where the target data is stored, such as a memory or a server. In addition, if the storage location requires authorized login, the corresponding account and password can also be stored in the preset configuration file.

[0091] Among them, the column storage database refers to storing the collected target data according to the column storage index. The column attributes are stored in units of attributes (columns) in the relational database. The data information of the same attribute is stored together in the data table, and the attribute information of different attributes in a record is stored in different storage units.

[0092] In an embodiment of the present invention, the preset configuration file is equivalent to providing a standard for storing target data. And in combination with such a storage standard, the role of using a column storage database to implement storage of target data is: for multiple heterogeneous data sources, their data volume can be tens of billions, and the number of types of attribute information obtained by collecting these heterogeneous data sources is also very large, and then the number of attribute fields required to store these attribute information using the created database is also very large. If the data information is stored in a row-based storage manner, then during the retrieval operation, a large number of attribute fields will be traversed to search for data information, which will waste too much retrieval processing costs and result in low retrieval efficiency. However, if the data information is stored in a column storage manner, during the retrieval operation, the next attribute field will be found based on the column storage index, that is, the attribute information in one attribute field is traversed, so there is no need to traverse all attribute fields to find matching retrieval results.

[0093] As described above, an embodiment of the present invention provides a data storage method. The embodiment of the present invention adds annotations to the heterogeneous data source to be collected in advance, and then when receiving the data collection instruction, performs the data collection operation according to the embedded annotations of the heterogeneous data source to obtain the target data, and then stores the target data in the column storage database using the preset configuration file, thereby realizing the collection and storage operations of each heterogeneous data source in a general manner. Compared with the prior art, the problem of high cost and disadvantageous subsequent maintenance and management caused by customized development of collection and storage functions for each heterogeneous data source is solved. The present invention optimizes the collection and storage operations of heterogeneous data sources, and does not need to customize the development for each heterogeneous data source to achieve the collection and storage functions, that is, there is no need to invasively modify the code writing, but to realize the collection and storage functions in a general manner, reduce the processing cost of data collection and storage, improve data processing efficiency, and also facilitate subsequent maintenance and management.

[0094] In order to explain the above embodiment in more detail, the embodiment of the present invention also provides another data storage method for heterogeneous data sources, such as Figure 2 As shown, this method is a detailed explanation of the above embodiment, and this embodiment of the present invention provides the following specific steps:

[0095] 201. Receive a data collection instruction, where the data collection instruction includes an identifier of a heterogeneous data source.

[0096] In the embodiment of the present invention, this step is explained with reference to step 101 and will not be described again here.

[0097] 202. Based on the identifier of the heterogeneous data source, find the corresponding heterogeneous data source.

[0098] 203. According to the annotations embedded in the heterogeneous data source, a data collection operation is performed to obtain target data.

[0099] Among them, annotations are embedded in the heterogeneous data sources, and the annotations are used to configure the data collection method of the heterogeneous data sources. The annotations include the interface and data conversion format of the heterogeneous data sources.

[0100] In an embodiment of the present invention, a specific implementation method for realizing data collection by using annotations embedded in heterogeneous data sources includes the following:

[0101] The first step is to obtain the interface and data conversion format of heterogeneous data sources from annotations.

[0102] Secondly, data is collected from heterogeneous data sources according to the interface, and according to the data conversion format, the collected data is converted into a unified format to obtain the target data.

[0103] It should be noted that for multiple heterogeneous data sources, they differ in data structure, storage format, etc. Therefore, it is necessary to use the data conversion format mentioned in the embodiment of the present invention to realize standardized format processing of data information collected from multiple heterogeneous data sources, so as to obtain standard and unified data information as target data for subsequent storage operations.

[0104] 204. Obtain multiple preset attribute fields, multiple storage thresholds, and multiple storage addresses from the preset configuration file, wherein each storage threshold is associated with a different storage address.

[0105] Among them, the preset attribute field is a standardized and universal attribute field obtained by integrating databases in multiple technical fields. Specifically, for the target data collected from heterogeneous data sources in the embodiment of the present invention, since they come from different heterogeneous data sources, and the number of attribute types contained in multiple heterogeneous data sources is also very large, therefore, in order to effectively store more diverse data information, the embodiment of the present invention can obtain attribute fields from these heterogeneous data sources to construct a standardized and universal attribute field.

[0106] For example, common attribute fields are obtained by taking the intersection of attribute fields contained in multiple heterogeneous data sources; secondly, attribute fields are derived based on common attribute fields. Specifically, for derived attribute fields, semantic analysis methods can be used, such as merging attribute fields with similar semantics, or expanding attribute fields of more subordinate concepts and refined branches based on high-frequency attribute fields. To further illustrate, the attribute types contained in multiple heterogeneous data sources are diverse and large in number. For those attribute types that appear very infrequently, their data value is not high, and they can be cleaned from the collected target data instead of being stored in a column storage database. Specifically, these very low-frequency attribute fields can be set in the database to avoid storing too much useless redundant data in the database and occupying storage resources.

[0107] Among them, the storage capacity threshold refers to the data capacity threshold of the storage space, the storage address refers to the address of the storage space, and each storage capacity threshold is associated with a different storage address, so as to select the corresponding storage space according to the amount of data collected from the target data. It should be noted that the collected target data may include dynamic data and static data. Dynamic data is data that changes frequently and directly reflects the transaction process, such as website visits, number of online users, daily sales, etc. Since dynamic data has a short update cycle and a large amount of data, the optimized processing method is to process dynamic data and static data separately.

[0108] In the embodiment of the present invention, the preset attribute fields, storage thresholds and storage addresses in the preset configuration files can be used to automatically create a corresponding database for target data and complete storage.

[0109] 205. Obtain a storage capacity threshold corresponding to a data volume greater than the target data from the multiple storage capacity thresholds as the target storage capacity threshold.

[0110] 206. Determine a corresponding associated target storage address from a plurality of storage addresses according to the target storage amount threshold.

[0111] In the embodiment of the present invention, steps 205-206 are explained, and multiple storage thresholds are stored in the preset configuration file, and each storage threshold corresponds to a storage address. In order to avoid the situation where the storage space corresponding to the storage address is insufficient to accommodate the storage target data, the data volume of the target data can be compared with different storage thresholds, and the storage address corresponding to the storage threshold that is greater than the data volume of the target data can be found. Of course, if there are multiple such storage addresses, any one can be selected from them, so as to obtain the storage space using the storage address to complete the storage of the target data. Among them, each storage address can include: a starting address and an end address stored in pairs.

[0112] In another embodiment, in order to improve the utilization rate of storage space, after finding multiple storage addresses corresponding to storage capacity thresholds that are greater than the data volume of the target data, a storage address corresponding to the smallest storage capacity threshold can be selected from the multiple storage addresses, thereby ensuring that the corresponding address space is occupied as much as possible to prevent waste of resources.

[0113] 207. Create a column storage database in the storage space corresponding to the target storage address according to the multiple preset attribute fields.

[0114] In the embodiment of the present invention, the column storage database is mainly created according to the preset attribute fields in the preset configuration file, so the column storage database includes these preset attribute fields. Further, these preset attribute fields can also be optimized, specifically including the following:

[0115] Obtain a target attribute field from multiple preset attribute fields, that is, the preset attribute field that needs to be optimized, referred to as the target attribute field for short. The optimization method for the target attribute field is as follows:

[0116] For example, by preprocessing the target attribute field, the data storage format corresponding to the target attribute field is set.

[0117] For example, in some specific fields with special meanings, or for example, according to actual business needs, when storing the names of provinces, cities, districts, and counties, it is necessary to store both the text names and their mapped codes.

[0118] Alternatively, as a parallel solution, if there are multiple target attribute fields, the association relationship between the multiple target attribute fields is obtained by parsing the multiple target attribute fields. Based on this association relationship, data information can be stored preferentially in the target attribute field with the association relationship. For example, after storing data information in a target attribute field, if the target attribute field has an association relationship with another target attribute field, based on this association relationship, the data information is preferentially stored in the other target attribute field, and then the data storage operation on the database is completed according to other attribute fields.

[0119] It should be noted that storing data based on association relationships can not only effectively improve data storage efficiency during storage, but also, during subsequent retrieval, based on a target attribute field, it can bring out other target attribute fields that have an association relationship with it, thereby effectively improving data query efficiency.

[0120] Exemplarily, as mentioned above, the provinces, cities, districts, and counties correspond to different attribute fields respectively, and the association relationship between these attribute fields can be established in advance according to a certain order. Based on the association relationship, data information is stored in the different attribute fields corresponding to the provinces, cities, districts, and counties in turn according to this order. Since the collected target data may also be related, priority storage operations are completed based on the target attribute fields with related relationships, thereby improving data storage efficiency.

[0121] 208. Store the target data in a column storage database.

[0122] In the embodiment of the present invention, the target data collected is subjected to data analysis processing to determine the attribute field to be stored, thereby completing the storage operation. However, the target attribute field preset in step 207 needs to be described as follows:

[0123] In the process of storing target data using multiple preset attribute fields, the corresponding data is stored in the target attribute field according to the data storage format. If there are multiple target attribute fields, the corresponding data is stored in the multiple target attribute fields according to the association relationship between the multiple target attribute fields.

[0124] Furthermore, based on the column storage database obtained by the data storage method for heterogeneous data sources provided in the embodiment of the present invention, the embodiment of the present invention also provides a data retrieval method for heterogeneous data sources, such as Figure 3 As shown, the embodiment of the present invention provides the following specific steps:

[0125] 301. Receive a data retrieval instruction, where the data retrieval instruction carries retrieval information.

[0126] 302. Search for a search result matching the search information by traversing each attribute column in the column storage database.

[0127] In the embodiment of the present invention, the column storage database stores data information from multiple heterogeneous data sources, which is equivalent to integrating and preprocessing the data information in multiple heterogeneous data sources through data collection and storage. Then, for receiving a search instruction, it can be indirectly equivalent to a search operation for these heterogeneous data sources. Therefore, finding the search result by searching the column storage database is equivalent to the search result obtained by initiating a search instruction to the original heterogeneous data source.

[0128] In the embodiment of the present invention, the search results are obtained based on searching the column storage database, which replaces the original process of initiating search instructions to each heterogeneous data source to obtain relevant data information, thereby greatly improving the search efficiency and search versatility.

[0129] Furthermore, in order to explain the search operation in detail, the embodiment of the present invention also provides another data search method for heterogeneous data sources, such as Figure 4 As shown, the embodiment of the present invention provides the following specific steps:

[0130] 401. Receive a data retrieval instruction, where the data retrieval instruction carries retrieval information.

[0131] 402. Analyze the search conditions and search keywords from the search information.

[0132] 403. According to the search keyword, the attribute information under each attribute field in the column storage database is traversed one by one to find matching target attribute information.

[0133] The embodiment of the present invention utilizes a column-based storage database to store a large number of types of attribute information obtained by collecting these heterogeneous data sources, that is, the number of attribute fields in the column-based storage database is also large. Based on the column-based storage characteristics of the column-based storage database, it is also possible to read data in an entire column when reading data, that is, when an attribute field is retrieved, the data information of the entire attribute column can be read. Such a search can avoid traversing a large number of attribute fields. Instead, after determining the attribute field where the search keyword is located, there is no need to traverse other attribute fields, thereby greatly improving the search efficiency.

[0134] 404. Determine the target attribute field corresponding to the attribution according to the target attribute information.

[0135] 405. Under the target attribute field, search for search results that match the search criteria.

[0136] In the embodiment of the present invention, after the matching target attribute information is determined according to the search keyword, a search operation is further performed under the target attribute field according to the search condition to efficiently obtain the search result.

[0137] Furthermore, as a response to the above Figure 1 , Figure 2 The implementation of the method shown in the figure, the embodiment of the present invention provides a data storage device for heterogeneous data sources. The device embodiment corresponds to the aforementioned method embodiment. For ease of reading, the device embodiment will not repeat the details of the aforementioned method embodiment one by one, but it should be clear that the device in this embodiment can implement all the contents of the aforementioned method embodiment. The device is used to implement the collection and storage operations in a universal manner for different heterogeneous data sources, specifically as follows Figure 5 As shown, the device comprises:

[0138] A receiving unit 31, configured to receive a data collection instruction, wherein the data collection instruction includes an identifier of a heterogeneous data source;

[0139] A search unit 32, configured to search for a corresponding heterogeneous data source based on the identifier of the heterogeneous data source, wherein the heterogeneous data source has an embedded annotation, and the annotation is used to configure a data collection method of the heterogeneous data source;

[0140] A collection unit 33, configured to perform a data collection operation to obtain target data according to the annotations embedded in the heterogeneous data source;

[0141] The storage unit 34 is used to store the target data in a column storage database using a preset configuration file.

[0142] Further, such as Figure 6 As shown, the acquisition unit 33 includes:

[0143] An acquisition module 331 is used to acquire the interface and data conversion format of the heterogeneous data source from the annotation;

[0144] A collection module 332, used for collecting data from the heterogeneous data source according to the interface;

[0145] The processing module 333 is used to perform unified format conversion processing on the collected data according to the data conversion format to obtain target data.

[0146] Further, such as Figure 6 As shown, the storage unit 34 includes:

[0147] An acquisition module 341 is used to acquire a plurality of preset attribute fields, a plurality of storage thresholds and a plurality of storage addresses from the preset configuration file, wherein each of the storage thresholds is associated with a different storage address;

[0148] The acquisition module 341 is further configured to acquire, from the multiple storage thresholds, a storage threshold corresponding to a data volume greater than the target data as a target storage threshold;

[0149] A determination module 342, configured to determine a corresponding associated target storage address from the plurality of storage addresses according to the target storage amount threshold;

[0150] A creation module 343, configured to create a column storage database in the storage space corresponding to the target storage address according to the plurality of preset attribute fields;

[0151] The storage module 344 is used to store the target data in the column storage database.

[0152] Further, such as Figure 6 As shown, the storage unit 34 also includes:

[0153] The acquisition module 341 is further used to acquire a target attribute field from the plurality of preset attribute fields;

[0154] A setting module 345, configured to set a data storage format corresponding to the target attribute field, so as to store data information in the target attribute field in the data storage format;

[0155] Establish module 346, which is used to parse the multiple target attribute fields to obtain the association relationship between the multiple target attribute fields when there are multiple target attribute fields, so as to preferentially store data information in the target attribute fields having the association relationship when storing the target data in the column storage database.

[0156] Further, such as Figure 6 As shown, the storage module 344 is also specifically used for:

[0157] In the process of storing the target data using the plurality of preset attribute fields, storing corresponding target data in the target attribute fields according to the data storage format;

[0158] When there are multiple target attribute fields, corresponding target data are stored in the multiple target attribute fields according to the association relationship between the multiple target attribute fields.

[0159] Furthermore, as a response to the above Figure 3 , Figure 4 In order to realize the method shown in the figure, the embodiment of the present invention provides a data retrieval device for heterogeneous data sources. The device embodiment corresponds to the aforementioned method embodiment. For ease of reading, the device embodiment will not repeat the details of the aforementioned method embodiment one by one, but it should be clear that the device in this embodiment can correspond to all the contents of the aforementioned method embodiment. The device is used to complete the retrieval operation for a database containing heterogeneous data sources, specifically as follows Figure 7 As shown, the device comprises:

[0160] A receiving unit 41, configured to receive a data retrieval instruction, wherein the data retrieval instruction carries retrieval information;

[0161] The search unit 42 is used to search for a search result matching the search information by traversing each attribute column in the column storage database.

[0162] Further, such as Figure 8 As shown, the search unit 42 includes:

[0163] The parsing module 421 is used to parse the search information to obtain the search conditions and search keywords;

[0164] A search module 422 is used to traverse the attribute information under each attribute field in the column storage database one by one according to the search keyword to find matching target attribute information;

[0165] A determination module 423, configured to determine a target attribute field corresponding to the attribution according to the target attribute information;

[0166] The search module 422 is further configured to search for search results matching the search condition under the target attribute field.

[0167] In summary, the embodiment of the present invention provides a data storage and data retrieval method and device. The present invention adds annotations to the heterogeneous data source to be collected in advance, and then when receiving the data collection instruction, according to the embedded annotations of the heterogeneous data source, performs the data collection operation to obtain the target data, and then stores the target data in the column storage database using the preset configuration file, thereby realizing the collection and storage operation of each heterogeneous data source in a general manner. And when receiving the retrieval instruction, the corresponding retrieval result can be fed back using the column storage database. Compared with the prior art, the problem of high cost and disadvantageous subsequent maintenance and management caused by customized development for each heterogeneous data source to provide collection, storage and retrieval functions is solved. The embodiment of the present invention optimizes the collection and storage operation of the heterogeneous data source, and does not need to customize the development for each heterogeneous data source to achieve the collection and storage functions, that is, there is no need to invasively modify the code, but to realize the collection and storage functions in a general way, and correspondingly also provides a general retrieval function, according to which the general processing method reduces the processing cost, is efficient, and is also conducive to subsequent maintenance and management.

[0168] The data storage device of the heterogeneous data source includes a processor and a memory. The receiving unit, searching unit, collecting unit and storage unit are all stored in the memory as program units. The processor executes the program units stored in the memory to realize corresponding functions.

[0169] The data retrieval device for heterogeneous data sources includes a processor and a memory. The receiving unit and the searching unit are stored in the memory as program units. The processor executes the program units stored in the memory to realize corresponding functions.

[0170] The processor contains a kernel, which retrieves the corresponding program unit from the memory. One or more kernels can be set. By adjusting the kernel parameters, the acquisition, storage and retrieval operations of heterogeneous data sources can be realized in a common way, which optimizes the relevant data processing methods for heterogeneous data sources, reduces data processing costs, improves data processing efficiency, and is also conducive to subsequent maintenance and management.

[0171] An embodiment of the present invention provides a computer-readable storage medium having a computer program stored thereon. When the computer program is executed by a processor, the computer program implements the data storage method for heterogeneous data sources or the data retrieval method for heterogeneous data sources as described above.

[0172] An embodiment of the present invention provides an electronic device, comprising: a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein when the processor executes the computer program, the data storage method for heterogeneous data sources or the data retrieval method for heterogeneous data sources as described above is implemented.

[0173] The present application is described with reference to the flowcharts and / or block diagrams of the methods, devices (systems), and computer program products according to the embodiments of the present application. It should be understood that each process and / or box in the flowchart and / or block diagram, as well as the combination of the processes and / or boxes in the flowchart and / or block diagram, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, a special-purpose computer, an embedded processor, or other programmable data processing device to generate a machine, so that the instructions executed by the processor of the computer or other programmable data processing device generate instructions for implementing the processes in the flowchart and / or block diagram. Figure 1 A process or multiple processes and / or boxes Figure 1 A device that provides the functions specified in a block or multiple blocks.

[0174] In a typical configuration, the device includes one or more processors (CPU), memory and bus. The device may also include input / output interface, network interface and the like.

[0175] The memory may include non-permanent memory in a computer-readable medium, random access memory (RAM) and / or non-volatile memory in the form of read-only memory (ROM) or flash RAM, and the memory includes at least one memory chip. The memory is an example of a computer-readable medium.

[0176] Computer readable media include permanent and non-permanent, removable and non-removable media that can be implemented by any method or technology to store information. Information can be computer readable instructions, data structures, program modules or other data. Examples of computer storage media include, but are not limited to, phase change memory (PRAM), static random access memory (SRAM), dynamic random access memory (DRAM), other types of random access memory (RAM), read-only memory (ROM), electrically erasable programmable read-only memory (EEPROM), flash memory or other memory technology, compact disk read-only memory (CD-ROM), digital versatile disk (DVD) or other optical storage, magnetic cassettes, magnetic tape magnetic disk storage or other magnetic storage devices or any other non-transmission media that can be used to store information that can be accessed by a computing device. As defined herein, computer readable media does not include temporary computer readable media (transitory media), such as modulated data signals and carrier waves.

[0177] It should also be noted that the terms "include", "comprises" or any other variations thereof are intended to cover non-exclusive inclusion, so that a process, method, commodity or device including a series of elements includes not only those elements, but also other elements not explicitly listed, or also includes elements inherent to such process, method, commodity or device. In the absence of more restrictions, the elements defined by the sentence "comprises a ..." do not exclude the existence of other identical elements in the process, method, commodity or device including the elements.

[0178] Those skilled in the art will appreciate that the embodiments of the present application may be provided as methods, systems or computer program products. Therefore, the present application may adopt the form of a complete hardware embodiment, a complete software embodiment or an embodiment in combination with software and hardware. Moreover, the present application may adopt the form of a computer program product implemented on one or more computer-usable storage media (including but not limited to disk storage, CD-ROM, optical storage, etc.) that contain computer-usable program code.

[0179] The above are only embodiments of the present application and are not intended to limit the present application. For those skilled in the art, the present application may have various changes and variations. Any modification, equivalent insertion, improvement, etc. made within the spirit and principle of the present application should be included in the scope of the claims of the present application.

Claims

1. A data storage method for heterogeneous data sources, characterized in that: The method comprises: receiving a data collection instruction, wherein the data collection instruction includes an identifier of a heterogeneous data source; Based on the identifier of the heterogeneous data source, a corresponding heterogeneous data source is found, wherein the heterogeneous data source has an annotation embedded therein, and the annotation is used to configure a data collection method of the heterogeneous data source; According to the annotations embedded in the heterogeneous data sources, a data collection operation is performed to obtain target data; The target data is stored in a column storage database using a preset configuration file, including: obtaining multiple preset attribute fields, multiple storage thresholds and multiple storage addresses from the preset configuration file, wherein each storage threshold is associated with a different storage address; obtaining a storage threshold corresponding to a data volume greater than the target data from the multiple storage thresholds as a target storage threshold; determining a corresponding associated target storage address from the multiple storage addresses according to the target storage threshold; creating a column storage database on a storage space corresponding to the target storage address according to the multiple preset attribute fields; and storing the target data in the column storage database.

2. The method according to claim 1, characterized in that The annotation includes the interface and data conversion format of the heterogeneous data source, and the data collection operation is performed according to the annotation embedded in the heterogeneous data source to obtain the target data, including: Acquire the interface and data conversion format of the heterogeneous data source from the annotation; Collecting data from the heterogeneous data source according to the interface; According to the data conversion format, the collected data is converted into a unified format to obtain target data.

3. The method according to claim 1, characterized in that The method further comprises: Acquire a target attribute field from the plurality of preset attribute fields; Setting the data storage format corresponding to the target attribute field, so as to store data information in the target attribute field in the data storage format; and / or, If there are multiple target attribute fields, the multiple target attribute fields are parsed to obtain the association relationship between the multiple target attribute fields, so as to preferentially store data information in the target attribute fields having the association relationship when storing the target data in the column storage database.

4. The method according to claim 3, characterized in that The storing the target data in the column storage database includes: In the process of storing the target data using the plurality of preset attribute fields, storing corresponding target data in the target attribute fields according to the data storage format; If there are multiple target attribute fields, corresponding target data are stored in the multiple target attribute fields according to the association relationship between the multiple target attribute fields.

5. A data retrieval method for heterogeneous data sources, characterized in that: A column storage database obtained by the data storage method for heterogeneous data sources according to any one of claims 1 to 4, the method comprising: receiving a data retrieval instruction, wherein the data retrieval instruction carries retrieval information; By traversing each attribute column in the column storage database, a search result matching the search information is searched.

6. The method according to claim 5, characterized in that The searching for a search result matching the search information by traversing each attribute column in the column storage database includes: Parsing the search conditions and search keywords from the search information; According to the search keyword, traverse the attribute information under each attribute field in the column storage database one by one to find matching target attribute information; Determine the target attribute field corresponding to the attribute according to the target attribute information; Under the target attribute field, search for search results that match the search condition.

7. A data storage device for heterogeneous data sources, characterized in that: The device comprises: A receiving unit, configured to receive a data collection instruction, wherein the data collection instruction includes an identifier of a heterogeneous data source; A search unit, configured to search for a corresponding heterogeneous data source based on an identifier of the heterogeneous data source, wherein the heterogeneous data source has an embedded annotation, and the annotation is used to configure a data collection method of the heterogeneous data source; A collection unit, configured to perform a data collection operation to obtain target data according to the annotations embedded in the heterogeneous data source; A storage unit, used to store the target data in a column storage database using a preset configuration file; The storage unit includes: an acquisition module, which is used to acquire multiple preset attribute fields, multiple storage thresholds and multiple storage addresses from the preset configuration file, wherein each storage threshold is associated with a different storage address; the acquisition module is also used to acquire the storage threshold corresponding to the data volume greater than the target data from the multiple storage thresholds as the target storage threshold; a determination module, which is used to determine the corresponding associated target storage address from the multiple storage addresses according to the target storage threshold; a creation module, which is used to create a column storage database on the storage space corresponding to the target storage address according to the multiple preset attribute fields; and a storage module, which is used to store the target data in the column storage database.

8. A data retrieval device for heterogeneous data sources, characterized in that: The device comprises: A receiving unit, used to receive a data retrieval instruction, wherein the data retrieval instruction carries retrieval information; A search unit is used to search for search results matching the search information by traversing each attribute column in a column storage database, wherein the column storage database is a database obtained by applying the data storage method for heterogeneous data sources as described in any one of claims 1-4.

9. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores a computer program, and when the computer program is executed by the processor, the method for storing data of heterogeneous data sources according to any one of claims 1 to 4 is implemented; Or when the computer program is executed by a processor, the data retrieval method for heterogeneous data sources as described in claim 5 or 6 is implemented.

10. An electronic device, characterized in that: include: A memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor implements the data storage method for heterogeneous data sources according to any one of claims 1 to 4 when executing the computer program; Or when the computer program is executed by a processor, the data retrieval method for heterogeneous data sources as described in claim 5 or 6 is implemented.

Images