A data access method, device, apparatus and storage medium
Patent Information
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- SAIC GM WULING AUTOMOBILE CO LTD
- Filing Date
- 2026-03-30
- Publication Date
- 2026-07-21
AI Technical Summary
In existing technologies, the different data formats and field meanings of data sources lead to low efficiency, high cost, high error rate, and poor scalability in manual data access, which cannot meet the needs of intelligent access.
By using automated data access methods, data samples from the data source to be accessed are obtained. Based on the data samples, business information and original data structure information are determined, the target data structure information of the target data storage and management system is determined, and the data source is automatically accessed to the target data storage and management system based on data mapping information and transformation rules.
It has enabled automated data access, improved data access efficiency and accuracy, reduced labor costs, and enhanced the flexibility and scalability of data access.
Smart Images
Figure CN122432239A_ABST
Abstract
Description
Technical Field
[0001] This application relates to the field of data processing technology, and more specifically to a method, apparatus, device, and storage medium for data access. Background Technology
[0002] In recent years, with the development of internet big data technology, enterprises and institutions have faced the need to access massive, heterogeneous, and multi-source data. In some technologies, due to the varying data formats and field meanings of data sources, manual intervention is required when connecting these data sources to the data storage and management system. This involves manual operations such as data source identification, data format parsing, field mapping, and interface configuration. However, manual data access is inefficient, costly, error-prone, and lacks scalability, failing to meet the demands of intelligent data access. Summary of the Invention
[0003] In view of this, this application provides a method, apparatus, device, and storage medium for data access, which aims to improve data access efficiency, reduce costs, and enhance accuracy and scalability through automated data access.
[0004] In a first aspect, embodiments of this application provide a data access method, including: acquiring a data sample from a data source to be accessed; Based on the data sample of the data source to be accessed, determine the business information and original data structure information of the data source to be accessed; Based on the business information of the data source to be accessed, the target data storage and management system and the target data structure information of the target data storage and management system are determined. Based on the original data structure information and the target data structure information, data mapping information and conversion rules are determined. Based on the data mapping information and conversion rules, the data source to be connected is connected to the target data storage and management system.
[0005] In one possible implementation of the first aspect, determining the business information and original data structure information of the data source to be accessed based on the data sample of the data source to be accessed includes: Feature extraction is performed on the data samples of the data source to be accessed to obtain data features; the data features include at least structural features and contextual features; the structural features are used to characterize the data structure of the data source to be accessed; the contextual features are used to characterize the association information of the data source to be accessed. Based on the data characteristics, the business information and original data structure information of the data source to be accessed are determined.
[0006] In one possible implementation of the first aspect, determining the business information and original data structure information of the data source to be accessed based on the data characteristics includes: The data features are used as input to the first preset model, and the business information and original data structure information of the data source to be accessed are obtained from the output of the first preset model.
[0007] One possible implementation of the first aspect also includes: The data samples from the data source to be accessed are preprocessed; The step of extracting features from the data samples of the data source to be accessed, and obtaining data features, includes: Feature extraction is performed on the preprocessed data samples from the data source to be accessed to obtain data features.
[0008] In one possible implementation of the first aspect, the original data structure information includes at least the data format information and business meaning information of each field in the data source to be accessed; the target data structure information includes at least the data format information and business meaning information of each field in the target data storage and management system. The process of determining the data mapping information and transformation rules based on the original data structure information and the target data structure information includes: Based on the business meaning information of each field in the original data structure information of the data source to be accessed and the business meaning information of each field in the target data structure information, the data mapping information is determined; Based on the data mapping information, the data format information of each field in the original data structure information, and the data format information of each field in the target data structure information, the conversion rules are determined.
[0009] One possible implementation of the first aspect also includes: Obtain data source information; The data source to be connected is determined based on the data source information.
[0010] One possible implementation of the first aspect also includes: Obtain the access configuration information for data access; The step of connecting the data source to be accessed to the target data storage and management system based on the data mapping information and conversion rules includes: Based on the data mapping information and conversion rules, and in accordance with the data access configuration information, the data source to be accessed is connected to the target data storage and management system.
[0011] Secondly, embodiments of this application provide a data access device, including: The acquisition module is used to acquire data samples from the data source to be connected. The processing module is used to determine the business information and original data structure information of the data source to be accessed based on the data sample of the data source to be accessed; The processing module is further configured to determine the target data storage and management system and the target data structure information of the target data storage and management system based on the business information of the data source to be accessed, and to determine data mapping information and conversion rules based on the original data structure information and the target data structure information; The processing module is also used to connect the data source to be accessed to the target data storage and management system based on the data mapping information and conversion rules.
[0012] Thirdly, embodiments of this application provide an electronic device, including a memory for storing computer program instructions and a processor for executing the program instructions, wherein when the computer program instructions are executed by the processor, the electronic device is triggered to execute the method described in any of the first aspects above.
[0013] Fourthly, embodiments of this application provide a computer-readable storage medium including a stored program, wherein, when the program is executed, it controls the device where the computer-readable storage medium is located to perform the method described in any of the first aspects above.
[0014] The solution provided in this application involves: obtaining a data sample from the data source to be accessed; determining the business information and original data structure information of the data source based on the data sample; determining the target data storage and management system and its target data structure information based on the business information; determining data mapping information and conversion rules based on the original and target data structure information; and connecting the data source to the target data storage and management system based on the data mapping information and conversion rules. In this way, in this application embodiment, a data sample from the data source to be accessed can be obtained first, and the data sample can be analyzed to determine the business information and original data structure information of the data source. This allows for the determination of the target data storage and management system and its target data structure information based on the business information. The data mapping information and conversion rules are determined based on the original and target data structure information, enabling the data from the data source to be accessed to be converted into the target data format of the target data storage and management system and stored in the target data storage and management system. In other words, this embodiment can automatically determine data mapping information and transformation rules based on data samples from the data source to be accessed. This allows for the automatic connection of the data source to the target data storage and management system, reducing manual intervention and achieving automated data access. This improves data access efficiency and accuracy while reducing labor costs. Furthermore, automated data access is more easily applicable to various types of data sources and business scenarios, significantly enhancing the flexibility and scalability of data access. Attached Figure Description
[0015] To more clearly illustrate the technical solutions of the embodiments of this application, the drawings used in the embodiments will be briefly introduced below. Obviously, the drawings described below are only some embodiments of this application. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.
[0016] Figure 1 A flowchart illustrating a data access method provided in an embodiment of this application; Figure 2 A flowchart illustrating another data access method provided in an embodiment of this application; Figure 3 A flowchart illustrating another data access method provided in an embodiment of this application; Figure 4 This application provides a schematic diagram of the structure of a data access device according to an embodiment of the present application. Figure 5This application provides a schematic diagram of the structure of an electronic device. Detailed Implementation
[0017] To better understand the technical solution of this application, the embodiments of this application will be described in detail below with reference to the accompanying drawings.
[0018] It should be understood that the described embodiments are merely some, not all, of the embodiments in this application. All other embodiments obtained by those skilled in the art based on the embodiments in this application without inventive effort are within the scope of protection of this application.
[0019] The terminology used in the embodiments of this application is for the purpose of describing particular embodiments only and is not intended to be limiting of this application. The singular forms “a,” “the,” and “the” used in the embodiments of this application and the appended claims are also intended to include the plural forms unless the context clearly indicates otherwise.
[0020] It should be understood that the term "and / or" used in this article is merely a description of the relationship between related objects, indicating that three relationships can exist. For example, A and / or B can represent: A existing alone, A and B existing simultaneously, or B existing alone. Additionally, the character " / " in this article generally indicates that the preceding and following related objects have an "or" relationship.
[0021] In related technologies, due to the varying data formats and field meanings of data sources, connecting these data sources to data storage and management systems often requires manual intervention. This involves manually identifying data sources, parsing data formats, mapping fields, and configuring interfaces. However, manual data access is inefficient, costly, error-prone, and lacks scalability, failing to meet the demands of intelligent data integration.
[0022] To address the aforementioned issues, this application provides a data access method. The method involves: acquiring a data sample from a data source to be accessed; determining the business information and original data structure information of the data source based on the data sample; determining the target data storage and management system and its target data structure information based on the business information; determining data mapping information and conversion rules based on the original and target data structure information; and connecting the data source to the target data storage and management system based on the data mapping information and conversion rules. In this embodiment, a data sample from the data source to be accessed can be acquired first, and the data sample can be analyzed to determine the business information and original data structure information. This allows for the determination of the target data storage and management system and its target data structure information based on the business information. The data mapping information and conversion rules are then determined based on the original and target data structure information, enabling the data from the data source to be accessed to be converted into the target data format of the target data storage and management system and stored therein. In other words, this embodiment can automatically determine data mapping information and transformation rules based on data samples from the data source to be accessed. This allows for the automatic connection of the data source to the target data storage and management system, reducing manual intervention and achieving automated data access. This improves data access efficiency and accuracy while reducing labor costs. Furthermore, automated data access is more easily applicable to various types of data sources and business scenarios, significantly enhancing the flexibility and scalability of data access. A detailed explanation follows.
[0023] See Figure 1 This is a flowchart illustrating a data access method provided in an embodiment of this application. Figure 1 As shown, the method includes: Step S101: Obtain a data sample from the data source to be connected.
[0024] In this embodiment, when a data source needs to be connected to a data storage and management system, the original data structure information of the data source and the target data information of the data storage and management system must first be determined, thereby determining the data mapping information and conversion rules. Based on this, a data sample of the data source to be connected can be obtained first. In some embodiments, the data source to be connected can be determined by the user. For example, the user can choose which part of the data source to connect to the data storage and management system as the data source to be connected. After determining the data source to be connected, data can be collected from the data source to be connected to obtain a data sample of the data source to be connected.
[0025] In some embodiments, data samples can be collected from the data source to be accessed in a preset manner. For example, the data of the first N records in the data source to be accessed can be extracted as data samples, or N records can be randomly extracted as data samples. Alternatively, the data of N records can be extracted from the data source to be accessed after the data storage time has expired. Of course, data samples from the data source to be accessed can also be obtained through other methods, and this application embodiment does not limit this.
[0026] In some embodiments, when the data source to be accessed is stored in another device, such as a server, the data access device can establish a connection with the server, identify the data source to be accessed on the server, and thus obtain a data sample of the data source to be accessed from the connected server.
[0027] Step S102: Determine the business information and original data structure information of the data source to be accessed based on the data sample of the data source to be accessed.
[0028] In this embodiment, after obtaining a data sample from the data source to be accessed, the data sample can be analyzed. For example, the fields contained in the data sample, the business content of each field, and the data type can be analyzed to determine the business information and original data structure information of the data source to be accessed. The business information of the data source to be accessed can identify the business domain to which the data belongs, such as the financial transaction domain, the user behavior domain, or the Internet of Things sensor domain.
[0029] In some embodiments, to improve the accuracy of the business information and original data structure information of the data source to be accessed, the above-mentioned determination of the business information and original data structure information of the data source to be accessed based on the data sample of the data source to be accessed includes: Feature extraction is performed on the data samples from the data source to be connected, and data features are obtained. Based on the data features, the business information and original data structure information of the data source to be connected are determined.
[0030] Data features include at least structural features and contextual features; structural features are used to characterize the data structure of the data source to be accessed; contextual features are used to characterize the association information of the data source to be accessed.
[0031] In this embodiment, multi-dimensional feature extraction is performed on the data sample from the data source to be accessed. For example, the data sample from the input data source can be analyzed for data format, field data, data type, nesting level, delimiters, etc., to obtain the structural features of the data sample. The data sample from the input data source can also be analyzed for data source, associated metadata, etc., to obtain contextual features, etc. In this way, data features are obtained by acquiring multi-dimensional features.
[0032] After obtaining the data characteristics of the data sample, these characteristics can be analyzed to determine the business information and original data structure information of the data source to be connected. In this way, by extracting features from the data sample across multiple dimensions, the data characteristics of the data sample can be obtained more accurately, allowing for a more precise analysis of the business information and original data structure information of the data source to be connected.
[0033] In some embodiments, the aforementioned data features further include statistical features. That is, when performing multi-dimensional feature extraction on data samples from the data source to be accessed, statistical features can also be obtained by statistically analyzing the distribution, uniqueness, and range of field values of the data samples.
[0034] It should be noted that data features can also include other features, such as semantic features. The type of data features can be set according to the user's actual needs, so that when extracting features from data samples, the required types of features can be obtained to complete the data feature extraction. This application embodiment does not limit the types of features included in the data features.
[0035] In some embodiments, to further improve the accuracy of the business information and original data structure information of the data source to be accessed, reduce manual intervention, and improve automation performance, the above-mentioned determination of the business information and original data structure information of the data source to be accessed based on data characteristics includes: The data features are used as input to the first preset model, and the business information and original data structure information of the data source to be accessed are obtained from the output of the first preset model.
[0036] In other words, a first preset model can be pre-trained. After extracting data features, these features can be used as input values to the first preset model. Upon receiving the data features, the first preset model can perform convolution and other processing on the features, perform semantic understanding on the data samples, and output the business information and original data structure information of the data source to be accessed. In this way, by outputting the business information and original data structure information of the data source to be accessed through the first preset model, complexity is reduced and the possibility of intelligent data access is increased.
[0037] It should be noted that the first preset model can be a deep learning model, such as at least one of the following: convolutional neural network model, recurrent neural network model, and graph neural network model.
[0038] Step S103: Determine the target data storage and management system and the target data structure information of the target data storage and management system based on the business information of the data source to be accessed, and determine the data mapping information and conversion rules based on the original data structure information and the target data structure information.
[0039] In this embodiment, after determining the business information of the data source to be accessed, the business information of the data source can be analyzed to determine the business domain of the data source. Therefore, a data storage and management system that matches the business domain of the data source to be accessed can be selected as the target data storage and management system. Data information from the target data storage and management system is collected and analyzed. For example, the data format of the data stored in the target data storage and management system, the business meaning of each field, etc., can be analyzed to determine the target data structure information of the target data storage and management system. After determining the target data structure information of the target data storage and management system, the correspondence between each field in the data source to be accessed and each field in the target data storage and management system can be determined by analyzing the original data structure information and the target data structure information. The data format differences between each field in the data source to be accessed and the corresponding fields in the target data storage and management system can also be determined. This allows for further determination of the data mapping information and conversion rules between the data source to be accessed and the target data storage and management system.
[0040] The conversion rules characterize the data conversion information when the data source to be connected to the target data storage and management system. For example, they may include format conversion information, unit conversion information, encoding conversion information, aggregation calculation information, and regular expression extraction information.
[0041] In some embodiments, the original data structure information includes at least the data format information and business meaning information of each field in the data source to be accessed; the target data structure information includes at least the data format information and business meaning information of each field in the target data storage and management system.
[0042] Based on the original data structure information and the target data structure information, the data mapping information and transformation rules are determined as follows: Based on the business meaning information of each field in the original data structure information of the data source to be connected and the business meaning information of each field in the target data structure information, the data mapping information is determined.
[0043] Based on the data mapping information, the data format information of each field in the original data structure information, and the data format information of each field in the target data structure information, the conversion rules are determined.
[0044] In other words, the original data structure information can contain the data format and business meaning information of each field in the data source to be accessed. The target data structure information contains the data format and business meaning information of each field in the target data storage and management system. Thus, the data access device can determine, based on the original and target data structure information, which fields in the data source to be accessed have the same business meaning as which fields in the target data storage and management system. Furthermore, based on the business meaning information of each field in the original and target data structure information of the data source to be accessed, a data mapping relationship can be established between each field in the data source to be accessed and the fields with the same business meaning in the target data storage and management system. This allows the device to determine the correspondence between the fields in the data source to be accessed and the fields in the target data storage and management system, thereby obtaining the data mapping information.
[0045] Since there may be differences between the data formats of each field in the original data structure information and the corresponding data formats of the fields in the target data structure information, after determining the data mapping information, the conversion information between the data formats of each field in the original data structure information and the corresponding data formats of the fields in the target data structure information can be determined based on the data mapping information, and conversion rules can be generated.
[0046] In some embodiments, after determining the business meaning information of each field in the original data structure information of the data source to be accessed and the business meaning information of each field in the target data structure information, the semantic similarity between the fields of the original data structure information and the fields of the target data structure information can be calculated using an attention-based matching model or knowledge graph technology, based on the business meaning information of each field in the original data structure information of the data source to be accessed and the business meaning information of each field in the target data structure information. Based on the semantic similarity between the fields of the original data structure information and the fields of the target data structure information, data mapping information is determined. For example, the original data structure information contains data fields 1, 2, and 3. The target data structure information contains data fields a, b, c, and d. To calculate the semantic similarity of field 1 in the original data structure information, we can use the business meaning information of field 1 in the original data structure information and the business meaning information of field a in the target data structure information to calculate the semantic similarity between field 1 in the original data structure information and field a in the target data structure information. If the business meaning information of field 1 in the original data structure information and field a in the target data structure information are different, then the semantic similarity between field 1 in the original data structure information and field a in the target data structure information is calculated to be 0. We can then continue to calculate the semantic similarity between field 1 in the original data structure information and field b in the target data structure information, using the business meaning information of field b in the original data structure information and the business meaning information of field b in the target data structure information. If the business meaning information of field 1 in the original data structure information and field b in the target data structure information are the same, then the semantic similarity between field 1 in the original data structure information and field b in the target data structure information is calculated to be 100%. In this way, we can determine that field 1 in the original data structure information corresponds to field b in the target data structure information, establishing a data mapping relationship. Similarly, the semantic similarity between field 2 in the original data structure information and at least one field in the target data structure information can be calculated. That is, based on the business meaning information of field 2 in the original data structure information and the business meaning information of field a in the target data structure information, the semantic similarity between field 2 in the original data structure information and field a in the target data structure information can be calculated. Assuming that the business meaning information of field 2 in the original data structure information and the business meaning information of field a in the target data structure information are the same, then the semantic similarity between field 1 in the original data structure information and field a in the target data structure information is calculated to be 100%. In this way, it can be determined that field 2 in the original data structure information corresponds to field a in the target data structure information, establishing a data mapping relationship. Similarly, the semantic similarity between field 3 in the original data structure information and at least one field in the target data structure information can be calculated.In other words, based on the business meaning information of field 3 in the original data structure information and the business meaning information of field c in the target data structure information, the semantic similarity between field 3 in the original data structure information and field c in the target data structure information can be calculated. If the business meaning information of field 3 in the original data structure information and field c in the target data structure information are different, then the semantic similarity between field 3 in the original data structure information and field c in the target data structure information is calculated to be 0. Further, based on the business meaning information of field 3 in the original data structure information and field d in the target data structure information, the semantic similarity between field 3 in the original data structure information and field d in the target data structure information can be calculated to be 100%. Thus, it can be determined that field 3 in the original data structure information corresponds to field d in the target data structure information, establishing a data mapping relationship. Based on the established data mapping relationship, the data mapping information is obtained.
[0047] After determining the data mapping information, the differences in data format between each field in the original data structure information and the corresponding field in the target data structure information can be determined based on the data mapping information, and conversion rules can be generated.
[0048] Step S104: Based on the data mapping information and conversion rules, connect the data source to be accessed to the target data storage and management system.
[0049] In this embodiment of the application, after determining the data mapping information and conversion rules, the data in the data source to be accessed can be read. According to the data mapping information and conversion rules, the data of each field in the data to be accessed is converted into the corresponding data format in the target data storage and management system. The converted data of each field in the data source to be accessed is then accessed into the corresponding field in the target data storage and management system.
[0050] In this embodiment, a data sample from the data source to be accessed can be obtained first. This sample is then analyzed to determine the business information and original data structure of the data source. Based on the business information, the target data storage and management system and its target data structure can be determined. Data mapping information and conversion rules are then determined based on the original and target data structure information. This allows the data from the data source to be accessed to be converted to the target data format of the target data storage and management system and stored there. In other words, this embodiment automatically determines the data mapping information and conversion rules based on the data sample from the data source to be accessed. This allows for the automatic access of the data source to the target data storage and management system, reducing manual intervention and achieving automated data access. This improves data access efficiency and accuracy while reducing labor costs. Furthermore, automated data access is more easily applicable to various types of data sources and business scenarios, significantly improving the flexibility and scalability of data access.
[0051] In some embodiments, such as Figure 2 As shown, the above method also includes: Step S105: Obtain the access configuration information for data access.
[0052] In other words, after determining the data mapping information and transformation rules, it is necessary to connect the data from the data source to the target data storage and management system. At this point, the access configuration information can be obtained. This access configuration information records how to connect the data from the data source to the target data storage and management system. For example, the access configuration information records the data loading method of the data source to be connected, such as incremental loading or full loading, extraction frequency, error handling strategy information, etc.
[0053] In some embodiments, the access configuration information for data access can be preset by the user according to actual needs, and the data access device can obtain the access configuration information for data access from a storage device or a cloud server. In other embodiments, the access configuration information may not be preset, but may be set by the user in the data access device when needed. In this case, the data access device receives the access configuration information for data access set by the user. Of course, the data access device can also obtain the access configuration information for data access through other means, and the embodiments of this application do not limit this.
[0054] The above step S104, based on data mapping information and conversion rules, connects the data source to be accessed to the target data storage and management system, including: Based on data mapping information and conversion rules, and in accordance with the data access configuration information, the data source to be accessed is connected to the target data storage and management system.
[0055] In other words, after obtaining the data mapping information and conversion rules, the access configuration information for data access can be further obtained. After obtaining the access configuration information for data access, the data to be accessed can be obtained from the data source to be accessed according to the access configuration information, and the obtained data to be accessed can be accessed into the target data storage and management system according to the data mapping information and conversion rules.
[0056] In some embodiments, in order to improve the accuracy of data access, before extracting features from the data samples of the data source to be accessed and obtaining data features in the above steps, the method further includes: preprocessing the data samples of the data source to be accessed.
[0057] In other words, after obtaining data samples from the data source to be connected, the data samples can be preprocessed, such as cleaning, denoising, and standardizing, to make the data samples more accurate for subsequent feature extraction.
[0058] The above steps involve extracting features from the data samples of the data source to be accessed, including: extracting features from the preprocessed data samples of the data source to be accessed, and obtaining data features.
[0059] In other words, after preprocessing the data samples from the data source to be accessed, feature extraction can be performed on the preprocessed data samples to obtain data features.
[0060] In some embodiments, such as Figure 3 As shown, the above method also includes: Step S106: Obtain data source information.
[0061] In this embodiment, when data from a data source needs to be connected to the data storage and management system, user-inputted data source information can be received. This information determines which data source needs to be connected, and that data source is designated as the data source to be connected. In other words, the data source information identifies the data source to which the data needs to be connected to the data storage and management system. For example, the data source information could be the IP (Internet Protocol) address, URL (Uniform Resource Locator), file path, or keywords of the device where the data source is located.
[0062] In some embodiments, the data source information may also be a data source scan range preset by the user.
[0063] Step S107: Determine the data source to be connected based on the data source information.
[0064] In this embodiment, after obtaining the data source information, the data source to be accessed can be located based on the data source information. In some embodiments, the data access device can send the data source information to the data source server. The data source server receives the data source information and determines the data source to be accessed based on the data source information. The data source server then sends the information of the data source to be accessed to the data access device so that the data access device can obtain data samples from the data source to be accessed.
[0065] Figure 4 This is a schematic diagram of the structure of a data access device provided in an embodiment of this application. Figure 4 As shown, the data access device includes: The acquisition module 401 is used to acquire data samples from the data source to be connected.
[0066] The processing module 402 is used to determine the business information and original data structure information of the data source to be accessed based on the data sample of the data source to be accessed.
[0067] The processing module 402 is also used to determine the target data storage and management system and the target data structure information of the target data storage and management system based on the business information of the data source to be accessed, and to determine the data mapping information and conversion rules based on the original data structure information and the target data structure information.
[0068] The processing module 402 is also used to connect the data source to be connected to the target data storage and management system based on data mapping information and conversion rules.
[0069] In some embodiments, the processing module 402 is specifically used to extract features from data samples of the data source to be accessed, obtain data features, and determine the business information and original data structure information of the data source to be accessed based on the data features.
[0070] The data features include at least structural features and contextual features; structural features are used to characterize the data structure of the data source to be accessed; contextual features are used to characterize the association information of the data source to be accessed.
[0071] In some embodiments, the processing module 402 is specifically used to input data features as input to the first preset model, and obtain the business information and original data structure information of the data source to be accessed output by the first preset model.
[0072] In some embodiments, the processing module 402 is further configured to preprocess data samples from the data source to be accessed.
[0073] The processing module 402 is specifically used to extract features from the preprocessed data samples of the data source to be accessed, and to obtain data features.
[0074] In some embodiments, the original data structure information includes at least the data format information and business meaning information of each field in the data source to be accessed; the target data structure information includes at least the data format information and business meaning information of each field in the target data storage and management system.
[0075] The processing module 402 is specifically used to determine the data mapping information based on the business meaning information of each field in the original data structure information of the data source to be accessed and the business meaning information of each field in the target data structure information.
[0076] Based on the data mapping information, the data format information of each field in the original data structure information, and the data format information of each field in the target data structure information, the conversion rules are determined.
[0077] In some embodiments, the acquisition module 401 is also used to acquire data source information.
[0078] The processing module 402 is also used to determine the data source to be accessed based on the data source information.
[0079] In some embodiments, the acquisition module 401 is further configured to acquire access configuration information for data access.
[0080] The processing module 402 is specifically used to connect the data source to be accessed to the target data storage and management system based on data mapping information and conversion rules, and according to the access configuration information of data access.
[0081] Corresponding to the above embodiments, this application also provides an electronic device. Figure 5 This is a schematic diagram of the structure of an electronic device provided in an embodiment of the present invention. The electronic device 500 may include a processor 501, a memory 502, and a communication unit 503. These components communicate through one or more buses. Those skilled in the art will understand that the structure of the server shown in the figure does not constitute a limitation on the embodiment of the present invention. It may be a bus topology or a star topology, and may include more or fewer components than shown, or combine certain components, or have different component arrangements.
[0082] The communication unit 503 is used to establish a communication channel, enabling the storage device to communicate with other devices. It receives user data from other devices or sends user data to other devices.
[0083] The processor 501 serves as the control center of the storage device, connecting various parts of the electronic device via various interfaces and lines. It executes software programs and / or modules stored in the memory 502, and calls data stored in the memory to perform various functions of the electronic device and / or process data. The processor can be composed of integrated circuits (ICs), such as a single packaged IC or multiple packaged ICs with the same or different functions connected together. For example, the processor 501 may consist only of a central processing unit (CPU). In this embodiment of the invention, the CPU may have a single processing core or include multiple processing cores.
[0084] The memory 502 is used to store the execution instructions of the processor 501. The memory 502 can be implemented by any type of volatile or non-volatile storage device or a combination thereof, such as static random access memory (SRAM), electrically erasable programmable read-only memory (EEPROM), erasable programmable read-only memory (EPROM), programmable read-only memory (PROM), read-only memory (ROM), magnetic storage, flash memory, magnetic disk or optical disk.
[0085] When the execution instructions in memory 502 are executed by processor 501, the electronic device 500 is able to perform some or all of the steps in the above embodiments.
[0086] In a specific implementation, the present invention also provides a computer storage medium, wherein the computer storage medium may store a program, and the program, when executed, may include some or all of the steps of the data access method provided by the present invention in various embodiments. The storage medium may be a magnetic disk, optical disk, read-only memory (ROM), or random access memory (RAM), etc.
[0087] Those skilled in the art will clearly understand that the techniques in the embodiments of the present invention can be implemented using software plus necessary general-purpose hardware platforms. Based on this understanding, the technical solutions in the embodiments of the present invention, or the parts that contribute to the prior art, can be embodied in the form of a software product. This computer software product can be stored in a storage medium, such as ROM / RAM, magnetic disk, optical disk, etc., and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute the methods described in various embodiments or certain parts of the embodiments of the present invention.
[0088] The same or similar parts between the various embodiments in this specification can be referred to mutually. In particular, the device embodiments and terminal embodiments are basically similar to the method embodiments, so the description is relatively simple, and the relevant parts can be referred to the description in the method embodiments.
Claims
1. A data access method, characterized in that, include: Obtain data samples from the data source to be connected; Based on the data sample of the data source to be accessed, determine the business information and original data structure information of the data source to be accessed; Based on the business information of the data source to be accessed, the target data storage and management system and the target data structure information of the target data storage and management system are determined. Based on the original data structure information and the target data structure information, data mapping information and conversion rules are determined. Based on the data mapping information and conversion rules, the data source to be connected is connected to the target data storage and management system.
2. The method according to claim 1, characterized in that, The process of determining the business information and original data structure information of the data source to be accessed based on the data sample of the data source to be accessed includes: Feature extraction is performed on the data samples of the data source to be accessed to obtain data features; the data features include at least structural features and contextual features; the structural features are used to characterize the data structure of the data source to be accessed; the contextual features are used to characterize the association information of the data source to be accessed. Based on the data characteristics, the business information and original data structure information of the data source to be accessed are determined.
3. The method according to claim 2, characterized in that, The process of determining the business information and original data structure information of the data source to be accessed based on the data characteristics includes: The data features are used as input to the first preset model, and the business information and original data structure information of the data source to be accessed are obtained from the output of the first preset model.
4. The method according to claim 2 or 3, characterized in that, Also includes: The data samples from the data source to be accessed are preprocessed; The step of extracting features from the data samples of the data source to be accessed, and obtaining data features, includes: Feature extraction is performed on the preprocessed data samples from the data source to be accessed to obtain data features.
5. The method according to claim 1, characterized in that, The original data structure information includes at least the data format information and business meaning information of each field in the data source to be accessed; the target data structure information includes at least the data format information and business meaning information of each field in the target data storage and management system. The process of determining the data mapping information and transformation rules based on the original data structure information and the target data structure information includes: Based on the business meaning information of each field in the original data structure information of the data source to be accessed and the business meaning information of each field in the target data structure information, the data mapping information is determined; Based on the data mapping information, the data format information of each field in the original data structure information, and the data format information of each field in the target data structure information, the conversion rules are determined.
6. The method according to claim 1, characterized in that, Also includes: Obtain data source information; The data source to be connected is determined based on the data source information.
7. The method according to claim 1, characterized in that, Also includes: Obtain the access configuration information for data access; The step of connecting the data source to be accessed to the target data storage and management system based on the data mapping information and conversion rules includes: Based on the data mapping information and conversion rules, and in accordance with the data access configuration information, the data source to be accessed is connected to the target data storage and management system.
8. A data access device, characterized in that, include: The acquisition module is used to acquire data samples from the data source to be connected. The processing module is used to determine the business information and original data structure information of the data source to be accessed based on the data sample of the data source to be accessed; The processing module is further configured to determine the target data storage and management system and the target data structure information of the target data storage and management system based on the business information of the data source to be accessed, and to determine data mapping information and conversion rules based on the original data structure information and the target data structure information; The processing module is also used to connect the data source to be accessed to the target data storage and management system based on the data mapping information and conversion rules.
9. An electronic device, characterized in that, The device includes a memory for storing computer program instructions and a processor for executing the program instructions, wherein when the computer program instructions are executed by the processor, the device performs the method according to any one of claims 1 to 7.
10. A computer-readable storage medium, characterized in that, The computer-readable storage medium includes a stored program, wherein, when the program is executed, it controls the device on which the computer-readable storage medium is located to perform the method according to any one of claims 1 to 7.