Data management method, apparatus, and storage medium

Abstract

The application provides a data management method and device and a storage medium, relates to the technical field of computers, and aims to solve the technical problem of low standardization degree caused by the difficulty of unified management of data by general technology. The method comprises the following steps: receiving a data collection request for requesting collection of initial business data; the data collection request comprises data source information, storage configuration information and metadata generation rules of the initial business data; the initial business data is mapped into target business data based on the data source information and the storage configuration information; and metadata information for describing the target business data is generated according to the metadata generation rules.

Description

Technical Field

[0001] This application belongs to the field of computer technology, and in particular relates to a data management method, apparatus and storage medium. Background Technology

[0002] Currently, big data technology is being applied more and more widely. Users can manage different types of data through big data technology.

[0003] In general technologies, when managing different types of data, data in different data storage media is usually separated without using unified metadata for management. This requires users to fill in more information related to data storage media when configuring various big data services, increasing the complexity of using the big data platform and easily causing the big data platform to lack unified management of data, resulting in a low degree of standardization. Summary of the Invention

[0004] This application provides a data management method, apparatus, and storage medium to solve the technical problem that general technologies are difficult to use for unified data management, resulting in a low degree of standardization.

[0005] To achieve the above objectives, this application adopts the following technical solution:

[0006] Firstly, a data management method is provided, comprising: receiving a data collection request for requesting the collection of initial business data; the data collection request includes data source information, storage configuration information, and metadata generation rules for the initial business data; mapping the initial business data to target business data based on the data source information and storage configuration information; and generating metadata information describing the target business data according to the metadata generation rules.

[0007] Optionally, the data source information is used to indicate the data source to which the initial business data belongs and the first storage medium corresponding to the data source; the storage configuration information is used to indicate the second storage medium corresponding to the target business data; based on the data source information and the storage configuration information, mapping the initial business data to the target business data includes: calling the data reading service corresponding to the first storage medium to read the initial business data from the data source; determining the target mapping table corresponding to the first storage medium and the second storage medium in a preset mapping information set; the target mapping table is used to map the data in the first storage medium to the data in the second storage medium; the preset mapping information set includes multiple mapping tables that correspond one-to-one with multiple storage medium combinations; calling the data writing service corresponding to the second storage medium to write the initial business data into the second storage medium according to the target mapping table to obtain the target business data.

[0008] Optionally, the data management method further includes: receiving an information update request sent by an electronic device; the information update request is used to request an update of the data source information and storage configuration information; and updating the data source information and storage configuration information in response to the information update request.

[0009] Optionally, the data management method further includes: receiving a data processing request from an electronic device; the data processing request is used to request data processing of target business data; the data processing request includes data processing rules; and processing the target business data according to the data processing rules to obtain a processing result.

[0010] In a second aspect, a data management device is provided, comprising: a receiving unit and a processing unit;

[0011] The receiving unit is used to receive a data collection request for requesting the collection of initial business data; the data collection request includes the data source information, storage configuration information and metadata generation rules of the initial business data.

[0012] The processing unit is used to map initial business data to target business data based on data source information and storage configuration information;

[0013] The processing unit is also used to generate metadata information that describes the target business data according to the metadata generation rules.

[0014] Optionally, the data source information is used to indicate the data source to which the initial business data belongs and the first storage medium corresponding to the data source; the storage configuration information is used to indicate the second storage medium corresponding to the target business data; the processing unit is specifically used for:

[0015] Invoke the data read service corresponding to the first storage medium to read the initial business data from the data source;

[0016] In a preset mapping information set, a target mapping table corresponding to the first storage medium and the second storage medium is determined; the target mapping table is used to map data in the first storage medium to data in the second storage medium; the preset mapping information set includes multiple mapping tables that correspond one-to-one with multiple combinations of storage media.

[0017] Invoke the data write service corresponding to the second storage medium, and write the initial business data into the second storage medium according to the target mapping table to obtain the target business data.

[0018] Optionally, the receiving unit is also configured to receive an information update request sent by an electronic device; the information update request is used to request updates to the data source information and storage configuration information;

[0019] The processing unit is also used to update data source information and storage configuration information in response to information update requests.

[0020] Optionally, the receiving unit is further configured to receive a data processing request from an electronic device; the data processing request is used to request data processing of target business data; the data processing request includes data processing rules;

[0021] The processing unit is also used to process the target business data according to the data processing rules to obtain the processing results.

[0022] Thirdly, a data management device is provided, including a memory and a processor; the memory is used to store computer-executed instructions, and the processor is connected to the memory via a bus; when the data management device is running, the processor executes the computer-executed instructions stored in the memory to cause the data management device to perform the data management method as described in the first aspect.

[0023] The data management device can be a network device or a component of a network device, such as a chip system within the network device. The chip system supports the network device in implementing the functions involved in the first aspect and any of its possible implementations, such as receiving, determining, and routing the data and / or information involved in the aforementioned data management method. The chip system includes a chip, but may also include other discrete devices or circuit structures.

[0024] Fourthly, a computer-readable storage medium is provided, including computer-executable instructions that, when executed on a computer, cause the computer to perform the data management method of the first aspect.

[0025] It should be noted that the aforementioned computer instructions may be stored, in whole or in part, on the first computer-readable storage medium. The first computer-readable storage medium may be packaged together with the processor of the data management device, or it may be packaged separately from the processor of the data management device; this application does not impose any limitations on this.

[0026] In this application, the name of the aforementioned data management device does not limit the device or functional module itself. In actual implementation, these devices or functional modules may appear under other names. As long as the function of each device or functional module is similar to that of this application, it falls within the scope of the claims of this application and its equivalents.

[0027] These or other aspects of this application will become more readily apparent in the following description.

[0028] The technical solution provided in this application brings at least the following beneficial effects:

[0029] Based on any of the foregoing aspects, in this application, the data management device can receive a data acquisition request for requesting the collection of initial business data. The data acquisition request includes data source information, storage configuration information, and metadata generation rules for the initial business data. Furthermore, the data management device can map the initial business data to target business data based on the data source information and storage configuration information, and further generate metadata information describing the target business data according to the metadata generation rules.

[0030] Compared to general technologies that use different metadata to manage different types of data, this application supports user-defined metadata generation rules. This enables unified management of different data based on the same metadata format, reducing the complexity of big data platforms and avoiding the low standardization problem often caused by a lack of unified data management. Therefore, this application can solve the technical problem of low standardization caused by the difficulty of unified data management in general technologies. Attached Figure Description

[0031] Figure 1 This is a schematic diagram of the structure of a data management system provided in an embodiment of this application;

[0032] Figure 2 A schematic diagram of the structure of a data management device provided in this application embodiment. Figure 1 ;

[0033] Figure 3 A schematic diagram of the hardware structure of a data management device provided in an embodiment of this application;

[0034] Figure 4 A flowchart illustrating a data management method provided in this application embodiment. Figure 1 ;

[0035] Figure 5 A flowchart illustrating a data management method provided in this application embodiment. Figure 2 ;

[0036] Figure 6 A flowchart illustrating a data management method provided in this application embodiment. Figure 3 ;

[0037] Figure 7 A flowchart illustrating a data management method provided in this application embodiment. Figure 4 ;

[0038] Figure 8 A schematic diagram of a data acquisition process provided in an embodiment of this application;

[0039] Figure 9A schematic diagram of a data processing flow provided in an embodiment of this application;

[0040] Figure 10 A schematic diagram of the structure of a data management device provided in this application embodiment. Figure 2 . Detailed Implementation

[0041] The technical solutions of the embodiments of this application will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of this application, and not all embodiments. Based on the embodiments of this application, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of this application.

[0042] It should be noted that in the embodiments of this application, the words "exemplary" or "for example" are used to indicate examples, illustrations, or explanations. Any embodiment or design scheme described as "exemplary" or "for example" in the embodiments of this application should not be construed as being more preferred or advantageous than other embodiments or design schemes. Specifically, the use of the words "exemplary" or "for example" is intended to present the relevant concepts in a specific manner.

[0043] To facilitate a clear description of the technical solutions of the embodiments of this application, the terms "first" and "second" are used in the embodiments of this application to distinguish the same or similar items with essentially the same function and effect. Those skilled in the art can understand that the terms "first" and "second" are not intended to limit the quantity or execution order.

[0044] Furthermore, the terms "comprising" and "having" in the embodiments, claims, and drawings of this application are not exclusive. For example, a process, method, system, product, or device that includes a series of steps or modules is not limited to the listed steps or modules, but may also include steps or modules not listed.

[0045] To facilitate understanding of this application, the relevant elements involved in this application are described below.

[0046] Currently, big data technology is being applied more and more widely. Users can manage different types of data through big data technology.

[0047] In general technologies, when managing different types of data, data in different data storage media is usually separated without using unified metadata for management. This requires users to fill in more information related to data storage media when configuring various big data services, increasing the complexity of using the big data platform and easily causing the big data platform to lack unified management of data, resulting in a low degree of standardization.

[0048] To address the aforementioned issues, this application provides a data management method. In this method, a data management device can receive a data collection request for initial business data. The data collection request includes data source information, storage configuration information, and metadata generation rules for the initial business data. Furthermore, the data management device can map the initial business data to target business data based on the data source information and storage configuration information, and further generate metadata information describing the target business data according to the metadata generation rules.

[0049] Compared to general technologies that use different metadata to manage different types of data, this application supports user-defined metadata generation rules. This enables unified management of different data based on the same metadata format, reducing the complexity of big data platforms and avoiding the low standardization problem often caused by a lack of unified data management. Therefore, this application can solve the technical problem of low standardization caused by the difficulty of unified data management in general technologies.

[0050] This data management method is applicable to data management systems. Figure 1 One structure of the data management system 100 is shown. For example... Figure 1 As shown, the data management system 100 includes an electronic device 101 and a data management device 102. The electronic device 101 and the data management device 102 can be communicatively connected.

[0051] In practical applications, the data management device 102 can also communicate with multiple electronic devices 101.

[0052] For ease of understanding, this application uses the communication connection between a data management device 102 and an electronic device 101 as an example for illustration.

[0053] Figure 1 The electronic device 101 can be configured with a client of a big data service platform to support users in configuring data acquisition and data processing tasks.

[0054] Optionally, Figure 1The electronic device 101 in the document can be a terminal. This terminal can be a device providing voice and / or data connectivity to a user, a handheld device with wireless connectivity, or other processing devices connected to a wireless modem. The wireless terminal can communicate with one or more core networks via a radio access network (RAN). The wireless terminal can be a mobile terminal, such as a computer with a mobile terminal, or a portable, pocket-sized, handheld, or computer-embedded mobile device that exchanges voice and / or data with the radio access network, such as a mobile phone, tablet computer, laptop computer, netbook, or personal digital assistant (PDA). This application does not impose any limitations on this.

[0055] Figure 1 The data management device 102 can be the server side of a big data service platform, and can be used to provide various data and services. For example, the data management device 102 can be used to provide various data management services such as data collection, transmission and transformation.

[0056] Optionally, Figure 1 The data management device 102 can be configured with a storage medium. The storage medium can be used to store descriptive information of the data source and data assets, as well as task information for data acquisition and data processing.

[0057] Optionally, Figure 1 The data management device 102 can be a server. This server can be a single server, or a server cluster consisting of multiple servers. In some embodiments, the server cluster can also be a distributed cluster. This application does not impose any limitations in this regard.

[0058] Combination Figure 1 ,like Figure 2 The diagram shown is a structural schematic of a data management device 102 provided in an embodiment of this application. The data management device 102 may be configured with a data acquisition module, a data processing module, a data mapping module, an input adapter module, an output adapter module, and a data source and data asset management module.

[0059] The data acquisition module can be used to collect data from various data sources. The data processing module can be used to process the various types of data collected by the data acquisition module.

[0060] The data mapping module can be configured with schema mapping tables between various storage media. During the transfer of data between data assets or between data sources and data assets, the schema mapping table can be used to provide the correspondence between table structures across different storage media. For example, the schema mapping table can include the mapping between database table concepts and message queue topic concepts, and provide schema naming rules for data persistence.

[0061] The input adapter module can be used to match different types of data sources and to match different acquisition methods to different data sources. Furthermore, the input adapter module allows users to set the storage medium of the data source. Subsequently, various big data tasks on this data source do not need to concern themselves with the storage medium type of the data source.

[0062] The output adapter module can be used to match different types of data assets. It can map the input table structure to the data asset's table structure according to the data asset's storage medium type and a schema mapping table, completing the data writing process. Furthermore, the output adapter module allows users to set the storage medium type of the data asset. After the data acquisition task is completed, the acquired data will be stored in the output data asset.

[0063] The data source and data asset management module can be equipped with multiple data assets corresponding to multiple different types of data sources to support data collection for different data sources.

[0064] like Figure 3 The diagram shown illustrates a hardware structure of a data management device 102 provided in an embodiment of this application. The data management device 102 includes a processor 21, a memory 22, a communication interface 23, and a bus 24. The processor 21, memory 22, and communication interface 23 are connected via the bus 24.

[0065] Processor 21 is the control center of data management device 102. It can be a single processor or a collective term for multiple processing elements. For example, processor 21 can be a CPU or other general-purpose processors. Among them, general-purpose processors can be microprocessors or any conventional processors.

[0066] As one embodiment, processor 21 may include one or more CPUs, for example Figure 3 CPU0 and CPU1 are shown in the diagram.

[0067] The memory 22 may be a read-only memory (ROM) or other type of static storage device capable of storing static information and instructions, random access memory (RAM) or other type of dynamic storage device capable of storing information and instructions, or electrically erasable programmable read-only memory (EEPROM), disk storage media or other magnetic storage devices, or any other medium capable of carrying or storing desired program code in the form of instructions or data structures and accessible by a computer, but is not limited thereto.

[0068] In one possible implementation, the memory 22 can exist independently of the processor 21. The memory 22 can be connected to the processor 21 via a bus 24 and is used to store instructions or program code. When the processor 21 calls and executes the instructions or program code stored in the memory 22, it can implement the data management method provided in the following embodiments of this application.

[0069] In another possible implementation, the memory 22 can also be integrated with the processor 21.

[0070] The communication interface 23 is used for the data management device 102 to connect with other devices via a communication network, which may be Ethernet, wireless access network, wireless local area network (WLAN), etc. The communication interface 23 may include a receiving unit for receiving data and a sending unit for sending data.

[0071] Bus 24 can be an industry standard architecture (ISA) bus, a peripheral component interconnect (PCI) bus, or an extended industry standard architecture (EISA) bus, etc. This bus can be divided into address bus, data bus, control bus, etc. For ease of representation, Figure 3 The bus is represented by a single thick line, but this does not mean that there is only one bus or one type of bus.

[0072] It should be pointed out that, Figure 3 The structure shown does not constitute a limitation on the data management device 102, except... Figure 3In addition to the components shown, the data management device 102 may include more or fewer components than shown, or combine certain components, or have different component arrangements.

[0073] like Figure 4 The diagram shown is a flowchart illustrating a data management method provided in an embodiment of this application. This data management method is applied to... Figure 1 The data management device 102 in the data management system 100 shown. The data management method includes: S401-S403.

[0074] S401, The data management device receives a data collection request for requesting the collection of initial business data.

[0075] The data collection request includes the data source information, storage configuration information, and metadata generation rules for the initial business data.

[0076] Optionally, the initial business data can be business data about a specific business (such as food delivery business and logistics business).

[0077] In one possible approach, the data source information of the initial business data can be used to indicate the data source to which the initial business data belongs and the first storage medium corresponding to the data source.

[0078] In one possible approach, storage configuration information can be used to represent the second storage medium corresponding to the target business data. The storage configuration information may include a data naming scheme. This scheme can be used to generate data names for the target business data, enabling the target business data to be stored in the data management device as a data asset.

[0079] In one possible approach, the metadata generation rules can include information such as the order of various descriptive information and character encoding rules, which are used to represent the generation method of the metadata of the target business data.

[0080] In one possible approach, combining Figure 1 When the electronic device is used as a terminal, it can be configured with functional modules such as an input module and a display module. Based on this, users can edit data source information, storage configuration information, and metadata generation rules through the input module configured on the terminal, and perform data acquisition operations (e.g., clicking the acquisition button). In response to this data acquisition operation, the terminal can send a data acquisition request carrying data source information, storage configuration information, and metadata generation rules to the data management device.

[0081] Correspondingly, the data management device can receive data collection requests from the terminal, parse the data collection requests to determine the data source information, storage configuration information and metadata generation rules, and can further store the data source information and storage configuration information in the configured storage medium.

[0082] S402. The data management device maps the initial business data to the target business data based on the data source information and storage configuration information.

[0083] In one possible approach, the data management device can invoke the data read service corresponding to the first storage medium to read the initial business data from the data source to which the initial business data belongs. Then, the data management device can invoke the data write service corresponding to the second storage medium to write the initial business data into the second storage medium, thereby obtaining the target business data.

[0084] S403. The data management device generates metadata information to describe the target business data according to the metadata generation rules.

[0085] In one possible approach, to facilitate unified management of various types of data, the data management device can generate metadata information describing the target business data according to metadata generation rules, thereby enabling unified management of all initial business data from the data source based on metadata in a unified format.

[0086] In one embodiment, combined with Figure 4 In the aforementioned S402, when the data management device maps the initial business data to the target business data based on the data source information and storage configuration information, such as... Figure 5 As shown, this application embodiment provides an optional implementation method, including: S501-S503.

[0087] S501, The data management device calls the data reading service corresponding to the first storage medium to read the initial business data from the data source.

[0088] In one possible way, combining Figure 2 The data management device can match the data read service corresponding to the first storage medium through the configured input adapter module. Then, the data management device can call the data read service corresponding to the first storage medium to read the initial business data from the data source to which the initial business data belongs.

[0089] S502, The data management device determines the target mapping table corresponding to the first storage medium and the second storage medium from the preset mapping information set.

[0090] The target mapping table is used to map data in the first storage medium to data in the second storage medium. The preset mapping information set includes multiple mapping tables that correspond one-to-one with multiple combinations of storage media.

[0091] In one possible way, combining Figure 2The data management device is equipped with a data mapping module that can store a preset mapping information set. Based on this, the data management device can use the data mapping module to match the mapping table corresponding to the storage medium combination consisting of the first storage medium and the second storage medium from the preset mapping information set. Then, the data management device can determine the mapping table corresponding to the storage medium combination consisting of the first storage medium and the second storage medium as the target mapping table corresponding to the first storage medium and the second storage medium.

[0092] S503, the data management device calls the data writing service corresponding to the second storage medium, and writes the initial business data into the second storage medium according to the target mapping table to obtain the target business data.

[0093] In one possible way, combining Figure 2 The data management device can match the data write service corresponding to the second storage medium through the configured output adapter module. Then, the data management device can call the data write service corresponding to the second storage medium, map the initial business data according to the target mapping table, and write it into the second storage medium to obtain the target business data.

[0094] In one embodiment, after the storage of the target business data is completed, such as Figure 6 As shown, the data management method provided in this application embodiment further includes: S601-S602.

[0095] S601, The data management device receives an information update request sent by the electronic device.

[0096] Information update requests can be used to request updates to data source information and storage configuration information. For example, an information update request can be used to request modifications, deletions, and queries of data source information and storage configuration information. Alternatively, an information update request can be used to request the addition of new data source information and storage configuration information.

[0097] In one possible approach, combining Figure 1 When an electronic device functions as a terminal, it can be configured with functional modules such as an input module and a display module. Based on this, the user can perform information update operations (e.g., clicking an information update button) through the input module configured on the terminal. In response to this information update operation, the terminal can send an information update request to the data management device. Correspondingly, the data management device can receive the information update request from the terminal.

[0098] S602, The data management device responds to the information update request by updating the data source information and storage configuration information.

[0099] In one possible approach, after receiving an information update request from the terminal, the data management device can respond to the information update request by updating the data source information and storage configuration information.

[0100] In one embodiment, after the storage of the target business data is completed, such as Figure 7 As shown, the data management method provided in this application embodiment further includes: S701-S702.

[0101] S701, The data management device receives a data processing request from an electronic device.

[0102] A data processing request can be used to request data processing of target business data. A data processing request can include data processing rules. For example, data processing could be merging different items in the target business data, or adding identical items within the target business data.

[0103] In one possible approach, combining Figure 1 When the electronic device is a terminal, it can be configured with functional modules such as an input module and a display module. Based on this, users can edit data processing rules and other information through the input module configured on the terminal, and perform data processing operations (e.g., clicking a data processing button). In response to this data processing operation, the terminal can send a data processing request to the data management device. Correspondingly, the data management device can receive the data processing request from the terminal and parse it to determine the data processing rules.

[0104] S702. The data management device processes the target business data according to the data processing rules to obtain the processing result.

[0105] In one possible approach, after determining the data processing rules, the data management device can process the target business data according to the data processing rules to obtain the processing results.

[0106] In one embodiment, such as Figure 8 The diagram illustrates a data acquisition process according to an embodiment of this application. When setting data source information, storage configuration information, and metadata generation rules in the storage medium configured on the data management device, the user can also set data acquisition rules. Data acquisition rules can include parameters such as acquisition time, acquisition type, incrementing column, time column, database primary key, and acquisition period. Based on this, the data management device can use these data acquisition rules to generate data acquisition tasks, thereby acquiring data stored on data sources of different storage medium types into data assets of any storage medium type.

[0107] The data source information can include input data sources. Input data sources can represent the source of the initial business data. Storage configuration information can include output assets. Output assets can represent the destination of the initial business data. Incrementing columns can be fields used to indicate the time when the data was entered into the database. Time columns can be fields in the data used to represent time.

[0108] Data collection types can be categorized as incremental collection or full collection, allowing users to configure whether a data collection task will completely overwrite the data from the data source into the data asset, or incrementally collect data from the data source that is greater than the current value of the incrementing column. Collection time indicates the user's desired start time for the data collection task. The collection period (also known as the collection accounting period) can be used to achieve table partitioning. For example, if the preset collection period dimension is one month and the number of periods is 12, then when executing a data collection task, the data management device can partition the target business data into 12 tables according to the month of the selected time column.

[0109] Next, users can update the entered data source information and storage configuration information as needed. Furthermore, users can set up new data collection tasks based on the entered data source information, storage configuration information, and data collection rule parameters. Then, the data management device can determine in real time whether the current time is a collection period.

[0110] If so, the data management device can initiate a data acquisition task. Through the configured input and output adapter modules, it selects different data read and write services based on the storage medium type and data source information and storage configuration information. Then, the data management device can map the acquired initial business data to the target business data according to the schema mapping relationship of different data storage.

[0111] In one embodiment, such as Figure 9 The diagram illustrates a data processing flow according to an embodiment of this application. After initial business data from different data sources are mapped to target business data, each target business data can be considered a data asset. Users can set data processing rules in the storage medium configured on the data management device. Based on the data processing rules, users can set data processing tasks to achieve data transfer between different data assets.

[0112] The data processing rules can include parameters such as input data assets, output data assets, billing cycle offset, scheduling time, and scheduling interval. Input data assets represent the source of data assets in a data processing task. Output data assets represent the source of data assets in a data processing task. Billing cycle offset indicates that multiple data assets are processed simultaneously in a data processing task. Scheduling time represents the executable time period of a data processing task. Scheduling interval represents the time interval between executions of a data processing task.

[0113] In one possible approach, when the computation engine of the data management device uses the Spark engine and the Flink engine, the user can also set the structured query language (SQL) for the processing tasks, as well as Spark user-defined functions (UDFs) and Flink UDFs. The processing task SQL can be Spark SQL or Flink SQL, used to represent the specific content executed by the computation engine. The Spark UDF can be a user-defined function used in Spark SQL. The Flink UDF can be a user-defined function used in Flink SQL.

[0114] Based on this, users can input multiple data assets and their corresponding storage media types into the storage media configured on the data management device. Next, users can configure data processing tasks based on the input data assets, write Spark SQL or Flink SQL, and add data processing rules for the task. Then, the data management device can determine in real time whether the current time is a scheduling period.

[0115] If so, the data management device can initiate a data processing task, reading the source data assets into memory through the configured input adapter module. Then, after performing calculations, the data management device can use the output adapter module to query the schema mapping table and write the calculation results into the schema of the target data asset.

[0116] In this embodiment, the data management device can receive a data acquisition request for collecting initial business data. The data acquisition request includes data source information, storage configuration information, and metadata generation rules for the initial business data. Furthermore, the data management device can map the initial business data to target business data based on the data source information and storage configuration information, and further generate metadata information describing the target business data according to the metadata generation rules.

[0117] Compared to general technologies that use different metadata to manage different types of data, this application supports user-defined metadata generation rules. This enables unified management of different data based on the same metadata format, reducing the complexity of big data platforms and avoiding the low standardization problem often caused by a lack of unified data management. Therefore, this application can solve the technical problem of low standardization caused by the difficulty of unified data management in general technologies.

[0118] The foregoing mainly describes the solutions provided by the embodiments of this application from a methodological perspective. To achieve the above functions, it includes corresponding hardware structures and / or software modules for executing each function. Those skilled in the art should readily recognize that, based on the units and algorithm steps of the examples described in conjunction with the embodiments disclosed herein, this application can be implemented in hardware or a combination of hardware and computer software. Whether a function is executed in hardware or by computer software driving hardware depends on the specific application and design constraints of the technical solution. Those skilled in the art can use different methods to implement the described functions for each specific application, but such implementation should not be considered beyond the scope of this application.

[0119] This application embodiment can divide the data management device into functional modules according to the above method example. For example, each function can be divided into its own functional module, or two or more functions can be integrated into one processing module. The integrated module can be implemented in hardware or as a software functional module. Optionally, the module division in this application embodiment is illustrative and only represents one logical functional division; other division methods may be used in actual implementation.

[0120] like Figure 10 The diagram shown is a structural schematic of a data management device provided in an embodiment of this application. This data management device can be used to perform tasks such as... Figures 4-7 The data management method shown is illustrated. The data management device includes: a receiving unit 801 and a processing unit 802.

[0121] The receiving unit 801 is configured to receive a data acquisition request for collecting initial business data; the data acquisition request includes data source information, storage configuration information, and metadata generation rules for the initial business data; for example, combined with... Figure 4 The receiving unit 801 can be used to execute S401.

[0122] Processing unit 802 is used to map initial business data to target business data based on data source information and storage configuration information; for example, combining... Figure 4 The processing unit 802 can be used to execute S402.

[0123] Processing unit 802 is also used to generate metadata information describing the target business data according to metadata generation rules. For example, combined with Figure 4 The processing unit 802 can be used to execute S403.

[0124] Optionally, the data source information is used to indicate the data source to which the initial business data belongs and the first storage medium corresponding to the data source; the storage configuration information is used to indicate the second storage medium corresponding to the target business data; the processing unit 802 is specifically used for:

[0125] Invoke the data read service corresponding to the first storage medium to read initial business data from the data source; for example, 0 combined with Figure 5 The processing unit 802 can be used to execute S501.

[0126] Within a preset mapping information set, target mapping tables corresponding to the first and second storage media are determined; these target mapping tables are used to map data within the first storage media to data within the second storage media; the preset mapping information set includes multiple mapping tables that correspond one-to-one with combinations of multiple storage media; for example, combining... Figure 5 The processing unit 802 can be used to execute S502.

[0127] 5. Invoke the data write service corresponding to the second storage medium and write the initial business data according to the target mapping table.

[0128] The data is stored in a second storage medium to obtain the target business data. For example, combined with... Figure 5 The processing unit 802 can be used to execute S503.

[0129] Optionally, the receiving unit 801 is further configured to receive an information update request sent by an electronic device; information update

[0130] The request is used to request updates to data source information and storage configuration information; for example, combined with Figure 6 The receiving unit 0801 can be used to execute S601.

[0131] The processing unit 802 is also used to update the data source information and storage configuration information in response to the information update request.

[0132] For example, combining Figure 6 The processing unit 802 can be used to execute S602.

[0133] Optionally, the receiving unit 801 is further configured to receive a data processing request from an electronic device; data processing

[0134] A data processing request is used to request data processing of target business data; a data processing request includes data processing rules; 5. For example, combining... Figure 7 The receiving unit 801 can be used to execute S701.

[0135] The processing unit 802 is also used to process the target business data according to data processing rules to obtain the processing result. For example, combined with Figure 7 The processing unit 802 can be used to execute S702.

[0136] Those skilled in the art will recognize that, in one or more of the examples above, the functions described in this application can be implemented using hardware, software, firmware, or any combination thereof. When implemented in software, these functions can be stored in a computer-readable medium or as one or more pointers on a computer-readable medium.

[0137] The transmission of commands or codes. Computer-readable media includes computer-readable storage media and communication media, wherein communication media includes any medium that facilitates the transmission of computer programs from one place to another. Storage media can be any available medium that can be accessed by a general-purpose or special-purpose computer.

[0138] Through the above description of the implementation methods, those skilled in the art can clearly understand that, for the sake of convenience and brevity, only the division of the above functional modules is used as an example. In practical applications, it can be based on...

[0139] If necessary, the above functions may be assigned to different functional modules, that is, the internal structure of the device may be divided into different functional modules to complete all or part of the functions described above.

[0140] In the several embodiments provided in this application, it should be understood that the disclosed apparatus and methods can be used in various ways.

[0141] Other methods can be used to achieve this. For example, the device embodiments described above are merely illustrative. For instance, the division of modules or units is only a logical functional division, and other division methods may exist in actual implementation. For example, multiple units or components may be combined or integrated into another device, or some features may be ignored or not executed. Furthermore, the shown or discussed mutual couplings, direct couplings, or communication connections may be through some interfaces; indirect couplings or communication connections between devices or units may be electrical, mechanical, or other forms. Units described as separate components may or may not be physically separate. Components shown as units may be one physical unit or multiple physical units, i.e., they may be located in one place or distributed in multiple different places. Some or all of the units can be selected to achieve the purpose of this embodiment according to actual needs.

[0142] The above description is merely a specific embodiment of this application, but the scope of protection of this application is not limited thereto. Any variations or substitutions that can be easily conceived by those skilled in the art within the technical scope disclosed in this application should be included within the scope of protection of this application. Therefore, the scope of protection of this application should be determined by the scope of the claims.

Claims

1. A data management method, characterized in that, include: Receive data collection requests for initial business data. The data acquisition request includes the data source information, storage configuration information, and metadata generation rules of the initial business data; The data source information is used to indicate the data source to which the initial business data belongs and the first storage medium corresponding to the data source; the storage configuration information is used to indicate the second storage medium corresponding to the target business data. Invoke the data reading service corresponding to the first storage medium to read the initial business data from the data source; In the preset mapping information set, determine the target mapping table corresponding to the first storage medium and the second storage medium; The target mapping table is used to map data in the first storage medium to data in the second storage medium; The preset mapping information set includes multiple mapping tables that correspond one-to-one with multiple storage medium combinations; Invoke the data write service corresponding to the second storage medium, and write the initial business data into the second storage medium according to the target mapping table to obtain the target business data; According to the metadata generation rules, metadata information describing the target business data is generated.

2. The data management method according to claim 1, characterized in that, Also includes: Receive information update requests sent by electronic devices; The information update request is used to request an update to the data source information and the storage configuration information; In response to the information update request, the data source information and the storage configuration information are updated.

3. The data management method according to claim 1, characterized in that, Also includes: Receive a data processing request from an electronic device; the data processing request is used to request data processing of the target business data; the data processing request includes data processing rules; The target business data is processed according to the data processing rules to obtain the processing result.

4. A data management device, characterized in that, include: Receiving unit and processing unit; The receiving unit is used to receive a data collection request for requesting the collection of initial business data; The data acquisition request includes the data source information, storage configuration information, and metadata generation rules of the initial business data; the data source information is used to indicate the data source to which the initial business data belongs and the first storage medium corresponding to the data source; the storage configuration information is used to indicate the second storage medium corresponding to the target business data; The processing unit is specifically used for: Invoke the data reading service corresponding to the first storage medium to read the initial business data from the data source; In the preset mapping information set, determine the target mapping table corresponding to the first storage medium and the second storage medium; The target mapping table is used to map data in the first storage medium to data in the second storage medium; The preset mapping information set includes multiple mapping tables that correspond one-to-one with multiple storage medium combinations; Invoke the data write service corresponding to the second storage medium, and write the initial business data into the second storage medium according to the target mapping table to obtain the target business data; The processing unit is further configured to generate metadata information describing the target business data according to the metadata generation rules.

5. The data management device according to claim 4, characterized in that, The receiving unit is further configured to receive an information update request sent by an electronic device; the information update request is used to request an update of the data source information and the storage configuration information; The processing unit is further configured to update the data source information and the storage configuration information in response to the information update request.

6. The data management device according to claim 4, characterized in that, The receiving unit is further configured to receive a data processing request from an electronic device; the data processing request is used to request data processing of the target business data; the data processing request includes data processing rules; The processing unit is further configured to process the target business data according to the data processing rules to obtain the processing result.

7. A data management device, characterized in that, It includes a memory and a processor; the memory is used to store computer execution instructions, and the processor is connected to the memory via a bus; when the data management device is running, the processor executes the computer execution instructions stored in the memory, so that the data management device performs the data management method as described in any one of claims 1-3.

8. A computer-readable storage medium, characterized in that, The computer-readable storage medium includes computer-executable instructions that, when executed on a computer, cause the computer to perform the data management method as described in any one of claims 1-3.

Images