A data acquisition method and device, electronic equipment and readable storage medium

Abstract

The present application relates to a kind of data acquisition methods, comprising: obtaining the data blood relationship corresponding to the first data type;Determine the dependent data type corresponding to the first data type and the metadata of dependent data type based on data blood relationship;Wherein, dependent data type indicates the data type corresponding to the data required for generating the data of the first data type, metadata includes at least one of the description information, constituting information and flow information of dependent data type;Generate the data acquisition plan corresponding to dependent data type based on metadata;Execute data acquisition plan, obtain the dependent data for generating target data.Through this method, the dependent data type for generating target data can be obtained according to data blood relationship, so as to formulate data acquisition plan and execute according to the metadata of dependent data type, and then obtain dependent data, and the configuration efficiency of data acquisition plan is improved, so that the acquisition efficiency of the data corresponding to new data type is improved.

Description

Technical Field

[0001] This invention relates to the field of data processing technology, and more specifically to a data acquisition method, apparatus, electronic device, and readable storage medium. Background Technology

[0002] With the rapid development of computer and sensor technologies, the types of data that can be collected are increasing. Collected data typically includes descriptive information such as name, size, and data type; compositional information such as length, fields, and columns; and data flow information such as references, citations, lineage analysis, impact analysis, and change analysis. This data used to describe the data is called metadata. Collected data generally undergoes a process of generation, processing, manipulation, fusion, and eventual disappearance. A natural relationship forms between data points, recording the chain of data generation. These relationships are similar to human bloodlines, hence the term "data lineage." Data lineage typically consists of data nodes, data flow paths, and data flow rules. Data lineage and metadata are primarily used for data analysis.

[0003] Data analysis requires the initial collection of relevant data. Data acquisition techniques typically begin with investigating data lineage and metadata, then identifying the data of interest, finding the corresponding data sources, collecting data from each source, and finally saving the collected data. The investigation, data source identification, and database creation for data storage all require manual intervention. Furthermore, the data acquisition process itself involves manually writing and executing code.

[0004] However, using this data collection method, the data collection plan needs to be configured manually for each new data type, which is very inefficient and results in low data collection efficiency for new data types. Summary of the Invention

[0005] One objective of this invention is to provide a data acquisition method to solve the problem of low data acquisition efficiency for new data types in the prior art; a second objective is to provide a data acquisition device; a third objective is to provide an electronic device; and a fourth objective is to provide a storage medium.

[0006] To achieve the above objectives, the technical solution adopted by the present invention is as follows:

[0007] A data acquisition method, the method comprising:

[0008] Retrieve the data lineage corresponding to the first data type;

[0009] Based on the data lineage, the metadata of the dependent data type corresponding to the first data type is determined; wherein, the dependent data type represents the data type corresponding to the data required to generate the data of the first data type, and the metadata includes at least one of the description information, composition information and circulation information of the dependent data type;

[0010] Generate a data collection plan corresponding to the dependent data type based on the metadata;

[0011] The data acquisition plan is executed to obtain the dependency data used to generate the target data.

[0012] Optionally, determining the metadata of the dependent data type corresponding to the first data type based on the data lineage includes:

[0013] A data master node is determined from the data nodes in the data lineage relationship; the data nodes include data outflow nodes, intermediate nodes, and data inflow nodes; the data master node represents one of the data nodes.

[0014] The dependent data type is determined based on the data type information corresponding to the data master node;

[0015] Obtain the metadata of the dependent data type.

[0016] Optionally, generating the data collection plan corresponding to the dependent data type based on the metadata includes:

[0017] Extract the collection information corresponding to the data of the dependent data type from the metadata;

[0018] The data acquisition plan is generated based on the acquired information.

[0019] Optionally, generating the data collection plan corresponding to the dependent data type based on the metadata includes:

[0020] Determine the sampling frequency corresponding to the dependent data type and the amount of data to be collected;

[0021] The collection method corresponding to the dependent data type is determined based on the amount of data collected.

[0022] The data acquisition plan is generated based on the acquisition frequency, the acquisition method, and the metadata.

[0023] Optionally, the data collection method includes full data collection and incremental data collection, and determining the data collection method for the dependent data based on the data volume includes:

[0024] If the amount of data collected is greater than or equal to the first threshold, the collection method is determined to be the full data collection.

[0025] If the amount of data collected is less than a first threshold, the collection method is determined to be incremental collection.

[0026] Optionally, the data acquisition method includes full data acquisition and incremental data acquisition, and the method further includes:

[0027] When the collection method is full collection, the dependent data is saved to the corresponding database based on a preset duration, and the collection data in the database that has a duration greater than or equal to the preset duration is deleted.

[0028] When the acquisition method is incremental acquisition, the dependent data is added to the corresponding database for storage.

[0029] Optionally, executing the data acquisition plan to obtain dependent data for generating the target data includes:

[0030] Determine the sensor corresponding to the dependent data type;

[0031] Based on the acquisition frequency, the sensor is controlled to acquire the natural language information corresponding to the sensor, so that the sensor generates and sends an electrical signal based on the natural language information and data conversion rules;

[0032] In response to the electrical signal sent by the sensor, the electrical signal is restored based on the data conversion rules to obtain the dependent data.

[0033] A data acquisition device, characterized in that the device comprises:

[0034] The first acquisition module is used to acquire the data lineage relationship corresponding to the first data type;

[0035] The determining module is used to determine the metadata of the dependent data type corresponding to the first data type based on the data lineage; wherein, the dependent data type represents the data type corresponding to the data required to generate the data of the first data type, and the metadata includes at least one of the description information, composition information and circulation information of the dependent data type;

[0036] The generation module is used to generate a data collection plan corresponding to the dependent data type based on the metadata;

[0037] The execution module is used to execute the data acquisition plan and obtain the dependency data used to generate the target data.

[0038] Optionally, the determining module includes:

[0039] The first determining submodule is used to determine the data master node from the data nodes in the data lineage relationship;

[0040] The second determining submodule is used to determine the dependent data type based on the data type information corresponding to the data master node;

[0041] The `get` submodule is used to retrieve metadata for the dependent data types.

[0042] Optionally, the generation module includes:

[0043] An extraction submodule is used to extract the collection information corresponding to the data of the dependent data type from the metadata.

[0044] The first generation submodule is used to generate the data acquisition plan based on the acquired information.

[0045] Optionally, the generation module includes:

[0046] The third determining submodule is used to determine the collection frequency corresponding to the dependent data type and the amount of data to be collected;

[0047] The fourth determining submodule is used to determine the collection method corresponding to the dependent data type based on the amount of collected data;

[0048] The second generation submodule is used to generate the data acquisition plan based on the acquisition frequency, the acquisition method, and the metadata.

[0049] Optionally, the data acquisition method includes full data acquisition and incremental data acquisition, and the fourth determining submodule includes:

[0050] The first determining unit is configured to determine that the acquisition method is the full acquisition when the amount of acquired data is greater than or equal to a first threshold.

[0051] The second determining unit is used to determine that the acquisition method is incremental acquisition when the amount of acquired data is less than the first threshold.

[0052] Optionally, the acquisition method includes full acquisition and incremental acquisition, and the device further includes:

[0053] The first storage module is used to save the dependent data to the corresponding database for storage based on a preset duration when the collection method is full collection, and to delete the collection data in the database that has a duration greater than or equal to the preset duration.

[0054] The second storage module is used to add the dependent data into the corresponding database for storage when the acquisition method is incremental acquisition.

[0055] Optionally, the execution module includes:

[0056] The fifth determining submodule is used to determine the sensor corresponding to the dependent data type;

[0057] The acquisition submodule is used to control the sensor to acquire the natural language information corresponding to the sensor based on the acquisition frequency, so that the sensor generates an electrical signal based on the natural language information and data conversion rules and sends it.

[0058] The restoration submodule is used to respond to the electrical signal sent by the sensor and perform restoration processing on the electrical signal based on the data conversion rules to obtain the dependent data.

[0059] An electronic device, characterized in that it includes a processor and a memory, the memory storing a program or instructions executable on the processor, the program or instructions, when executed by the processor, implementing the data acquisition method as described above. A readable storage medium, characterized in that it stores a program or instructions, the program or instructions, when executed by a processor, implementing the data acquisition method as described above.

[0060] The beneficial effects of this invention are:

[0061] This invention obtains the data lineage corresponding to a first data type, determines the dependent data type and its metadata based on the data lineage, whereby the dependent data type represents the data type corresponding to the data required to generate the first data type, and the metadata includes at least one of the descriptive information, composition information, and flow information of the dependent data type. Based on the metadata, a data acquisition plan corresponding to the dependent data type is generated, and the data acquisition plan is executed to obtain the dependent data used to generate the target data. The dependent data type used to generate the target data can be directly derived from the data lineage, thereby formulating and executing a data acquisition plan based on the metadata of the dependent data type to obtain the dependent data. This achieves automated configuration of the data acquisition plan, improves the configuration efficiency of the data acquisition plan, and thus improves the data acquisition efficiency corresponding to the new data type. Attached Figure Description

[0062] Figure 1 This is a flowchart illustrating a data acquisition method provided in an embodiment of the present invention;

[0063] Figure 2A flowchart illustrating another data acquisition method provided in an embodiment of the present invention;

[0064] Figure 3 A flowchart illustrating the error handling method for metadata extraction provided in this embodiment of the invention;

[0065] Figure 4 A flowchart illustrating the error handling method for data acquisition provided in this embodiment of the invention;

[0066] Figure 5 A logic block diagram of a data acquisition device provided in an embodiment of the present invention;

[0067] Figure 6 This is a schematic diagram of the structure of an electronic device provided in an embodiment of the present invention. Detailed Implementation

[0068] The embodiments of the present invention will be described below with reference to the accompanying drawings and preferred embodiments. Those skilled in the art can easily understand other advantages and effects of the present invention from the content disclosed in this specification. The present invention can also be implemented or applied through other different specific embodiments, and various details in this specification can also be modified or changed based on different viewpoints and applications without departing from the spirit of the present invention. It should be understood that the preferred embodiments are only for illustrating the present invention and not for limiting the scope of protection of the present invention.

[0069] It should be noted that the illustrations provided in the following embodiments are only schematic representations of the basic concept of the present invention. Therefore, the drawings only show the components related to the present invention and are not drawn according to the actual number, shape and size of the components in the actual implementation. In the actual implementation, the form, quantity and proportion of each component can be arbitrarily changed, and the layout of the components may also be more complex.

[0070] Reference Figure 1 , Figure 1 This is a flowchart illustrating a data acquisition method provided in an embodiment of the present invention. The method may include:

[0071] Step 101: Obtain the data lineage relationship corresponding to the first data type.

[0072] In embodiments of the present invention, data lineage can represent a relationship that naturally forms during the process of data generation, processing, manipulation, fusion, flow, and eventual disappearance. Data lineage generally consists of data nodes, data flow paths, and data flow rules, which directly or indirectly indicate lineage information. Data nodes can represent carriers with independent data functions, reflecting the business attributes and storage location of the data. Data nodes include three types: data outflow nodes, which provide the most basic data, generally the underlying source data; intermediate nodes, which are the most diverse type of node in the data lineage, both receiving incoming data and providing data to incoming nodes; and data inflow nodes, which are the terminal nodes of the entire data lineage, receiving data from intermediate nodes and then not flowing further. The data inflow nodes can be the output of business systems, used for visualization reports or dashboards, and in rare cases, can also provide data feedback to other business systems. Data flow paths indicate the inflow and outflow of data by representing three dimensions: data flow direction, data update magnitude, and data update frequency. The direction of data flow can be indicated by arrows, and the importance of data can be judged by the magnitude and frequency of data updates. Data flow rules demonstrate the changes that occur during data flow and how data becomes part of other entities. Each data flow path can contain one or more flow rules.

[0073] In embodiments of the present invention, the first data type corresponding to the data to be observed can first be identified. Then, the data lineage corresponding to the first data type can be obtained based on various characteristics of the first data type. This data lineage can be generated along with the generation of the first data type, indicating the flow process of the data corresponding to the first data type. In embodiments of the present invention, the first data type can be the data type corresponding to the data inflow node; that is, the data corresponding to the first data type can be the output data of the entire business system, which has direct reference value.

[0074] Step 102: Determine the metadata of the dependent data type corresponding to the first data type based on the data lineage; wherein, the dependent data type represents the data type corresponding to the data required to generate the first data type, and the metadata includes at least one of the description information, composition information, and flow information of the dependent data type.

[0075] In embodiments of the present invention, the dependent data type can represent the data type corresponding to the data required to generate the first data type, and the metadata can include at least one of the description information, composition information and flow information of the dependent data type.

[0076] In embodiments of the present invention, the dependent data types corresponding to the first data type can be determined based on the data node information, data flow path, and data flow rules in the data lineage. Specifically, the first data type can correspond to a data node in the data lineage. This data node can be an intermediate node or a data inflow node. Then, all data outflow nodes corresponding to this data node can be identified based on the data lineage, thereby confirming that the data types corresponding to these data outflow nodes are dependent data types. A data outflow node indicates that there is no data inflow to the node, and the data corresponding to the node can be directly acquired by the corresponding sensor. After obtaining the dependent data types, the metadata corresponding to these dependent data types can be obtained. This metadata can include the descriptive information, composition information, and flow information of the data corresponding to the dependent data types. Among them, the data descriptive information can include the data name, size, data type, etc.; the data composition information can include the data length, fields, data columns, etc.; and the data flow information can include the data's referentiality, citation, lineage analysis, impact analysis, change analysis, etc.

[0077] Step 103: Generate a data collection plan corresponding to the dependent data type based on the metadata.

[0078] In embodiments of the present invention, after obtaining the metadata of the dependent data type, a data acquisition plan corresponding to the dependent data type can be generated based on one or more pieces of information in the metadata. This data plan includes various reference information required during the data acquisition process. For example, the data name storing the data in the database can be determined based on the data name, and a storage path can be established. The storage space for storing the data in the database can be determined based on the data size, and so on. It should be noted that the data acquisition plan information generated based on metadata provided in the embodiments of the present invention is merely illustrative and does not constitute a limitation on the method. All methods for generating corresponding data acquisition plans based on metadata fall within the protection scope of the embodiments of the present invention, and the present invention does not limit them.

[0079] Step 104: Execute the data acquisition plan to obtain the dependency data used to generate the target data.

[0080] In embodiments of the present invention, after obtaining the data acquisition plan, data acquisition can be performed based on the data acquisition plan, that is, the data acquisition plan can be executed, thereby obtaining dependent data. This dependent data can correspond to a dependent data type and can be used to generate target data. In embodiments of the present invention, the target data can be data corresponding to a first data type.

[0081] In embodiments of the present invention, by obtaining the data lineage corresponding to the first data type, the dependent data type and its metadata are determined based on the data lineage. The dependent data type represents the data type corresponding to the data required to generate the first data type. The metadata includes at least one of the description information, composition information, and flow information of the dependent data type. A data acquisition plan corresponding to the dependent data type is generated based on the metadata. The data acquisition plan is executed to obtain the dependent data used to generate the target data. The dependent data type used to generate the target data can be directly derived from the data lineage. Thus, a data acquisition plan is formulated and executed based on the metadata of the dependent data type to obtain the dependent data. This achieves the effect of automating the configuration of the data acquisition plan, improving the configuration efficiency of the data acquisition plan, and thus improving the data acquisition efficiency corresponding to the new data type.

[0082] Reference Figure 2 , Figure 2 This is a flowchart illustrating another data acquisition method provided in an embodiment of the present invention. The method may include:

[0083] Step 201: Obtain the data lineage relationship corresponding to the first data type.

[0084] In the embodiments of the present invention, the meaning of the first data type and data lineage relationship, as well as the acquisition method, can be referred to the content of step 101 in the above embodiments, and will not be repeated here.

[0085] Step 202: Determine the data master node from the data nodes in the data lineage.

[0086] In embodiments of the present invention, the data lineage may include multiple data nodes, which may include data outflow nodes, intermediate nodes, and data inflow nodes. A data master node may represent one of these data nodes. The method for determining the data master node may be to determine the data node corresponding to the first data type from the data lineage corresponding to that data type. The data node type of the data master node may be one of a data outflow node, an intermediate node, or a data inflow node.

[0087] Step 203: Determine the dependent data type based on the data type information corresponding to the data master node.

[0088] In embodiments of the present invention, after confirming the data master node, the dependent data type of the data corresponding to the data master node can be determined based on the data type information corresponding to the data master node. Specifically, the data flow path of the data corresponding to the data master node can be determined in the data lineage, and then the dependent data type of the data that generates the data corresponding to the data master node can be obtained through the data flow path and data flow rules. The dependent data type can indicate that the data corresponding to the data master node is obtained by a series of processing based on the data corresponding to the dependent data type.

[0089] Step 204: Obtain the metadata of the dependent data type.

[0090] In embodiments of the present invention, metadata may include descriptive information, composition information, and flow information of data corresponding to the data type. Specifically, the data descriptive information may include the data's name, size, and data type; the data composition information may include the data's length, fields, and data columns; and the data flow information may include data referentiality, citation, lineage analysis, impact analysis, and change analysis. Based on the descriptive characteristics of metadata, metadata dependent on the data type can be obtained.

[0091] In embodiments of the present invention, by determining the data master node from the data nodes in the data lineage, determining the dependent data type based on the data type information corresponding to the data master node, and obtaining the metadata of the dependent data type, the data master node corresponding to the data to be observed can be confirmed based on the data lineage, and then the dependent data type corresponding to the data master node in the data lineage can be obtained, thereby obtaining the metadata of the dependent data type. This avoids manually setting the data source connection and improves the configuration efficiency of the data acquisition plan.

[0092] Step 205: Extract the collection information corresponding to the data of the dependent data type from the metadata.

[0093] In embodiments of the present invention, metadata includes a large amount of information, and the metadata that can be used to generate a data acquisition plan can be a portion of the metadata. Therefore, it is possible to identify which data in the metadata is the acquisition information required for the acquisition of the dependent data type. This acquisition information is then extracted from the metadata. In embodiments of the present invention, the required acquisition information includes, but is not limited to, information such as the data source and data structure of the dependent data type. The data source may include database connection information, database type, and data volume; the data structure may include data length, fields, and data columns. It should be noted that the examples of data sources and data structures above do not constitute a limitation; other data with corresponding characteristics are within the scope of protection of the embodiments of the present invention.

[0094] In an embodiment of the present invention, reference is made to Figure 3 , Figure 3 This is a flowchart illustrating the error handling method for metadata extraction provided in this embodiment of the invention. In the flowchart, a collection plan is first established based on the data corresponding to the dependent data types. Then, the required collection information is extracted from the metadata. Next, it is confirmed whether the collection information has been successfully extracted. If the extraction is successful, the data collection frequency and method can be further confirmed. If the extraction fails, an error message is output to the user, who then handles the error and completes the data that was not successfully extracted.

[0095] Step 206: Generate the data acquisition plan based on the acquired information.

[0096] In an embodiment of the present invention, after confirming the collection information required for collection, a corresponding data collection plan can be generated based on this collection information.

[0097] Specifically, step 206 may include the following sub-steps:

[0098] Sub-step 2061: Determine the sampling frequency corresponding to the dependent data type and the amount of data to be collected.

[0099] In embodiments of the present invention, for data corresponding to a dependent data type that needs to be collected, the corresponding collection frequency and data volume can be determined. For a given type of data, different collection frequencies and data volumes may exist for different applications. Therefore, the collection frequency and data volume can be determined based on the application direction of the data corresponding to the dependent data type.

[0100] Sub-step 2062: Determine the collection method corresponding to the dependent data type based on the amount of collected data.

[0101] In embodiments of the present invention, the acquisition method corresponding to the dependent data type can be determined based on the size of the acquired data volume.

[0102] Specifically, sub-step 2062 may include the following sub-steps:

[0103] Sub-step a1: If the amount of collected data is greater than or equal to the first threshold, determine that the collection method is the full collection.

[0104] In embodiments of the present invention, the first threshold can represent a preset data volume threshold. When the amount of collected data is greater than or equal to the first threshold, the collected data may include data that lacks reference value. Therefore, in this case, the collection method can be determined as full collection. Full collection can mean collecting and saving data of the preset duration and deleting collected data in the database that has a duration greater than or equal to the preset duration.

[0105] For example, in a full data collection process, if the preset duration is 30 days, then for each day's data collected, the corresponding data from 30 days ago will be deleted, and so on.

[0106] Sub-step a2: If the amount of collected data is less than the first threshold, determine that the collection method is incremental collection.

[0107] In embodiments of the present invention, when the amount of data collected is less than a first threshold, the collection method can be determined to be incremental collection. Incremental collection can mean collecting new data as new data and adding the new data to the corresponding database. Furthermore, the incrementally collected data can have a timestamp to indicate the collection time of the corresponding data. Data change analysis corresponding to this data type can be obtained based on the timestamp and the corresponding data.

[0108] In embodiments of the present invention, when the amount of collected data is greater than or equal to a first threshold, the collection method is determined to be full collection; when the amount of collected data is less than the first threshold, the collection method is determined to be incremental collection. The collection method corresponding to the data to be collected can be clearly defined according to the preset data volume threshold, which facilitates the corresponding processing of the collected data and improves the efficiency of the early data management.

[0109] Sub-step 2063: Generate the data acquisition plan based on the acquisition frequency, the acquisition method, and the metadata.

[0110] In embodiments of the present invention, after confirming the collection frequency and collection method of the data to be collected, a data collection plan can be generated by combining the metadata corresponding to the data to be collected. When executing the data collection plan, the data to be collected can be collected according to the collection frequency, collection method, and other information indicated in the metadata.

[0111] In embodiments of the present invention, by determining the collection frequency corresponding to the dependent data type and the amount of data to be collected, the collection method corresponding to the dependent data type is determined based on the amount of data to be collected, and a data collection plan is generated based on the collection frequency, collection method and metadata. This refines the information contained in the data collection plan, improves the configuration information of the data collection plan, and enhances the execution efficiency of the data collection plan.

[0112] In embodiments of the present invention, by extracting the collection information corresponding to the data of dependent data types from the metadata and generating a data collection plan based on the collection information, useless data in the metadata constituting the data collection plan can be filtered out, further improving the configuration efficiency of the data collection plan.

[0113] Step 207: Execute the data acquisition plan to obtain the dependency data used to generate the target data.

[0114] In embodiments of the present invention, the data acquisition plan can be executed according to the acquisition frequency, acquisition method, and other information specified in the metadata, thereby acquiring the data to be acquired and obtaining dependent data. The dependent data can be data corresponding to the dependent data type, i.e., the data to be acquired. The dependent data can generate target data based on data lineage, and the target data can be data corresponding to the first data type.

[0115] Specifically, step 207 may include the following sub-steps:

[0116] Sub-step 2071: Determine the sensor corresponding to the dependent data type.

[0117] In embodiments of the present invention, during a typical data acquisition process, it is necessary to capture the data to be collected using a corresponding sensor. Therefore, before acquisition, the sensor corresponding to the dependent data type can be identified. This sensor can collect the data corresponding to the dependent data type and convert the data into a corresponding electrical signal, which is then sent to the host computer connected to the sensor.

[0118] Sub-step 2072: Based on the acquisition frequency, control the sensor to acquire the natural language information corresponding to the sensor, so that the sensor generates an electrical signal based on the natural language information and data conversion rules and sends it.

[0119] In embodiments of the present invention, after identifying the sensor corresponding to the dependent data type, the sensor can be controlled to collect the corresponding natural language information according to the acquisition frequency. For example, a temperature sensor can acquire corresponding temperature information, a pressure sensor can acquire corresponding pressure information, and so on. After acquiring the corresponding natural language information, the sensor converts the information into an electrical signal with the same frequency as the acquisition frequency and sends it to the host computer, which can receive these electrical signals.

[0120] Sub-step 2073: In response to the electrical signal sent by the sensor, the electrical signal is restored based on the data conversion rules to obtain the dependent data.

[0121] In embodiments of the present invention, the electrical signal is data obtained by processing the collected data in one step. Therefore, the electrical signal can be restored according to the data conversion rules of the sensor to obtain the dependency data. To store the dependency data, it can be encoded and stored as a string in the corresponding storage area of ​​the storage device. When needed, the storage area can be accessed, and the string can be decoded to obtain the dependency data.

[0122] In an embodiment of the present invention, reference is made to Figure 4 , Figure 4 This is a flowchart illustrating the error handling method for data acquisition provided in an embodiment of the present invention. In the flowchart, data is first acquired according to the acquisition frequency, i.e., a data acquisition plan is executed. Then, it is determined whether the data acquisition plan was executed successfully. If successful, a success message is sent to the user; if unsuccessful, an error message is sent to the user. The user can then promptly handle the relevant errors based on this error message.

[0123] In an embodiment of the present invention, by determining the sensor corresponding to the dependent data type, the sensor is controlled to collect the natural language information corresponding to the sensor based on the acquisition frequency, so that the sensor generates and sends an electrical signal based on the natural language information and data conversion rules. In response to the electrical signal sent by the sensor, the electrical signal is restored based on the data conversion rules to obtain the dependent data. The corresponding data can be collected after the data acquisition plan is generated, thus completing the verification of the data acquisition plan configuration and ensuring the reliability of the data acquisition plan configuration.

[0124] Step 208: When the collection method is full collection, the dependent data is saved to the corresponding database based on a preset duration, and the collection data in the database with a duration greater than or equal to the preset duration is deleted.

[0125] In embodiments of the present invention, after obtaining the dependency data, the dependency data can be processed according to the acquisition method. When the acquisition method is full acquisition, the dependency data can be saved to the database corresponding to the dependency data type according to the preset duration specified by the full acquisition. Then, acquisition data in the database with a duration greater than or equal to the preset duration can be deleted.

[0126] Step 209: If the acquisition method is incremental acquisition, add the dependent data into the corresponding database for storage.

[0127] In an embodiment of the present invention, after obtaining the dependency data, if the acquisition method is incremental acquisition, the acquired dependency data can be directly added to the corresponding database for storage.

[0128] In the embodiments of the present invention, when the collection method is full collection, the dependent data is updated to the corresponding database and saved based on a preset time period. When the collection method is incremental collection, the dependent data is added to the corresponding database and saved. This can save the collected data, improve the data collection process, avoid the loss of collected data, and facilitate subsequent data processing.

[0129] Reference Figure 5 , Figure 5 This is a logic block diagram of a data acquisition device provided in an embodiment of the present invention. The data acquisition device may include:

[0130] The first acquisition module is used to acquire the data lineage relationship corresponding to the first data type;

[0131] The determining module is used to determine the metadata of the dependent data type corresponding to the first data type based on the data lineage; wherein, the dependent data type represents the data type corresponding to the data required to generate the data of the first data type, and the metadata includes at least one of the description information, composition information and circulation information of the dependent data type;

[0132] The generation module is used to generate a data collection plan corresponding to the dependent data type based on the metadata;

[0133] The execution module is used to execute the data acquisition plan and obtain the dependency data used to generate the target data.

[0134] Optionally, the determining module includes:

[0135] The first determining submodule is used to determine the data master node from the data nodes in the data lineage relationship;

[0136] The second determining submodule is used to determine the dependent data type based on the data type information corresponding to the data master node;

[0137] The `get` submodule is used to retrieve metadata for the dependent data types.

[0138] Optionally, the generation module includes:

[0139] An extraction submodule is used to extract the collection information corresponding to the data of the dependent data type from the metadata.

[0140] The first generation submodule is used to generate the data acquisition plan based on the acquired information.

[0141] Optionally, the generation module includes:

[0142] The third determining submodule is used to determine the collection frequency corresponding to the dependent data type and the amount of data to be collected;

[0143] The fourth determining submodule is used to determine the collection method corresponding to the dependent data type based on the amount of collected data;

[0144] The second generation submodule is used to generate the data acquisition plan based on the acquisition frequency, the acquisition method, and the metadata.

[0145] Optionally, the data acquisition method includes full data acquisition and incremental data acquisition, and the fourth determining submodule includes:

[0146] The first determining unit is configured to determine that the acquisition method is the full acquisition when the amount of acquired data is greater than or equal to a first threshold.

[0147] The second determining unit is used to determine that the acquisition method is incremental acquisition when the amount of acquired data is less than the first threshold.

[0148] Optionally, the acquisition method includes full acquisition and incremental acquisition, and the device further includes:

[0149] The first storage module is used to save the dependent data to the corresponding database for storage based on a preset duration when the collection method is full collection, and to delete the collection data in the database that has a duration greater than or equal to the preset duration.

[0150] The second storage module is used to add the dependent data into the corresponding database for storage when the acquisition method is incremental acquisition.

[0151] Optionally, the execution module includes:

[0152] The fifth determining submodule is used to determine the sensor corresponding to the dependent data type;

[0153] The acquisition submodule is used to control the sensor to acquire the natural language information corresponding to the sensor based on the acquisition frequency, so that the sensor generates an electrical signal and sends it based on the natural language information and data conversion rules;

[0154] The restoration submodule is used to respond to the electrical signal sent by the sensor and perform restoration processing on the electrical signal based on the data conversion rules to obtain the dependent data.

[0155] As the device embodiment is basically similar to the method embodiment, the description is relatively simple, and relevant parts can be found in the description of the method embodiment.

[0156] The data acquisition device described above has the same advantages over the prior art as the data acquisition method described in the foregoing embodiments, and will not be repeated here.

[0157] This invention provides an electronic device, see [link to relevant documentation]. Figure 6The electronic device 60 includes a processor 601, a memory 602, and a computer program 6021 stored in the memory 602 and executable on the processor 601. When the processor 601 executes the program, it implements the data acquisition method of the foregoing embodiments.

[0158] This invention provides a readable storage medium that, when the instructions in the storage medium are executed by the processor of an electronic device, enables the electronic device to perform the aforementioned data acquisition method.

[0159] The algorithms and displays provided herein are not inherently related to any particular computer, virtual system, or other device. The structure required to construct such a system is readily apparent from the above description. Furthermore, this invention is not directed to any particular programming language. It should be understood that the contents of the invention described herein can be implemented using various programming languages, and the above description of specific languages ​​is for the purpose of disclosing the best mode of implementation of the invention.

[0160] Numerous specific details are set forth in the specification provided herein. However, it will be understood that embodiments of the invention may be practiced without these specific details. In some instances, well-known methods, structures, and techniques have not been shown in detail so as not to obscure the understanding of this specification.

[0161] Similarly, it should be understood that, in order to simplify the invention and aid in understanding one or more of the various inventive aspects, in the above description of exemplary embodiments of the invention, various features of the invention are sometimes grouped together in a single embodiment, figure, or description thereof. However, this disclosure should not be construed as reflecting an intention that the claimed invention requires more features than are expressly recited in each claim. Rather, as reflected in the following claims, inventive aspects lie in fewer than all features of a single foregoing disclosed embodiment. Therefore, the claims following the detailed description are hereby expressly incorporated into this detailed description, wherein each claim itself is a separate embodiment of the invention.

[0162] Those skilled in the art will understand that modules in the device of the embodiments can be adaptively changed and placed in one or more devices different from that embodiment. Modules, units, or components in the embodiments can be combined into a single module, unit, or component, and further, they can be divided into multiple sub-modules, sub-units, or sub-components. Except where at least some of such features and / or processes or units are mutually exclusive, any combination can be used to combine all features disclosed in this specification (including the accompanying claims, abstract, and drawings) and all processes or units of any method or device so disclosed. Unless expressly stated otherwise, each feature disclosed in this specification (including the accompanying claims, abstract, and drawings) may be replaced by an alternative feature that serves the same, equivalent, or similar purpose.

[0163] The various component embodiments of the present invention can be implemented in hardware, or as software modules running on one or more processors, or a combination thereof. Those skilled in the art will understand that microprocessors or digital signal processors (DSPs) can be used in practice to implement some or all of the functions of some or all of the components in the sorting device according to the present invention. The present invention can also be implemented as a device or apparatus program for performing part or all of the methods described herein. Such a program implementing the present invention can be stored on a computer-readable medium, or can be in the form of one or more signals. Such signals can be downloaded from an Internet website, provided on a carrier signal, or provided in any other form.

[0164] It should be noted that the above embodiments are illustrative of the invention and not restrictive, and that those skilled in the art can devise alternative embodiments without departing from the scope of the appended claims. In the claims, any reference signs placed between parentheses should not be construed as limiting the claims. The word "comprising" does not exclude the presence of elements or steps not listed in the claims. The word "a" or "an" preceding an element does not exclude the presence of a plurality of such elements. The invention can be implemented by means of hardware comprising several different elements and by means of a suitably programmed computer. In the unit claims enumerating several means, several of these means may be embodied by the same item of hardware. The use of the words first, second, and third, etc., does not indicate any order. These words can be interpreted as names.

[0165] Those skilled in the art will understand that, for the sake of convenience and brevity, the specific working processes of the systems, devices, and units described above can be referred to the corresponding processes in the foregoing method embodiments, and will not be repeated here.

[0166] The above description is only a preferred embodiment of the present invention and is not intended to limit the present invention. Any modifications, equivalent substitutions, and improvements made within the spirit and principles of the present invention should be included within the protection scope of the present invention.

[0167] The above description is merely a specific embodiment of the present invention, but the scope of protection of the present invention is not limited thereto. Any variations or substitutions that can be easily conceived by those skilled in the art within the technical scope disclosed in the present invention should be included within the scope of protection of the present invention. Therefore, the scope of protection of the present invention should be determined by the scope of the claims.

[0168] It should be noted that the various data-related processes in the embodiments of the present invention are carried out in compliance with the relevant data protection laws and policies of the country where the location is located, and with the authorization granted by the owner of the corresponding device.

[0169] The above embodiments are merely preferred embodiments provided to fully illustrate the present invention, and the scope of protection of the present invention is not limited thereto. Equivalent substitutions or modifications made by those skilled in the art based on the present invention are all within the scope of protection of the present invention.

Claims

1. A data acquisition method, characterized in that, The method includes: Retrieve the data lineage corresponding to the first data type; Based on the data lineage, the metadata of the dependent data type corresponding to the first data type is determined; wherein, the dependent data type represents the data type corresponding to the data required to generate the data of the first data type, and the metadata includes at least one of the description information, composition information and circulation information of the dependent data type; Generate a data collection plan corresponding to the dependent data type based on the metadata; The data acquisition plan is executed to obtain dependent data for generating target data; the dependent data is data corresponding to the dependent data type, and the target data is data corresponding to the first data type. The step of generating a data collection plan corresponding to the dependent data type based on the metadata includes: Determine the sampling frequency corresponding to the dependent data type and the amount of data to be collected; The collection method corresponding to the dependent data type is determined based on the amount of data collected. The data acquisition plan is generated based on the acquisition frequency, the acquisition method, and the metadata.

2. The method according to claim 1, characterized in that, The step of determining the metadata of the dependent data type corresponding to the first data type based on the data lineage includes: The data master node is determined from the data nodes in the data lineage relationship; The dependent data type is determined based on the data type information corresponding to the data master node; Obtain the metadata of the dependent data type.

3. The method according to claim 1, characterized in that, The step of generating a data collection plan corresponding to the dependent data type based on the metadata includes: Extract the collection information corresponding to the data of the dependent data type from the metadata; The data acquisition plan is generated based on the acquired information.

4. The method according to claim 1, characterized in that, The data collection methods include full data collection and incremental data collection. Determining the data collection method for the dependent data based on the data volume includes: If the amount of data collected is greater than or equal to the first threshold, the collection method is determined to be the full data collection. If the amount of data collected is less than a first threshold, the collection method is determined to be incremental collection.

5. The method according to claim 1, characterized in that, The data acquisition methods include full data acquisition and incremental data acquisition, and the method further includes: When the collection method is full collection, the dependent data is saved to the corresponding database based on a preset duration, and the collection data in the database that has a duration greater than or equal to the preset duration is deleted. When the acquisition method is incremental acquisition, the dependent data is added to the corresponding database for storage.

6. The method according to claim 1, characterized in that, The execution of the data acquisition plan, to obtain dependent data for generating the target data, includes: Determine the sensor corresponding to the dependent data type; Based on the acquisition frequency, the sensor is controlled to acquire the natural language information corresponding to the sensor, so that the sensor generates and sends an electrical signal based on the natural language information and data conversion rules; In response to the electrical signal sent by the sensor, the electrical signal is restored based on the data conversion rules to obtain the dependent data.

7. A data acquisition device, characterized in that, The device includes: The first acquisition module is used to acquire the data lineage relationship corresponding to the first data type; The determining module is used to determine the dependent data type corresponding to the first data type and the metadata of the dependent data type based on the data lineage; wherein, the dependent data type represents the data type corresponding to the data required to generate the data of the first data type, and the metadata includes at least one of the description information, composition information and circulation information of the dependent data type; The generation module is used to generate a data collection plan corresponding to the dependent data type based on the metadata; An execution module is used to execute the data acquisition plan and obtain dependent data for generating target data; the dependent data is data corresponding to the dependent data type, and the target data is data corresponding to the first data type; The generation module includes: The third determining submodule is used to determine the collection frequency corresponding to the dependent data type and the amount of data to be collected; The fourth determining submodule is used to determine the collection method corresponding to the dependent data type based on the amount of collected data; The second generation submodule is used to generate the data acquisition plan based on the acquisition frequency, the acquisition method, and the metadata.

8. An electronic device, characterized in that, The electronic device includes a processor and a memory, the memory storing programs or instructions that can run on the processor, the programs or instructions being executed by the processor to implement the data acquisition method as described in any one of claims 1 to 6.

9. A readable storage medium, characterized in that, The readable storage medium stores a program or instructions that, when executed by a processor, implement the data acquisition method as described in any one of claims 1 to 6.

Images