Data acquisition method, device and equipment
Through annotation and parameterization technology, the target data source is configured for data acquisition tasks, which solves the problem of excessive resource consumption of traditional data acquisition tools, and realizes the data acquisition effect of automation and resource saving.
Patent Information
- Application Number
- CN202411691918.5
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2024-11-25
- Publication Date
- 2025-03-14
AI Technical Summary
Traditional data acquisition tools have problems such as corresponding number of data sources and acquisition programs, complex encoding, and excessive connections to consume data sources, resulting in excessive consumption of server resources and network resources.
By creating multiple data acquisition sources, the data capture tasks of each data acquisition source are annotated, and the target data source is parameterized into the annotation of the data capture task, the data capture tasks are performed based on the annotation, and the data capture tasks of the same target data source are merged to reduce resource consumption.
This method reduces the need to manually write complex code, and the system can automatically perform tasks according to annotations. Multiple data sources can share the same acquisition logic, avoid resource waste, and save network and server resources by merging tasks from the same target data source.
Smart Images
Figure CN119621318A_ABST
Abstract
Description
Technical Field
[0001] The present application relates to the field of data processing technology, and in particular to a data acquisition method, device and equipment. Background Art
[0002] With the rapid development of the Internet and enterprise informatization, more and more enterprises have built a large number of enterprise application systems in the process of digital transformation. These application systems are distributed in different departments and business scenarios, often based on different technical architectures, using a variety of databases and data interfaces to manage and store data. As a result, these data sources have significant heterogeneity.
[0003] In the process of digital transformation of enterprises, the data center has gradually become an important platform for enterprise data management. The construction of the data center aims to uniformly manage various heterogeneous data of enterprises and provide data services and support for upper-level business systems. However, traditional data collection tools have problems such as the corresponding number of data sources and collection programs, complex coding, and excessive consumption of data source connections, which leads to excessive consumption of server resources and network resources. Summary of the invention
[0004] In view of this, the main purpose of the present application is to provide a data acquisition method, device and equipment, the purpose of which is to reduce the consumption of computer resources.
[0005] The first aspect of the present application provides a data acquisition method, the method comprising:
[0006] Create multiple data acquisition sources, annotate the data capture tasks corresponding to each data acquisition source, and parameterize the target data source corresponding to each data acquisition source into the annotation of the data capture task corresponding to the data acquisition source;
[0007] Data capture tasks are executed based on annotations to obtain source data from a target data source through a data acquisition source, wherein data capture tasks with the same target data source are executed in a merged manner.
[0008] In some implementations of the first aspect of the present application, the method further includes:
[0009] Perform data normalization on source data to transform the source data into the target format.
[0010] In some implementations of the first aspect of the present application, the target format is JSON format.
[0011] In some implementations of the first aspect of the present application, the method further includes:
[0012] Distribute the source data converted into the target format to the corresponding target data processing task for data processing to obtain the target data;
[0013] Load data based on target data.
[0014] In some implementations of the first aspect of the present application, multiple data collection sources are created, including:
[0015] The class implementing the data acquisition source interface in the specified package is obtained through the package scanning technology, and the class of the data acquisition source interface is instantiated using the Java reflection technology to obtain multiple data acquisition sources.
[0016] In some implementations of the first aspect of the present application, annotating the data capture task corresponding to each data acquisition source, and parameterizing the target data source corresponding to each data acquisition source into the annotation of the data capture task corresponding to the data acquisition source, including:
[0017] The data capture tasks corresponding to each data collection source are identified by using Java annotation technology, and the annotations corresponding to each data capture task are obtained;
[0018] For each data acquisition source, determine the target data source corresponding to the data acquisition source, and parameterize the target data source corresponding to the data acquisition source into the annotation of the data capture task corresponding to the data acquisition source.
[0019] In some implementations of the first aspect of the present application, performing a data capture task based on the annotation includes:
[0020] By using the package scanning technology, the classes with data capture task annotations in the specified package are obtained, and the target data sources corresponding to the classes with data capture task annotations are determined;
[0021] For each class annotated with a data capture task, execute the data capture technology corresponding to the target data source to capture data.
[0022] In some implementations of the first aspect of the present application, the target data is data in a character string format.
[0023] A second aspect of the present application provides a data acquisition device, the device comprising:
[0024] A task annotation module is used to create multiple data acquisition sources, annotate the data capture tasks corresponding to each data acquisition source, and parameterize the target data source corresponding to each data acquisition source into the annotation of the data capture task corresponding to the data acquisition source;
[0025] The task execution module is used to execute data capture tasks based on annotations to obtain source data from a target data source through a data acquisition source, wherein data capture tasks with the same data source are executed in a merged manner.
[0026] A third aspect of the present application provides a data acquisition device, which includes a memory and a processor, and the processor is used to execute a program stored in the memory to run any data acquisition method provided in the first aspect of the present application.
[0027] The technical solution provided by this application has the following beneficial effects:
[0028] In the technical solution provided in the present application, first, multiple data acquisition sources are created, the data capture tasks corresponding to each data acquisition source are annotated, and the target data source corresponding to each data acquisition source is parameterized into the annotation of the data capture task corresponding to the data acquisition source; then, the data capture task is executed based on the annotation to obtain the source data from the target data source through the data acquisition source, wherein the data capture tasks with the same target data source are merged for execution. In the above scheme, the target data source is flexibly configured for the data acquisition task through annotations, and there is no need to manually write complex code to handle the differences between data sources. The system can automatically execute tasks according to the annotations. In addition, the data source is parameterized, and multiple data sources can share the same acquisition logic, avoiding the development of acquisition code for each data source separately, thereby reducing resource waste. In addition, if multiple data capture tasks collect data from the same data source, that is, the target data source is the same, the system will merge these tasks and use a single data connection to complete all tasks, avoiding repeated creation and data sources, saving network resources and server resources. BRIEF DESCRIPTION OF THE DRAWINGS
[0029] Figure 1 A flowchart of a data acquisition method provided in an embodiment of the present application;
[0030] Figure 2 A flowchart of another data acquisition method provided in an embodiment of the present application;
[0031] Figure 3 A flowchart of another data acquisition method provided in an embodiment of the present application;
[0032] Figure 4 A flowchart of another data acquisition method provided in an embodiment of the present application;
[0033] Figure 5 A schematic diagram of the process of data acquisition, conversion and loading provided in the embodiment of the present application;
[0034] Figure 6 A schematic diagram of the structure of a data acquisition device provided in an embodiment of the present application;
[0035] Figure 7 A structural schematic diagram of a data acquisition device provided in an embodiment of the present application. DETAILED DESCRIPTION
[0036] The following will be combined with the drawings in the embodiments of the present application to clearly and completely describe the technical solutions in the embodiments of the present application. Obviously, the described embodiments are only part of the embodiments of the present application, not all of the embodiments. Based on the embodiments in the present application, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of this application.
[0037] The embodiments of the present application provide a data acquisition method, device and equipment.
[0038] See also Figure 1 As shown, the embodiment of the present application provides a data acquisition method, which specifically includes the following steps:
[0039] S101: Create multiple data acquisition sources, annotate the data capture tasks corresponding to each data acquisition source, and parameterize the target data source corresponding to each data acquisition source into the annotation of the data capture task corresponding to the data acquisition source.
[0040] In an embodiment of the present application, it is first necessary to create respective data acquisition sources for different data sources. The data acquisition source refers to a component that can obtain data from a specific source, for example, obtaining data from different databases or interfaces such as MySQL, HTTP API, etc.; wherein each data acquisition source is associated with one or more data capture tasks, which is a data capture task associated with a certain data acquisition source, defining how to obtain data and the type of data source.
[0041] In addition, in order to reduce hard coding to reduce development complexity, the embodiment of the present application identifies each data capture task through annotation. Specifically, the present application annotates each data capture task so as to quickly locate the data capture task corresponding to each data acquisition source. At the same time, the embodiment of the present application also parameterizes the target data source expected to be collected into the annotation of each data capture task. Under this parameterized mechanism, the system can automatically identify the corresponding target data source and use a unified acquisition process to obtain the source data. It can be understood that each data acquisition source can correspond to one or more data sources, and the user can decide from which data sources to collect data based on actual needs. Therefore, the parameterized target data sources in the annotation of each data capture task can be the same or different.
[0042] In some implementations of the embodiments of the present application, in order to reduce the complexity of manually instantiating each data acquisition source and make the data acquisition process more flexible and automated, the step of creating multiple data acquisition sources in step S101 is specifically: obtaining the class that implements the data acquisition source interface in the specified package through the package scanning technology, and instantiating the class of the data acquisition source interface using the Java reflection technology to obtain multiple data acquisition sources.
[0043] Specifically, the process of creating multiple data collection sources in this application adopts package scanning technology and Java reflection technology. After the initialization of all tools and environments is completed at the program entry, first, all classes that implement the data collection source interface are searched from the specified package through the package scanning technology. The package scanning technology is a technology for automatically detecting class files in the package, which can identify which classes implement the specified interface or inherit a base class. Then, the Java reflection technology is used to automatically instantiate these classes that implement the data collection source interface, thereby generating multiple data collection sources; wherein, Java reflection technology is a technology that allows a program to dynamically access class information at runtime, and can instantiate class objects or call their methods. For example, suppose there are two different data collection sources, which are used to collect data from MySQL database and API interface respectively. Both data collection sources implement the data collection source interface. Through the package scanning technology, the system can automatically discover the classes that implement the interface, and then instantiate these classes through reflection technology to create the corresponding data collection source.
[0044] In some implementations of the embodiments of the present application, see Figure 2 As shown, the data capture task corresponding to each data acquisition source is annotated, and the target data source corresponding to each data acquisition source is parameterized into the annotation of the data capture task corresponding to the data acquisition source, specifically including the following steps:
[0045] S201: Identify the data capture tasks corresponding to each data collection source through Java annotation technology, and obtain the annotations corresponding to each data capture task.
[0046] In the embodiment of the present application, the annotation corresponding to the data capture task is also called the data capture task annotation.
[0047] S202: For each data acquisition source, determine the target data source corresponding to the data acquisition source, and parameterize the target data source corresponding to the data acquisition source into the annotation of the data capture task corresponding to the data acquisition source.
[0048] In an embodiment of the present application, for each data acquisition source, the target data source is determined according to actual needs, that is, the specific database, API interface or other data source is determined, and then the target data source information is bound to the annotation of the annotation capture task in a parameterized manner, and the data source is passed in the form of a variable parameter, and the target data source that is expected to be obtained is dynamically configured without modifying the code logic.
[0049] S102: Execute a data capture task based on the annotation to obtain source data from a target data source through a data acquisition source, wherein data capture tasks with the same target data source are executed in a merged manner.
[0050] In the embodiment of the present application, the system will identify and execute the data capture task according to the annotation in step S101, and at the same time, identify the target data source of each data capture task. The corresponding data capture task is automatically executed based on the annotation, and the source data is obtained from the target data source through the corresponding data acquisition source, thereby realizing the acquisition of multi-dimensional heterogeneous data, and supporting one acquisition program to collect data from multiple data sources.
[0051] During this process, in order to reduce the waste of server and network resources and improve the efficiency of data collection, when the system detects that the target data sources of multiple data capture tasks are the same, the data capture tasks with the same target data source will be merged and executed. Here, merged execution refers to multiple tasks with the same target data source. The system can establish a connection with the target data source only once, and then the data capture tasks with the same target data source share the connection, and each obtains its own data on demand, avoiding repeatedly establishing connections with the target data source.
[0052] In some implementations of the embodiments of the present application, see Figure 3 As shown, data capture tasks are performed based on annotations, including:
[0053] S301: Obtain the class with the data capture task annotation in the specified package through the package scanning technology, and determine the target data source corresponding to the class with the data processing task annotation.
[0054] In the embodiment of the present application, the system automatically scans all classes with data processing and capture annotations in the specified package directory through package scanning technology. These classes usually represent different data capture tasks, and the annotations contain relevant configuration information, including task names and target data sources. After the scan is completed, the system will determine the target data source corresponding to each class based on the information in the annotations, that is, where the class obtains data from.
[0055] S302: For each class with a data processing task annotation, execute the data capture technology corresponding to the target data source to capture data.
[0056] In the embodiment of the present application, each class with a data processing task annotation corresponds to a specific data source, and the system will automatically select the appropriate crawling technology to complete the data collection process based on the configuration information in the annotation. For example, if the target data source is a MySQL database, the system will use the binary file binlog to connect and crawl the data in the database; if the target data source is an HTTP API interface, the system will use a scheduled task to call the HTTPAPI to obtain data.
[0057] In this way, it is possible to select appropriate crawling technologies according to different data sources and automatically perform data collection tasks, avoiding developers from writing independent collection logic for each data source. At the same time, through parameterized data sources and automatically selected crawling technologies, developers do not need to repeatedly implement similar data crawling processes, thus saving development time and improving system efficiency.
[0058] exist Figure 1 In the process shown, first, multiple data acquisition sources are created, the data capture tasks corresponding to each data acquisition source are annotated, and the target data source corresponding to each data acquisition source is parameterized into the annotation of the data capture task corresponding to the data acquisition source; then, the data capture task is executed based on the annotation to obtain the source data from the target data source through the data acquisition source, wherein the data capture tasks with the same target data source are merged for execution. In the above scheme, the target data source is flexibly configured for the data acquisition task through annotations, and there is no need to manually write complex code to handle the differences between data sources. The system can automatically execute tasks according to the annotations. In addition, by parameterizing the data source, multiple data sources can share the same acquisition logic, avoiding the development of acquisition code for each data source separately, thereby reducing resource waste. In addition, if multiple data capture tasks collect data from the same data source, that is, the target data source is the same, the system will merge these tasks and use one data connection to complete all tasks, avoiding repeated creation and data sources, saving network resources and server resources.
[0059] See also Figure 4 As shown, the embodiment of the present application further provides a data acquisition method, which may specifically include the following steps:
[0060] S401: Create multiple data acquisition sources, annotate the data capture tasks corresponding to each data acquisition source, and parameterize the target data source corresponding to each data acquisition source into the annotation of the data capture task corresponding to the data acquisition source.
[0061] In the embodiment of the present application, the specific implementation of step S401 can refer to S101, and the present application will not elaborate on it here.
[0062] S402: Execute a data capture task based on the annotation to obtain source data from a target data source through a data acquisition source, wherein data capture tasks with the same target data source are executed in a merged manner.
[0063] In the embodiment of the present application, the specific implementation of step S402 can refer to S102, and the present application will not elaborate on it here.
[0064] S403: Perform data normalization on the source data to convert the source data into a target format.
[0065] In an embodiment of the present application, source data collected from different sources are subjected to data normalization processing and converted into a unified target format, thereby achieving standardization of multivariate heterogeneous data.
[0066] In an embodiment of the present application, data normalization is performed on the source data by converting the source data into a unified format.
[0067] In some implementations of the embodiments of the present application, the target format is a lightweight data exchange format JSON format.
[0068] Since the data formats of different data sources may be quite different, such as XML format, from date format, etc. By writing corresponding converters, the data in these formats are uniformly converted to JSON format, such as XML to JSON data converter, from date to JSON data converter, so that the subsequent data processing and loading steps can use a consistent data structure, thereby simplifying the data processing process.
[0069] S404: Distribute the source data converted into the target format to the corresponding target data processing task for data processing to obtain the target data.
[0070] In the embodiment of the present application, the source data that has been converted into the target format is distributed according to the target processing task of the data. Since each data processing task may correspond to different business requirements and processing logic, the data is accurately distributed to the data flow of the corresponding data processing task according to the association between different data sources and data processing tasks to obtain the target data, which is the data that the user ultimately expects to obtain.
[0071] Among them, data processing tasks such as data cleaning and data analysis can be set according to actual needs and will not affect the implementation of the embodiments of the present application.
[0072] S405: Load data based on target data.
[0073] In an embodiment of the present application, after completing the data processing task, the system stores the processed target data in the data writer to complete the data loading operation through the data writer.
[0074] In some implementations of the embodiments of the present application, the target data is data in a string format. That is, after the data processing task is completed, the target data is processed into a string format. Whether the data is obtained from complex structured data or unstructured data sources, after normalization and processing, it will be converted into a string format, and the data loading will be completed in a string format.
[0075] exist Figure 4 In the process shown, after the source data is obtained, first, the source data is normalized to convert the source data into the target format. Then, the source data converted into the target format is distributed to the corresponding target data processing task for data processing to obtain the target data. Finally, data loading is performed based on the target data. In this process, the standardization of multivariate heterogeneous data is completed, which simplifies the difficulty of subsequent data processing; the source data is distributed to the corresponding data processing task according to the data processing requirements to ensure that the expected data is obtained.
[0076] See also Figure 5 As shown, Figure 5 The entire process from program entry to data loading and successful completion of data storage is demonstrated.
[0077] First, the program entry starts and builds the system's operating environment and establishes the required environment configuration. After that, the data acquisition source is initialized.
[0078] Then, the program starts to use Java annotation technology and package scanning technology to obtain all classes that implement the data acquisition source interface under the specified package. At this time, Java reflection technology is used to automatically instantiate these classes and determine the target data source corresponding to each class through annotations. Once the scanned data acquisition source class is initialized, the program will feedback the status of initialization completion. At this time, each data acquisition source is ready and waiting to start the acquisition task.
[0079] After that, the acquisition-conversion-loading program is started to execute the data acquisition task to obtain source data from different data acquisition sources; the collected source data is normalized and converted into a unified format. The data in the unified format is then distributed to the corresponding data processing tasks to complete the data cleaning and conversion and obtain the target data.
[0080] Finally, the cleaned and converted target data is passed to the data writer, which is responsible for loading the data into the system's storage. After the data is successfully stored, the system will feedback the success status to the program entry to notify that the data loading is complete.
[0081] See also Figure 6 As shown, the present application provides a data acquisition device, which includes:
[0082] The task annotation module 601 is used to create multiple data acquisition sources, annotate the data capture tasks corresponding to each data acquisition source, and parameterize the target data source corresponding to each data acquisition source into the annotation of the data capture task corresponding to the data acquisition source;
[0083] The task execution module 602 is used to execute the data capture task based on the annotation to obtain source data from the target data source through the data acquisition source, wherein data capture tasks with the same data source are executed in a merged manner.
[0084] In some implementations of the embodiments of the present application, the device further includes:
[0085] The format conversion module is used to normalize the source data so as to convert the source data into a target format.
[0086] In some implementations of the embodiments of the present application, the target format is JSON format.
[0087] In some implementations of the embodiments of the present application, the device further includes:
[0088] The data processing module is used to distribute the source data converted into the target format to the corresponding target data processing task for data processing to obtain the target data;
[0089] The data loading module is used to load data based on target data.
[0090] In some implementations of the embodiments of the present application, multiple data collection sources are created, including:
[0091] The class implementing the data acquisition source interface in the specified package is obtained through the package scanning technology, and the class of the data acquisition source interface is instantiated using the Java reflection technology to obtain multiple data acquisition sources.
[0092] In some implementations of the embodiments of the present application, the data capture task corresponding to each data acquisition source is annotated, and the target data source corresponding to each data acquisition source is parameterized into the annotation of the data capture task corresponding to the data acquisition source, including:
[0093] The data capture tasks corresponding to each data collection source are identified by using Java annotation technology, and the annotations corresponding to each data capture task are obtained;
[0094] For each data acquisition source, determine the target data source corresponding to the data acquisition source, and parameterize the target data source corresponding to the data acquisition source into the annotation of the data capture task corresponding to the data acquisition source.
[0095] In some implementations of the embodiments of the present application, performing a data capture task based on annotations includes:
[0096] By using the package scanning technology, the classes with data capture task annotations in the specified package are obtained, and the target data sources corresponding to the classes with data capture task annotations are determined;
[0097] For each class annotated with a data capture task, execute the data capture technology corresponding to the target data source to capture data.
[0098] In some implementations of the embodiments of the present application, the target data is data in a string format.
[0099] like Figure 7 As shown, the embodiment of the present application further provides a data acquisition device, including: a memory 701, a processor 702;
[0100] Wherein, the memory 701 is used to store programs;
[0101] The processor 702 is used to execute the program in the memory to implement each step of the data acquisition method provided in the embodiment of the present application.
[0102] Finally, it should be noted that, in the embodiments of the present application, relational terms such as first and second, etc. are only used to distinguish one entity or operation from another entity or operation, and do not necessarily require or imply that there is any such actual relationship or order between these entities or operations. Moreover, the terms "include", "comprise" or any other variants thereof are intended to cover non-exclusive inclusion, so that the process, method, article or equipment including a series of elements includes not only those elements, but also includes other elements that are not clearly listed, or also includes elements inherent to such process, method, article or equipment. In the absence of more restrictions, the elements limited by the statement "comprise one..." do not exclude the presence of other identical elements in the process, method, article or equipment including the elements.
[0103] The above description of the disclosed embodiments enables those skilled in the art to implement or use the present invention. Various modifications to these embodiments will be apparent to those skilled in the art, and the general principles defined herein may be implemented in other embodiments without departing from the spirit or scope of the present invention. Therefore, the present invention will not be limited to the embodiments shown herein, but rather to the widest scope consistent with the principles and novel features disclosed herein.
Claims
1. A data acquisition method, characterized in that: The method comprises: Creating multiple data acquisition sources, annotating the data capture tasks corresponding to each of the data acquisition sources, and parameterizing the target data sources corresponding to each of the data acquisition sources into the annotations of the data capture tasks corresponding to the data acquisition sources; The data capture task is executed based on the annotation to obtain source data from the target data source through the data acquisition source, wherein the data capture tasks with the same target data source are merged for execution.
2. The method according to claim 1, characterized in that The method further comprises: The source data is normalized to convert the source data into a target format.
3. The method according to claim 2, characterized in that The target format is JSON format.
4. The method according to claim 2, characterized in that: The method further comprises: Distribute the source data converted into the target format to the corresponding target data processing task for data processing to obtain the target data; Data loading is performed based on the target data.
5. The method according to claim 1, characterized in that The creating of multiple data collection sources includes: The class implementing the data acquisition source interface in the specified package is obtained by using the package scanning technology, and the class implementing the data acquisition source interface is instantiated by using the Java reflection technology to obtain multiple data acquisition sources.
6. The method according to claim 1, characterized in that The step of annotating the data capture task corresponding to each of the data acquisition sources, and parameterizing the target data source corresponding to each of the data acquisition sources into the annotation of the data capture task corresponding to the data acquisition source, includes: Identify the data capture tasks corresponding to each of the data acquisition sources by using Java annotation technology, and obtain annotations corresponding to each of the data capture tasks; For each of the data acquisition sources, the target data source corresponding to the data acquisition source is determined, and the target data source corresponding to the data acquisition source is parameterized into the annotation of the data capture task corresponding to the data acquisition source.
7. The method according to claim 1, characterized in that The performing the data capture task based on the annotation comprises: By using a package scanning technology, a class with a data capture task annotation in a specified package is obtained, and the target data source corresponding to the class with the data capture task annotation is determined; For each class annotated with a data capture task, the data capture technology corresponding to the target data source is executed to capture data.
8. The method according to claim 4, characterized in that The target data is data in a character string format.
9. A data acquisition device, characterized in that: The device comprises: A task annotation module, used to create multiple data acquisition sources, annotate the data capture tasks corresponding to each of the data acquisition sources, and parameterize the target data source corresponding to each of the data acquisition sources into the annotation of the data capture tasks corresponding to the data acquisition sources; A task execution module is used to execute the data capture task based on the annotation to obtain source data from the target data source through the data acquisition source, wherein the data capture tasks with the same data source are executed in a merged manner.
10. A data acquisition device, characterized in that: The device includes a memory and a processor, and the processor is used to execute a program stored in the memory to run the data acquisition method according to any one of claims 1 to 8.