Data acquisition system and method

Abstract

The invention provides a data acquisition system and method, and relates to the technical field of big data. The data acquisition system is deployed on a big data cloud platform, a front-end page of the big data cloud platform is deployed on a client page of a system where a data source is located, and a user logs in the big data cloud platform to enter a data acquisition system page based on the front-end page and logs in the big data cloud platform to enter the data acquisition system page on the data acquisition system page. Through a data source configuration module, a mapping configuration module, an operation configuration module and an operation and maintenance module which are included in the data acquisition system, configuration and operation of a data acquisition task are realized, and meanwhile, through platformization of a one-stop data acquisition process for configuration and operation of the data acquisition task, the usability of the data acquisition system is improved.

Description

Technical Field

[0001] The present application relates to the field of big data technology, and in particular to a data acquisition system and method. Background Art

[0002] The big data cloud platform transforms the collected data into data assets, provides data capability components and operation mechanisms, forms the ability to aggregate data collection, integration, cleaning, processing, mining and analysis, and provides data to the outside in the form of shared services, thereby creating linkage with the business, quickly strengthening business needs, supporting data integration and sharing, analysis and mining, and data operations, and creating business value. At present, as the demand for data on the big data cloud platform becomes more and more extensive, users have more and more scenarios for the collection, processing and application of real-time data, and at the same time, the requirements for the richness of data content, data quality and data production cycle are becoming more and more stringent. As the entrance and beginning of all data on the big data cloud platform, data collection, the timeliness and quality of real-time data collection, the completeness and ease of use of data collection functions lay the foundation for all subsequent data processing operations in the big data cloud platform, and also determine the quality reliability and value of the real-time data analysis conclusions to a certain extent.

[0003] In the related art, the big data cloud platform collects data by combining the server and the data collection component, that is, when the user collects data, the use of the big data cloud platform and the use of the data collection client need to be combined. Specifically, by downloading the data collection component from the big data cloud platform, the data collection component is deployed on the server of the system where the data source is located, and after configuring the data collection task on the big data cloud platform page, the big data cloud platform pushes the data collection task to the data collection component, and further, when the server receives the instruction information sent by the user based on the page of the system where the data source is located, the data collection component is started based on the instruction information, so that the data collection component starts to execute the above data collection task, and pushes the collected data to the big data cloud platform, so that the big data cloud platform performs subsequent data processing operations on the collected data.

[0004] However, the data collection methods of the above-mentioned big data cloud platforms are less user-friendly. Summary of the invention

[0005] The present application provides a data collection system and method to solve the problem of low usability of data collection methods of big data cloud platforms in the prior art.

[0006] In a first aspect, the present application provides a data collection system, which is deployed on a big data cloud platform. A front-end page of the big data cloud platform is deployed on a client page of a system where a data source is located. A user enters a data collection system page by logging into the big data cloud platform based on the front-end page. The data collection system includes: a data source configuration module, a mapping configuration module, an operation configuration module, and an operation and maintenance module;

[0007] A data source configuration module is used to provide a data source configuration page, which is used to provide data source configuration and target data source configuration;

[0008] A mapping configuration module is used to configure the target library table corresponding to the selected target data source for the data management component according to the selected data source to create a data collection task. The data management component is deployed on the big data cloud platform;

[0009] The operation configuration module is used to provide an operation configuration page, which is used to provide operation resource configuration and data collection starting point configuration for the data collection task;

[0010] The operation and maintenance module is used to execute data collection tasks configured with operating resources based on the data collection starting point.

[0011] In one possible design, the data source configuration module includes a data source configuration unit and a target data source configuration unit; the data source configuration unit is used to provide a data source configuration page, which is used to provide an application system configuration to which the data source belongs, a data source connection string information configuration, and a data source type configuration; the target data source configuration unit is used to provide a target data source configuration page, which is used to provide a target data source configuration.

[0012] In a possible design, the mapping configuration module includes an interface mapping unit and a field mapping unit; the interface mapping unit is used to obtain design state metadata information from the docking architecture asset and architecture management platform according to the selected data source, and automatically match the physical library table corresponding to the design state metadata information according to the design state metadata information, and generate a target library table corresponding to the selected target data source based on the physical library table, and publish the target library table to the registration library table of the data management component; the field mapping unit is used to provide a field mapping page, which is used to provide data collection granularity configuration, and the level of data collection granularity includes the field level.

[0013] In a possible design, the run configuration page is also used to provide incremental data collection configuration, and incremental and full data collection configuration.

[0014] In one possible design, the data acquisition system also includes a verification configuration module; the verification configuration module is used to provide a verification switch configuration, and is also used to verify whether the collected data is accurate based on data verification rules when the verification switch is turned on, and write the collected data into an error queue in response to a verification of an error in the collected data.

[0015] In one possible design, the data collection system also includes a data view module; the data view module is used to display views corresponding to the data access scope, data access quality and data access timeliness related to the data collection task.

[0016] In a possible design, the data acquisition system also includes a task configuration module; the task configuration module is used to provide a task configuration page, and the task configuration page is used to provide a custom parameter configuration corresponding to the data acquisition task, and a collection task name configuration.

[0017] In a possible design, the operation and maintenance module is also used to provide one or more of collection task management configuration, alarm indicator configuration, instance management configuration, instance operation monitoring configuration, log viewing configuration, and operation analysis configuration.

[0018] In a second aspect, the present application provides a data collection method, which is applied to the data collection system provided in the first aspect, and the data collection method includes:

[0019] Display the data source configuration page, which is used to configure the data source and target data source; receive interactive operations acting on the data source configuration page, and obtain the data source and target data source; based on the data source, configure the target library table corresponding to the target data source for the data management component to create a data collection task, and the data management component is deployed on the big data cloud platform; configure running resources and the starting point of data collection for the data collection task; based on the data collection starting point, execute the data collection task configured with running resources.

[0020] In a third aspect, the present application provides a computer-readable storage medium, in which computer-executable instructions are stored. When a processor executes the computer-executable instructions, the data acquisition method provided in the second aspect above is implemented.

[0021] In a fourth aspect, the present application provides a computer program product, including a computer program, which, when executed by a processor, implements the data collection method provided in the second aspect above.

[0022] The data collection system and method provided by the present application are characterized in that the data collection system is deployed on a big data cloud platform, and the front-end page of the big data cloud platform is deployed on the client page of the system where the data source is located. Based on the front-end page, the user enters the data collection system page by logging into the big data cloud platform. On the data collection system page, the data source configuration module, mapping configuration module, operation configuration module and operation and maintenance module included in the data collection system are used to configure and run the data collection tasks. At the same time, the usability of the data collection system is improved by platformizing the one-stop data collection process for the configuration and operation of the data collection tasks. BRIEF DESCRIPTION OF THE DRAWINGS

[0023] The accompanying drawings, which are incorporated in and constitute a part of this specification, illustrate embodiments consistent with the present application and, together with the description, serve to explain the principles of the present application.

[0024] Figure 1 A schematic diagram of an application scenario applicable to the embodiments of the present application;

[0025] Figure 2 The system architecture of the data acquisition system provided in the embodiment of the present application Figure 1 ;

[0026] Figure 3 The system architecture of the data acquisition system provided in the embodiment of the present application Figure 2 ;

[0027] Figure 4 A flowchart of a data collection method provided in an embodiment of the present application.

[0028] The above drawings have shown clear embodiments of the present application, which will be described in more detail later. These drawings and text descriptions are not intended to limit the scope of the present application in any way, but to illustrate the concept of the present application to those skilled in the art by referring to specific embodiments. DETAILED DESCRIPTION

[0029] Exemplary embodiments will be described in detail herein, examples of which are shown in the accompanying drawings. When the following description refers to the drawings, the same numbers in different drawings represent the same or similar elements unless otherwise indicated. The implementations described in the following exemplary embodiments do not represent all implementations consistent with the present application. Instead, they are merely examples of devices and methods consistent with some aspects of the present application as detailed in the appended claims.

[0030] In the technical solution of this application, the collection, storage, use, processing, transmission, provision and disclosure of information such as financial data or user data involved shall comply with the provisions of relevant laws and regulations and shall not violate public order and good morals.

[0031] It should be noted that in the embodiments of the present application, certain software, components, models and other existing solutions in the industry may be mentioned, and they should be regarded as exemplary. Their purpose is only to illustrate the feasibility of implementing the technical solution of the present application, but it does not mean that the applicant has or will necessarily use the solution.

[0032] In related technologies, the big data cloud platform collects data by combining the server and data collection components. That is, when users collect data, they need to combine the use of the big data cloud platform with the use of the data collection client. The specific steps include:

[0033] Preparation: Download the data collection component from the big data cloud platform and deploy it on the server side of the system where the data source is located. After the deployment is complete, modify the corresponding configuration file on the server side and configure the network access relationship between the server side and the big data cloud platform.

[0034] Configure the data collection task: Based on the big data cloud platform page, configure the data collection task in a configurable manner. The parameters that need to be configured when configuring the data collection task include the source of the data to be collected and the target data source.

[0035] Start the data collection component: When the server receives the instruction information sent by the user based on the page of the system where the data source is located, the data collection component is started based on the instruction information, so that the data collection component starts to execute the configured data collection task;

[0036] Monitoring and logging: Use the data collection component to view log files during the execution of data collection tasks;

[0037] Data transmission: The configuration tool in the data collection component transmits the data to be collected to the target location according to the configured data collection task. The target location can be local storage, cloud storage, or sent to a remote server through the network;

[0038] Regular maintenance and updates: Regularly check the update and maintenance status of data collection components to ensure their normal operation and adapt to changes in business needs.

[0039] However, the above method of data collection based on data collection components has the following disadvantages:

[0040] 1) The threshold for using data collection components is high: The use of data collection components is complex and technically demanding. Configuring and maintaining data collection components requires certain technical knowledge. For non-technical personnel, additional training and learning costs may be required, which makes the threshold for using data collection components high. This leads to a low ease of use of data collection methods that deploy data collection components on the server side.

[0041] 2) Compatibility issues: There may be compatibility issues between different versions of clients. The data collection component may run normally on some systems, but needs to be adapted and adjusted on other systems.

[0042] 3) Data integrity and timeliness issues: There may be unexpected interruptions or errors in the data collection process, resulting in data loss or incompleteness;

[0043] 4) Resource consumption problem: The data collection component may require higher computing and storage resources to process real-time data, which may have a certain impact on the performance of the system. It is also impossible to isolate task resources, and there is a situation of resource preemption when deployed on the same server;

[0044] 5) Client operation state intervention problem: The big data cloud platform currently on the market that uses the client to implement data collection is only responsible for the client's information configuration, heartbeat information, etc., and has insufficient control over the resource consumption, collection speed and frequency of the client's operation state;

[0045] 6) Maintenance and update issues: With the continuous improvement of functions and changes in business needs, data collection components also need to be regularly maintained and updated, and data collection users need to invest additional time and energy for this.

[0046] In response to the technical problems existing in the relevant technologies, the present application proposes the following technical ideas: by building a data collection system based on a big data cloud platform, and deploying the front-end page of the big data cloud platform on the client page of the system where the data source is located, the user can enter the data collection system page by logging into the big data cloud platform based on the front-end page, and realize the configuration and operation of the data collection tasks through the data source configuration module, mapping configuration module, operation configuration module and operation and maintenance module included in the data collection system. At the same time, through the platformization of the one-stop data collection process for the configuration and operation of the data collection tasks, the usability of the data collection system is improved.

[0047] The technical solution of the present application and how the technical solution of the present application solves the above-mentioned technical problems are described in detail below with specific embodiments. The following specific embodiments can be combined with each other, and the same or similar concepts or processes may not be repeated in some embodiments. The embodiments of the present application will be described below in conjunction with the accompanying drawings.

[0048] The following first explains the terms involved in the embodiments of the present application:

[0049] Design metadata: Design metadata is metadata defined during the data system design and modeling phase, which provides information about data structures, definitions, specifications, and relationships. These metadata play a key role in the design period of data warehouses, databases, or other data management systems. Design metadata is the design blueprint of the data management system and provides the basis for the correct use and interpretation of data. It plays an important role throughout the data life cycle, helping data engineers, analysts, and other relevant personnel understand and operate data. The maintenance and management of design metadata is an important part of data governance.

[0050] Physical data sources: usually refer to the raw data actually stored in a computer system, database or other physical storage medium. Physical data sources are usually the starting point in the data pipeline and data integration process. Data engineers need to extract, transform and load data from these physical data sources to meet business needs.

[0051] The following first describes in detail the application scenarios of the embodiments of the present application.

[0052] Figure 1 Schematic diagram of application scenarios applicable to the embodiments of the present application. Figure 1 As shown, the application scenarios applicable to the embodiment of the present application include a client 11 and a server 12. Among them, the client 11 is deployed with a system 13 where the data source is located, and the server 12 is deployed with a big data cloud platform 14.

[0053] Exemplarily, the client 11 and the server 12 are communicatively connected via a wired network or a wireless network.

[0054] Exemplarily, the client specifically includes but is not limited to various personal computers, laptops, smart phones, tablet computers, portable wearable devices, etc.

[0055] The data acquisition system provided in the embodiment of the present application is deployed on the big data cloud platform 14, and the front-end page of the big data cloud platform is also deployed in the client 11. Based on the front-end page, the user enters the page of the data acquisition system provided in the embodiment of the present application by logging into the big data cloud platform 14, and configures the data acquisition task and executes the data acquisition task based on the functional modules deployed in the page of the data acquisition system, such as the data source configuration module, the mapping configuration module, the operation configuration module, and the operation and maintenance module.

[0056] The data acquisition system and method provided in the embodiments of the present application are described in detail below in conjunction with specific embodiments.

[0057] It should be noted that in the data acquisition system provided in the embodiment of the present application, the data acquisition system is deployed on a big data cloud platform, and the front-end page of the big data cloud platform is deployed on the client page of the system where the data source is located. Based on the front-end page, the user enters the data acquisition system page by logging into the big data cloud platform.

[0058] It should be noted that the data acquisition system provided in the embodiment of the present application may be a real-time data acquisition system.

[0059] Figure 2 The system architecture of the data acquisition system provided in the embodiment of the present application Figure 1 .like Figure 2 As shown, the data collection system includes a data source configuration module 210 , a mapping configuration module 220 , an operation configuration module 230 and an operation and maintenance module 240 .

[0060] The data source configuration module 210 is used to provide a data source configuration page, and the data source configuration page is used to provide data source configuration and target data source configuration;

[0061] A mapping configuration module 220 is used to configure a target library table corresponding to the selected target data source for the data management component according to the selected data source to create a data collection task. The data management component is deployed on the big data cloud platform;

[0062] The operation configuration module 230 is used to provide an operation configuration page, which is used to provide operation resource configuration and data collection starting point configuration for the data collection task;

[0063] The operation and maintenance module 240 is used to execute the data collection task configured with the operation resources based on the data collection starting point.

[0064] Exemplarily, the data source, that is, the location where the data is collected, may be different application systems, or different database tables under any application system, etc.

[0065] Exemplarily, the target data source may be a storage location of the collected data. The type of the target data source may be a high-throughput distributed publish-subscribe messaging system.

[0066] A possible implementation of configuring the data source and the target data source based on the data source configuration page may be to select one or more data sources through the data source control in the data source configuration page, and to select one or more target data sources through the target data source control in the data source configuration page.

[0067] In the data acquisition system provided in the embodiment of the present application, there is no limitation on the number of configured data sources and target data sources, and the specific number can be determined according to actual application requirements.

[0068] In the embodiment of the present application, the configuration and operation of data collection tasks are realized through the data source configuration module, mapping configuration module, operation configuration module and operation and maintenance module included in the data collection system. At the same time, the usability of the data collection system is improved by platformizing the one-stop data collection process for the configuration and operation of data collection tasks.

[0069] Optionally, the data source configuration module includes a data source configuration unit and a target data source configuration unit. The data source configuration unit is used to provide a data source configuration page, which is used to provide the application system configuration to which the data source belongs, the data source connection string information configuration, and the data source type configuration; the target data source configuration unit is used to provide a target data source configuration page, which is used to provide the target data source configuration.

[0070] Exemplarily, the data source connection string information may be data information in a physical library table corresponding to the data source, such as a user name and password.

[0071] Exemplarily, data source types include various database types, message queues, front-end tracking points, back-end tracking point data, logs, and files.

[0072] It should be noted that, compared with the related art, for a configured data collection task, only one corresponding target data source can be configured. In the data collection system provided in the embodiment of the present application, multiple target data sources are configured to achieve cross-domain multiple data transmission. That is, when only one data collection task is configured, multiple target data sources can be configured for the data collection task, thereby achieving the ability to collect data from multiple target data sources from the same data source, thereby enhancing data availability and business flexibility.

[0073] Optionally, the mapping configuration module includes an interface mapping unit and a field mapping unit; the interface mapping unit is used to obtain design state metadata information from the docking architecture assets and the architecture management platform according to the selected data source, and automatically match the physical library table corresponding to the design state metadata information according to the design state metadata information, and generate a target library table corresponding to the selected target data source based on the physical library table, and publish the target library table to the registration library table of the data management component; the field mapping unit is used to provide a field mapping page, which is used to provide data collection granularity configuration, and the level of data collection granularity includes the field level.

[0074] Exemplarily, the docking architecture asset and architecture management platform is a system independent of the big data cloud platform. The docking architecture asset and architecture management platform stores design state metadata information corresponding to different physical libraries and tables.

[0075] Exemplarily, the physical library table may be a table structure containing physical data sources.

[0076] Taking financial institutions as an example, in one possible implementation method, based on the relevant provisions and processes of the financial institution's docking architecture assets and architecture management on data governance, that is, the specifications for library and table design, the design-state metadata information corresponding to the physical library table that registers and stores the data source on the docking architecture assets and architecture management platform. The design-state metadata information is based on the relevant provisions and processes of the financial institution's docking architecture assets and architecture management on data governance, that is, the specifications for library and table design, and is obtained according to information such as the name of the physical library table, the fields contained in the physical library table, and the length of the field.

[0077] Exemplarily, a possible implementation method of publishing the target library table to the registration library table of the data management component may be: based on a one-key publishing control in the interface mapping page, publishing the target library table to the registration library table of the data management component.

[0078] It is understandable that, compared with the related art, the real-time data collection of the big data cloud platform is difficult to identify the basic characteristics and attributes of real-time data, and the metadata table registration process and the collection task creation process are out of touch. In the data collection system provided by the embodiment of the present application, by introducing the docking architecture asset and the architecture management platform, the problem of the metadata table registration process and the collection task creation process being out of touch can be solved based on the mapping configuration module.

[0079] It can be understood that compared with the related art, which requires manual pre-construction of the target library table corresponding to the target data source, the data acquisition system provided in the embodiment of the present application implements the metadata parsing function based on the interface mapping unit, can perform consistency comparison between the designed metadata information and the actual physical metadata information, and supports one-click persistence of the metadata information to the registration library table of the data management component, which can realize system automation processing and generate the target library table corresponding to the target data source.

[0080] It can be understood that, compared with the related art, the data collection granularity supported by the big data cloud platform only includes the library table level, and the granularity of metadata management of the collected data is relatively coarse, which is not conducive to the subsequent data governance work. In the data collection system provided by the embodiment of the present application, the data collection granularity supported by the data collection system can be the field level, that is, the fields in the physical library table can be queried. Based on the data collection granularity being the field level, it has the following beneficial effects: on the one hand, it supports the comparison of the design state metadata information with the actual physical state metadata information, and generates a verification of the consistency between the design state and the test state. In terms of data governance, it can be verified at the data collection end whether the library table actually used is consistent with the library table specification in the architecture management specification of the docking architecture asset and the architecture management platform, and report the indicator as one of the standards for data quality inspection to generate a data quality view, which helps the development of architecture data governance; on the other hand, it supports data collection granularity to the field level, and users support configuration to collect only some fields in the table during the configuration stage, avoiding performance problems in the large library table collection process, and only collecting the required fields, reducing the amount of data transmission, improving efficiency, especially in network transmission and storage costs. Convenient and more optimized.

[0081] Optionally, the run configuration page is also used to provide incremental data collection configuration, and incremental and full data collection configuration.

[0082] Exemplarily, the run configuration page is also used to provide acquisition configuration at a specified time point.

[0083] It can be understood that the data collection system provided in the embodiment of the present application also supports one or more configurations of incremental collection, full + incremental collection, and collection at a specified time point by running the configuration module to meet a variety of data collection operation scenarios.

[0084] Optionally, the data acquisition system provided in the embodiment of the present application also includes a verification configuration module, which is used to provide a verification switch configuration and to verify whether the collected data is accurate based on data verification rules when the verification switch is turned on. In response to a verification of an error in the collected data, the collected data is written into an error queue.

[0085] It is understandable that, compared with the lack of a verification mechanism in the related art, data with quality problems are still collected during the data collection process, and data quality problems are exposed during the data processing process, increasing the difficulty and workload of correction. In the data collection system provided in the embodiment of the present application, a configurable data verification switch is supported, and verification of the quality of collected data is supported. By verifying the collected data during the data collection process, and when errors are found in the collected data, the collected data is written into an error queue, thereby improving the quality of the collected data.

[0086] Optionally, the data collection system provided in the embodiment of the present application further includes a data view module, which is used to display views corresponding to data access scope, data access quality, and data access timeliness related to the data collection task.

[0087] Exemplarily, the data access scope may be a system where the data to be collected contained in the data collection task is located.

[0088] In a possible implementation, the data view corresponding to the data access scope may include the number of systems where the data to be collected is located, and the number of library tables corresponding to each system.

[0089] Exemplarily, the data in the data view corresponding to the data access range may be displayed in the form of percentage.

[0090] Exemplarily, the data access quality may be a display of the quality of collected data, or may be a display of consistency indicators of reported data.

[0091] In a possible implementation, based on the checking module, the data of correct collected data and the data of erroneous collected data are counted, and the data of correct collected data and the data of erroneous collected data are displayed in the form of percentage.

[0092] In another possible implementation, the design state metadata information is compared with the actual physical state metadata information to generate a verification of the consistency between the design state and the test state, and the consistency rate between the design state and the test state is statistically calculated in the form of a percentage. The consistency rates of different systems and each system are displayed through a data access quality view.

[0093] Exemplarily, the data access timeliness may be data collection speed, data collection peak value, data delay time, daily data access range, etc.

[0094] Exemplarily, the data collection peak value may be the maximum amount of data that can be collected per second.

[0095] Exemplarily, the daily data access range may be the data size, the number of data items, and the number of database tables covered by the data to be collected.

[0096] It is understandable that the data acquisition system provided in the embodiment of the present application realizes the full-process management function from metadata design to data acquisition entering the big data cloud platform. Therefore, the whole process control can be realized for the process from data design generation to actual acquisition. Compared with the related technology, it is difficult to ensure that the test state and production state data are used completely in accordance with the metadata specifications of the design state, and the lack of effective monitoring view intervention is not conducive to the subsequent data governance work. In the data acquisition system provided in the embodiment of the present application, by forming data view management for data access scope, data access timeliness, and data access quality, it can provide a basis and support for docking architecture assets and architecture management platforms to achieve subsequent data governance.

[0097] Optionally, the data acquisition system provided in the embodiment of the present application also includes a task configuration module, which is used to provide a task configuration page, and the task configuration page is used to provide a custom parameter configuration corresponding to the data acquisition task, and a collection task name configuration.

[0098] Exemplarily, the custom parameters may be a polling period for data collection and a maximum or minimum throughput per second, etc.

[0099] It is understandable that, compared with some system default configuration parameters corresponding to data acquisition tasks, in the data acquisition system provided in the embodiment of the present application, custom parameters can be configured based on the task configuration page according to actual application requirements.

[0100] Optionally, the operation and maintenance module is also used to provide one or more functions of collection task management configuration, alarm indicator configuration, instance management configuration, instance operation monitoring configuration, log viewing configuration and operation analysis configuration.

[0101] It can be understood that the operation and maintenance module in the data acquisition system provided in the embodiment of the present application supports one-stop completion of instance operation status monitoring, log viewing, scheduling analysis, task start and stop and other operations within the system, without the need to view logs and start and stop tasks through the client, providing users with a visual, one-stop data development experience.

[0102] Figure 3 The system architecture of the data acquisition system provided in the embodiment of the present application Figure 2 .like Figure 3 As shown, the data collection system includes a data source configuration module 210 , a mapping configuration module 220 , an operation configuration module 230 , an operation and maintenance module 240 , a verification configuration module 250 , a data view module 260 and a task configuration module 270 .

[0103] Among them, the specific functions of the data source configuration module, mapping configuration module, operation configuration module, operation and maintenance module, verification configuration module, data view module and task configuration module are similar to those described above and will not be repeated here.

[0104] Exemplarily, the design state in the interface mapping unit and the field mapping unit is used to represent the design state metadata information, and the target state is used to represent the target library table corresponding to the target data source.

[0105] It can be understood that the data collection task provided in the embodiment of the present application realizes one-stop configuration, operation, instance management, and monitoring of the page, abandons the data collection component, lowers the user usage threshold, solves various problems existing in the use of the data collection component, and improves the availability of the data collection system.

[0106] In summary, the beneficial effects that can be achieved by the data acquisition system provided in the embodiments of the present application are summarized below.

[0107] The data collection system based on the big data cloud platform supporting full-link management built in the embodiment of the present application has the following advantages:

[0108] 1) It simplifies the data collection process, lowers the threshold for using the data collection system, and improves the usability of the data collection system. At the same time, users only need to log in to the big data cloud platform to enter the data collection system page and start the data collection task after performing standardized configuration, which greatly simplifies the use process of the data collection system;

[0109] 2) Improve the flexibility and scalability of the data collection system, so that it can adapt to diverse data sources and environments. When a new data source is added, there is no need to install a client, but only to connect to the big data cloud platform and configure the corresponding interface;

[0110] 3) Reduce the maintenance cost of the data collection system. The unified implementation of the data collection platform maintenance is more centralized. It is no longer necessary to perform independent maintenance and updates for each data source client, which helps to reduce the operation and maintenance workload and improve the stability of the data collection platform;

[0111] 4) Improve data security. The unified data collection platform without client can ensure data security through standard security protocols and authentication mechanisms. The data collection platform can centrally manage and encrypt data transmission, reducing security risks caused by decentralized clients.

[0112] 5) Supporting cross-domain data collection, the unified data collection platform without the need for a client is easier to integrate with cloud computing or edge computing, and can better adapt to distributed and cross-regional data collection needs;

[0113] 6) Promote data governance, from the original data collection tool to a data management platform with data collection as the entry point, to achieve centralized management of data collection on a unified platform, and to make it easier to implement data governance strategies, including metadata management and data collection quality management, thereby improving data consistency, accuracy and credibility;

[0114] 7) Easy-to-use management and monitoring tools. The data collection system built on the big data cloud platform supports intuitive and easy-to-use management and monitoring tools. For data monitoring parties, they can intuitively manage and operate data through views such as data quality and data timeliness provided by the collection system.

[0115] Based on the data acquisition system provided in the above embodiment, the implementation method of the data acquisition method provided in the embodiment of the present application is described in detail below in combination with specific embodiments.

[0116] Figure 4 Schematic diagram of the flow of the data acquisition method provided in the embodiment of the present application. The data acquisition method is applied to the data acquisition system provided in the above embodiment. Figure 4 As shown, the specific implementation of the data collection method includes the following steps:

[0117] S401, displaying a data source configuration page, where the data source configuration page is used to configure a data source and a target data source.

[0118] In one possible implementation, the user enters the data collection system page by logging into the big data cloud platform based on the front-end page deployed by the big data cloud platform on the client page of the system where the data source is located, and displays the data source configuration page based on the data collection system page.

[0119] The data source and target data source are similar to those described above and will not be described here.

[0120] S402, receiving an interactive operation on a data source configuration page, and obtaining a data source and a target data source.

[0121] Exemplarily, the data source configuration page may include a data source control and a target data source control.

[0122] In a possible implementation, an interactive operation performed by a user on a data source control in a data source configuration page is received to obtain the data source.

[0123] In a possible implementation, an interactive operation performed by a user on a target data source control in a data source configuration page is received to obtain the target data source.

[0124] Exemplarily, the number of data sources and target data sources may be one or more. The embodiment of the present application does not limit the number of data sources and target data sources, which may be determined according to actual needs.

[0125] S403, based on the data source, configure the target library table corresponding to the target data source for the data management component to create a data collection task, and the data management component is deployed on the big data cloud platform.

[0126] In one possible implementation, based on the library table name of the data source, the design state metadata information corresponding to the data source is obtained from the docking architecture asset and architecture management platform, and by parsing the design state metadata information, the design state metadata information is automatically matched to the physical library table corresponding to the design state metadata information, and further the library table name and structure and other parameters of the physical library table are used to generate the target library table corresponding to the target data source, and the target library table is published to the data management component based on the one-key publishing control, thereby completing the creation of the data collection task.

[0127] Exemplarily, the same data collection task may include multiple target data sources to achieve distributed, cross-regional data collection.

[0128] S404: configure operating resources and a data collection starting point for the data collection task.

[0129] It is understandable that a created data collection task can be executed only when it is configured with execution resources.

[0130] S405: Execute the data collection task configured with the running resources based on the data collection starting point.

[0131] In a possible implementation, based on the data collection starting point, data is collected from a data source configured in a data collection task configured with running resources, and the collected data is transmitted to a target data source configured in the data collection task.

[0132] Exemplarily, when a data collection task includes multiple target data sources, the collected data may be simultaneously transmitted to different target data sources configured in the data collection task.

[0133] In the embodiment of the present application, by displaying a data source configuration page for configuring the data source and the target data source, receiving interactive operations acting on the data source configuration page, obtaining the data source and the target data source, and configuring the target library table corresponding to the target data source for the data management component deployed on the big data cloud platform based on the data source, so as to create a data collection task, further configuring the running resources and the data collection starting point for the data collection task, and executing the data collection task configured with the running resources based on the data collection starting point. In the embodiment of the present application, through the one-stop data collection of configuration and operation, the big data cloud platform supports data collection of full-link management.

[0134] It should be noted that the data collection method provided in the embodiment of the present application can be real-time data collection, periodic data collection, data collection at a specified time point, etc. The data collection method provided in the embodiment of the present application does not limit the data collection method, which can be determined according to the time application requirements.

[0135] Optionally, in the data collection method provided in the embodiment of the present application, the data collection granularity can be field level. In the data collection process, by setting only the required fields to be collected, the data transmission volume is reduced and the efficiency is improved, especially in terms of network transmission and storage costs.

[0136] Optionally, in the data collection method provided in the embodiment of the present application, during the data collection process, it also includes checking the collected data to achieve quality checking of the collected data.

[0137] In a possible implementation, when the check switch is turned on, the accuracy of the collected data is checked based on the data check rule, and in response to the collected data being checked to be wrong, the collected data is written into the error queue.

[0138] Optionally, the data collection method provided in the embodiment of the present application further includes: displaying views corresponding to data access scope, data access quality and data access timeliness related to the data collection task in the form of data views.

[0139] The data access scope, data access quality and data access timeliness are similar to those described above and will not be elaborated here.

[0140] An embodiment of the present application also provides a computer program product, including a computer program, which, when executed by a processor, implements the technical solution of the data acquisition method in the above embodiment.

[0141] An embodiment of the present application also provides a chip for running instructions, which is used to execute the technical solution of the data acquisition method in the above embodiment.

[0142] An embodiment of the present application further provides a computer-readable storage medium, in which computer-executable instructions are stored. When a processor executes the computer-executable instructions, the technical solution of the data collection method in the above embodiment is implemented.

[0143] The above-mentioned readable storage medium can be implemented by any type of volatile or non-volatile storage device or a combination thereof, such as static random access memory (SRAM), electrically erasable programmable read-only memory (EEPROM), erasable programmable read-only memory (EPROM), programmable read-only memory (PROM), read-only memory (ROM), magnetic storage, flash memory, magnetic disk or optical disk. The readable storage medium can be any available medium that can be accessed by a general or special-purpose computer.

[0144] An exemplary readable storage medium is coupled to a processor so that the processor can read information from the readable storage medium and write information to the readable storage medium. Of course, the readable storage medium can also be a component of the processor. The processor and the readable storage medium can be located in an application specific integrated circuit (Application Specific Integrated Circuits, referred to as ASIC). Of course, the processor and the readable storage medium can also exist in the device as discrete components.

[0145] The division of units is only a logical function division, and there may be other divisions in actual implementation, such as multiple units or components can be combined or integrated into another system, or some features can be ignored or not executed. Another point is that the mutual coupling or direct coupling or communication connection shown or discussed can be an indirect coupling or communication connection through some interface, device or unit, which can be electrical, mechanical or other forms.

[0146] The units described as separate components may or may not be physically separated, and the components shown as units may or may not be physical units, that is, they may be located in one place or distributed on multiple network units. Some or all of the units may be selected according to actual needs to achieve the purpose of the solution of this embodiment.

[0147] In addition, each functional unit in each embodiment of the present invention may be integrated into one processing unit, or each unit may exist physically separately, or two or more units may be integrated into one unit.

[0148] If the function is implemented in the form of a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present invention, or the part that contributes to the prior art or the part of the technical solution, can be embodied in the form of a software product. The computer software product is stored in a storage medium, including several instructions for a computer device (which can be a personal computer, a server, or a network device, etc.) to perform all or part of the steps of the methods of each embodiment of the present invention. The aforementioned storage medium includes: U disk, mobile hard disk, read-only memory (Read-Only Memory, referred to as ROM), random access memory (Random Access Memory, referred to as RAM), disk or optical disk and other media that can store program codes.

[0149] Those skilled in the art can understand that all or part of the steps of implementing the above-mentioned method embodiments can be completed by hardware related to program instructions. The aforementioned program can be stored in a computer-readable storage medium. When the program is executed, the steps of the above-mentioned method embodiments are executed; and the aforementioned storage medium includes: ROM, RAM, disk or optical disk and other media that can store program codes.

[0150] Finally, it should be noted that those skilled in the art will readily conceive of other embodiments of the present invention after considering the specification and practicing the invention disclosed herein. The present invention is intended to cover any variations, uses or adaptations of the present invention, which follow the general principles of the present invention and include common knowledge or customary technical means in the art not disclosed by the present invention, are not limited to the precise structure described above and shown in the drawings, and may be modified and changed in various ways without departing from the scope thereof. The scope of the present invention is limited only by the appended claims.

Claims

1. A data acquisition system, characterized in that: The data acquisition system is deployed on the big data cloud platform. The front-end page of the big data cloud platform is deployed on the client page of the system where the data source is located. Based on the front-end page, the user logs into the big data cloud platform to enter the data acquisition system page. The data acquisition system includes: a data source configuration module, a mapping configuration module, a running configuration module and an operation and maintenance module; The data source configuration module is used to provide a data source configuration page, and the data source configuration page is used to provide data source configuration and target data source configuration; The mapping configuration module is used to configure the target library table corresponding to the selected target data source for the data management component according to the selected data source to create a data collection task, and the data management component is deployed on the big data cloud platform; The operation configuration module is used to provide an operation configuration page, and the operation configuration page is used to provide operation resource configuration and data collection starting point configuration for the data collection task; The operation and maintenance module is used to execute the data collection task configured with operation resources based on the data collection starting point.

2. The data acquisition system according to claim 1, characterized in that: The data source configuration module includes a data source configuration unit and a target data source configuration unit; The data source configuration unit is used to provide a data source configuration page, wherein the data source configuration page is used to provide an application system configuration to which the data source belongs, a data source connection string information configuration, and a data source type configuration; The target data source configuration unit is used to provide a target data source configuration page, and the target data source configuration page is used to provide target data source configuration.

3. The data acquisition system according to claim 1, characterized in that: The mapping configuration module includes an interface mapping unit and a field mapping unit; The interface mapping unit is used to obtain design state metadata information from the docking architecture asset and the architecture management platform according to the selected data source, and automatically match the physical library table corresponding to the design state metadata information according to the design state metadata information, generate a target library table corresponding to the selected target data source based on the physical library table, and publish the target library table to the registration library table of the data management component; The field mapping unit is used to provide a field mapping page, and the field mapping page is used to provide data collection granularity configuration, and the level of data collection granularity includes the field level.

4. The data acquisition system according to claim 1, characterized in that: The operation configuration page is also used to provide incremental data collection configuration, and incremental and full data collection configuration.

5. The data acquisition system according to any one of claims 1 to 4, characterized in that: The data acquisition system also includes a verification configuration module; The check configuration module is used to provide a check switch configuration, and is also used to check whether the collected data is accurate based on data check rules when the check switch is turned on, and write the collected data into an error queue in response to checking that the collected data is wrong.

6. The data acquisition system according to any one of claims 1 to 4, characterized in that: The data acquisition system also includes a data view module; The data view module is used to display views corresponding to the data access scope, data access quality and data access timeliness related to the data collection task.

7. The data acquisition system according to any one of claims 1 to 4, characterized in that: The data acquisition system also includes a task configuration module; The task configuration module is used to provide a task configuration page, and the task configuration page is used to provide a custom parameter configuration corresponding to the data collection task and a collection task name configuration.

8. The data acquisition system according to any one of claims 1 to 4, characterized in that: The operation and maintenance module is also used to provide one or more of collection task management configuration, alarm indicator configuration, instance management configuration, instance operation monitoring configuration, log viewing configuration and operation analysis configuration.

9. A data collection method, characterized in that: Applied to the data acquisition system according to any one of claims 1 to 8, the data acquisition method comprises: Display a data source configuration page, where the data source configuration page is used to configure the data source and target data source; Receiving an interactive operation on the data source configuration page, and obtaining a data source and a target data source; Based on the data source, configuring the target library table corresponding to the target data source for the data management component to create a data collection task, wherein the data management component is deployed on the big data cloud platform; Configuring operating resources and a data collection starting point for the data collection task; Based on the data collection starting point, a data collection task configured with operating resources is executed.

10. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores computer-executable instructions, which are used to implement the method according to claim 9 when executed by a processor.

11. A computer program product, characterized in that The invention comprises a computer program, which implements the method of claim 9 when being executed by a processor.

Images