Data acquisition method, data management system, data acquisition device and computer equipment

Abstract

The embodiment of the present application relates to the technical field of data processing, and discloses a data acquisition method, which comprises the following steps: receiving a data source acquisition event notification sent by a data source reservation system, wherein the data source acquisition event notification carries a data source ID; obtaining encrypted data source information from a data source storage management system according to the data source ID and decrypting the encrypted data source information to obtain decrypted data source information; the encrypted data source information is obtained by pre-encrypting the data source information and the data source ID sent by the data source reservation system according to the data source storage management system; and performing data acquisition according to the decrypted data source information. Through the above method, the embodiment of the present application realizes effective protection of data source information in the data acquisition process.

Description

TECHNICAL FIELD

[0001] Embodiments of the present application relate to the field of data processing, in particular to a data collection method, a data management system, a data collection device, a computer device and a computer readable storage medium. BACKGROUND

[0002] In the communication field, the management of wireless network / transmission network / bearer network / core network and the like is generally based on network element OMC of equipment manufacturers to realize single manufacturer domain management, and based on fault management / performance management / capital management / configuration management and the like network management system to realize end-to-end management across manufacturers. The network management system needs to collect resource, performance, alarm and the like data from the OMC of each manufacturer through FTP / Socket / Corba / database and the like various interface forms, and processes and processes based on the collected data to realize alarm monitoring, performance analysis / optimization, resource life cycle management, configuration verification and the like network management functions.

[0003] However, the inventors have found in the implementation of embodiments of the present application that in the current data collection scheme, the information security problem of OMC data source information (name, IP, account password and the like) is generally ignored, and there is a large risk of sensitive information leakage in the whole process. SUMMARY

[0004] In view of the above problems, embodiments of the present application provide a data collection method, a data management system, a data collection device, a computer device and a computer readable storage medium, to solve the technical problem that sensitive information is easily leaked in the data collection process in the prior art.

[0005] According to an aspect of an embodiment of the present application, a data collection method is provided, the method comprising:

[0006] receiving a data source collection event notification sent by a data source reporting system, the data source collection event notification carrying a data source ID;

[0007] According to the data source ID, encrypted data source information is obtained from a data source storage management system and decrypted to obtain decrypted data source information; the encrypted data source information is obtained by the data source storage management system by pre-encrypting the data source information and data source ID sent by the data source reporting system;

[0008] According to the decrypted data source information, data collection is performed.

[0009] In an alternative manner, before the step of obtaining the encrypted data source information from the data source storage management system according to the data source ID and decrypting the encrypted data source information to obtain decrypted data source information, the method further comprises: obtaining an agent client software tool development package from the data source storage management system in advance and integrating the agent client software tool development package in a code; and establishing a data connection with the data source storage management system through the agent client and an access credential, wherein the access credential comprises an identity key and a secret key.

[0010] In an alternative manner, the step of obtaining the encrypted data source information from the data source storage management system according to the data source ID and decrypting the encrypted data source information to obtain decrypted data source information comprises: initiating a data source obtaining request to the data source storage management system according to the data source ID; the data source obtaining request comprises the data source ID; the data source storage management system obtains the encrypted data source information and a secret key encrypted by a secret key from the data source storage management system according to the data source obtaining request through the agent client; the agent client decrypts the secret key by using the secret key and decrypts the encrypted data source information according to the secret key to obtain decrypted data source information; and the encrypted data source information is data source information encrypted by a public key.

[0011] In an alternative manner, the step of obtaining the encrypted data source information from the data source storage management system according to the data source obtaining request through the agent client comprises: the agent client calculates a data signature of the data source obtaining request according to the secret key; and the agent client sends the data source ID, the data signature and the access credential to the data source storage management system, so that the data source storage management system verifies the access credential and the data signature and returns the encrypted data source information and the secret key encrypted by the secret key.

[0012] According to another aspect of the embodiment of the present application, a data collection method is provided, and the method comprises:

[0013] receiving encrypted data source information and a corresponding data source ID sent by a data source reservation system;

[0014] storing the encrypted data source information and the corresponding data source ID;

[0015] receiving a data source obtaining request sent by a data collection system and verifying the data source obtaining request, wherein the data source obtaining request comprises the data source ID;

[0016] When the verification passes, the encrypted data source information and the private key encrypted by the key are sent to the data collection system, so that the data collection system decrypts the encrypted data source information to obtain decrypted data source information, and collects data according to the decrypted data source information.

[0017] According to another aspect of the embodiments of the present application, a data management system is provided, which comprises a data source reservation system, a data collection system and a data source storage management system.

[0018] The data source reservation system is configured to obtain data source information input by a user, and generate a corresponding data source ID and event notification; and further configured to encrypt the data source information and the data source ID by a public key, and send the encrypted data source information and the data source ID to the data source storage management system.

[0019] The data source storage management system is configured to receive and store the encrypted data source information and the data source ID sent by the data source reservation system.

[0020] The data source reservation system is further configured to send a data source collection event notification to the data collection system, and the data source collection event notification carries a data source ID.

[0021] The data collection system is configured to obtain the encrypted data source information from the data source storage management system according to the data source collection event notification, and decrypt the encrypted data source information to obtain decrypted data source information; and further configured to collect data according to the decrypted data source information.

[0022] In an optional manner, the data source reservation system is further configured to send a data source collection event notification to the data collection system, and the data source collection event notification carries a data source ID, which comprises: initiating a data source obtaining request to the data source storage management system according to the data source ID; the data source obtaining request comprises the data source ID; the data source storage management system obtains the encrypted data source information and the private key encrypted by the key from the data source storage management system according to the data source obtaining request through a proxy client; the private key is obtained by decrypting the key through the proxy client, and the encrypted data source information is decrypted according to the private key to obtain decrypted data source information; the encrypted data source information is data source information encrypted by a public key.

[0023] According to another aspect of the embodiments of the present application, a data collection device is provided, which comprises:

[0024] The receiving module is configured to receive a data source collection event notification sent by a data source reservation system, and the data source collection event notification carries a data source ID.

[0025] decrypting module, configured to acquire the encrypted data source information from a data source storage management system according to the data source ID and decrypt the encrypted data source information to obtain decrypted data source information, wherein the encrypted data source information is obtained by the data source storage management system by encrypting data source information sent by the data source reservation system and the data source ID in advance.

[0026] acquisition module, configured to acquire data according to the decrypted data source information.

[0027] According to another aspect of the embodiment of the present application, a computer device is provided, comprising a processor, a memory, a communication interface and a communication bus, the processor, the memory and the communication interface complete communication with each other through the communication bus.

[0028] The memory is used to store at least one executable instruction, and the executable instruction causes the processor to perform the operations of the data acquisition method.

[0029] According to still another aspect of the embodiment of the present application, a computer readable storage medium is provided, the storage medium stores at least one executable instruction, and the executable instruction causes the computer device to perform the operations of the data acquisition method when the computer device runs.

[0030] The embodiment of the present application can receive a data source acquisition event notification sent by a data source reservation system, the data source acquisition event notification carries a data source ID, acquire encrypted data source information from a data source storage management system according to the data source ID and decrypt the encrypted data source information to obtain decrypted data source information, wherein the encrypted data source information is obtained by the data source storage management system by encrypting data source information sent by the data source reservation system and the data source ID in advance, and acquire data according to the decrypted data source information, which can effectively protect data source information from being leaked and protect data security in the data acquisition process.

[0031] The above description is only a summary of the technical scheme of the embodiment of the present application, in order to more clearly understand the technical means of the embodiment of the present application, the embodiment can be implemented according to the content of the specification, and in order to make the above and other purposes, characteristics and advantages of the embodiment of the present application more obvious and easy to understand, the specific implementation method of the present application is described below. BRIEF DESCRIPTION OF DRAWINGS

[0032] The accompanying drawings are only used to show the embodiments and are not considered as limitations of the present application. Moreover, the same reference signs are used to represent the same components throughout the drawings. In the drawings:

[0033] Figure 1A flowchart of the data management system provided by the embodiment of the present application is shown;

[0034] Figure 2 A work flow chart of the data source reservation system in the data management system provided by the embodiment of the present application is shown;

[0035] Figure 3 An API interface diagram of the agent client in the data management system provided by the embodiment of the present application is shown;

[0036] Figure 4 A work flow chart of the data collection system in the data management system provided by the embodiment of the present application is shown;

[0037] Figure 5 A flowchart of the data collection method provided by the embodiment of the present application is shown;

[0038] Figure 6 A flowchart of the data collection method provided by another embodiment of the present application is shown;

[0039] Figure 7 A structure diagram of the data collection device provided by the embodiment of the present application is shown;

[0040] Figure 8 A structure diagram of the computer device provided by the embodiment of the present application is shown. DETAILED DESCRIPTION

[0041] Exemplary embodiments of the present application will be described herein below with reference to the accompanying drawings. While exemplary embodiments of the present application are shown in the drawings, it is understood that the present application can be implemented in various forms and should not be limited by the embodiments set forth herein.

[0042] First, the prior art is further described in detail:

[0043] In the communication field, the management of wireless network / transmission network / bearer network / core network and the like is generally based on the network element OMC of equipment manufacturers to realize single-manufacturer domain management, and based on fault management / performance management / resource management / configuration management and the like network management system to realize end-to-end management across manufacturers. The network management system needs to collect resource, performance and alarm data from the OMC of each manufacturer through various interface forms such as FTP / Socket / Corba / database, and to realize alarm monitoring, performance analysis / optimization, resource life cycle management, configuration verification and the like network management functions based on data processing.

[0044] Generally considering the security of a communication network, network elements OMC and a wireless network / transmission network / bearer network / core network and the like communication network are divided into professional unified management and maintenance, and the OMC account password is replaced regularly according to the information security management requirements. When the first type of network management system is connected with the OMC, or the network management system fails to find and collect tasks after the OMC account password is replaced, the network management system construction personnel / operation personnel generally contacts the professional administrator offline through the ways of email, WeChat and telephone to obtain the account password and import the account password into the network management system to configure the collection task, and the account password is saved in the office computer and network management server in plaintext.

[0045] In the existing scheme of the network management system and the device manufacturer OMC for collecting data, the information security problem of the OMC data source information (name, IP, account password and the like) is generally ignored, and there is a large risk of sensitive information leakage in the whole process:

[0046] 1. The network management developers, operation personnel, maintenance personnel, outsourcing manufacturers and the like know and save the data source information in plaintext;

[0047] 2. The maintenance personnel of different professions may know the OMC data source information of other professions beyond the permission range of the profession;

[0048] 3. The data source information is sent in plaintext in the mail, WeChat and the like communication channels, and is saved in plaintext in the local computer and the network management server, and if intercepted, a large amount of data source information will be leaked.

[0049] The inventors of the present application find that in the process of data collection in the communication field or data collection in other industries, the current general concern is the information security of the collected data content in the collection, transmission, storage and consumption process and the security authentication of the collector and the collected party, and the security management of the data source information lacks attention.

[0050] Therefore, in view of the problem of the risk of leakage of the data source information in the data collection process, the embodiment of the present application proposes a data source information encryption collection system based on the idea of information security responsibility separation.

[0051] Next, the data management system of the embodiment of the present application is introduced. As shown in Figure 1 The data management system of the embodiment of the present application includes a data source reporting system, a data collection system and a data source storage management system.

[0052] The data source reservation system is configured to obtain data source information input by a user and generate a corresponding data source ID and event notification, and to send the data source information and the data source ID to the data source storage management system after encryption by a public key.

[0053] Specifically, the data source reservation system is only open to data source owners (users, such as OMC administrators of core networks, wireless networks, transmission networks, and bearer networks). The data source information input by the data source owner (user) in the data source reservation system, such as actively reported data source information newly added or updated by the data source owner, is obtained by the data source reservation system, encrypted by a public key, and then sent to the data source storage management system for storage. Specifically, as shown in Figure 2 The process of data source information reservation performed by the data source reservation system includes: S101: A data source owner (OMC administrator) configures an account, a password, and a permission for a data collection system on an OMC data source to be collected. S102: The data source owner logs in to the data source reservation system, selects data source information input in a data source addition or data source change process, and initiates reservation. The data source information includes information such as the province, specialty, vendor, name, IP, account, password, and path of the data source. The data source owner can actively report the province, specialty, vendor, name, IP, account, password, and path of the data source. S103: The data source reservation system obtains the data source information input by the user, verifies the user's permission, and checks the accuracy and compliance of the data source related information according to a preset rule. If there is any error information, non-compliant information, or conflict information, the user is prompted to modify. S104: If the check is passed, the reservation system generates a unique ID number for each data source and encrypts sensitive information in the data source information using a public key to obtain encrypted data source information. The public key is a public key previously distributed by the data source storage management system. S105: The data source reservation system submits the encrypted data source information to the data source storage management system for unified storage. S106: The data source reservation system sends the data source ID and event notification to the data collection system. The event notification includes an addition / change event message notification and does not contain data source sensitive information. The data collection system checks the data source task configuration based on the event notification and automatically adds / modifies the collection task information. S107: The data source reservation system completes data source reservation, storage, notification, and prompts the user that the process is complete.

[0054] The data source storage management system is configured to receive and store the encrypted data source information and the data source ID sent by the data source reservation system.

[0055] Specifically, the data source storage management system implements encrypted storage and management of data source sensitive information. According to the idea of information security responsibility separation, the data source storage management system is independent of the data source reservation system and the data collection system, and ensures that the data source sensitive information is not leaked. The data source storage management system includes a key management module, a user permission management module, a data source information access / update API, a data source information encrypted storage module, a log auditing module and a proxy client client. The key management module is used to generate a key pair, and send the public key in the key pair to the data source reservation system and the private key in the key pair to the data collection system. Specifically, the key pair can be generated according to the province and the data source type, and the public key can be obtained by the data source reservation system and the private key can be obtained by the proxy client in the form of API interface. The user permission management module is used to control the data source permission of the user account, and control the correspondence between the user account and the data source, to ensure that the user in the data source reservation system operates the data source within the permission. The data source information encrypted storage module is used to store the encrypted data source information and data source ID sent by the data source reservation system, so that the collection system saves the encrypted data source information based on an independent database, ensures that the data source information is separated from other data, and only opens access permission to the data source information access / update API module. The data source information access / update API module is used to interact with the data source reservation system and the data collection system, on the one hand to support the data source addition / change operation of the data source reservation system, and on the other hand to support the data interaction with the data collection in cooperation with the proxy client client. The proxy client client is used to establish data transmission between the data source storage management system and the data collection system. The data source storage management system provides proxy clients for various data source types such as FTP / sftp / jdbc in the form of sdk. The collection system accesses the data source and collects data by calling the interface in the proxy client to obtain the data source connection. As shown in Figure 3 The proxy client client exposes a unified interface based on the facade pattern design, and internally implements API interfaces of different protocol types according to different data source types.

[0056] The data collection system is configured to acquire the encrypted data source information from the data source storage management system according to the data source collection event notification and decrypt the encrypted data source information to obtain decrypted data source information, and is further configured to collect data according to the decrypted data source information. The data source reservation system is further configured to initiate a data source acquisition request to the data source storage management system according to the data source ID, wherein the data source acquisition request includes the data source ID. The data source storage management system acquires the encrypted data source information and the private key encrypted by the secret key from the data source storage management system according to the data source acquisition request through the proxy client. The proxy client decrypts the private key by using the secret key and decrypts the encrypted data source information according to the private key to obtain the decrypted data source information. The encrypted data source information is data source information encrypted by a public key.

[0057] Specifically, in the embodiment of the present application, to ensure that the data collection program and the data collection operation and maintenance personnel do not touch the data source sensitive information in the collection process, the data collection system does not directly call the original FTP / jdbc interface client to connect the data source, but obtains the data source connection through the proxy client provided by the data storage management system. Therefore, before obtaining the data source information, the data collection system needs to complete the following two steps in advance: 1. Pre-download the proxy software toolkit client sdk. If the collection system wants to connect the data source and collect data, it needs to download the proxy software toolkit client sdk from the data source storage management system and integrate it into the collection application code. 2. Pre-acquire access credentials from the data source storage management system. The access credentials include an identification key (IdentificationKey) and a secret key (SecretKey). The identification key is used to determine the identity of the visitor and verify whether the visitor has the right to access the related data source. The secret key is used for signing the request to prevent the request from being tampered with during transmission, and is also used as the encryption private key. Because the data source information stored in the data source management system is encrypted using a public key, the proxy client client needs to use the pre-acquired private key to decrypt the encrypted data source information after obtaining the encrypted data source information, so the transmission of the private key must be ensured to be secure. The access credentials need to be obtained by logging in to the data source storage management system interface, and the storage management system will return the corresponding access credentials according to the related permissions of the login account.

[0058] As Figure 4As shown, after the above two-step operation is completed, the data collection system interacts with the data source reservation system and the data source storage management system to obtain the data source connection, and the specific process of executing the collection task is as follows: (1) The data collection system calls the related interface of the proxy client client to obtain the data source connection. When calling the interface, the data source acquisition request needs to be passed in. (2) The proxy client client interacts with the data source storage management system according to the parameter information in the data source acquisition request to obtain the data source information and the data source encryption private key, and creates a data source connection after decrypting the data source information locally. The interaction between the proxy Client and the data source storage management system is based on the HTTP protocol, and uses the request data signature and message encryption data transmission mechanism to realize the security of data transmission. The specific steps are as follows: a. Calculate the data signature of the data source acquisition request. The proxy client client uses the hash algorithm to calculate the hash value of the data (including the request header + request body) in the data source acquisition request. The calculated hash value is encrypted using the key SecretKey to obtain the data signature of the data source acquisition request. b. Request data source information and data source encryption private key. The calculated data signature, IdentificationKey are sent to the data source storage management system together with the data source acquisition request. c. The data source storage management system performs signature and visitor identity authentication. After receiving the request, the data source storage management system calculates the signature of the request data in the same way, and compares it with the data signature in the data source acquisition request to confirm whether the request data has been tampered with. The identity and authority of the sender are verified through IdentificationKey to identify whether the requester has the authority to access the data source information of the specified data source ID. If the identity verification is passed, the encrypted data source information and the private key encrypted using the key secretKey are returned. d. After receiving the reply information, the proxy client uses the key secretKey to decrypt the private key, and then uses the private key to decrypt the encrypted data source information to obtain the plaintext data source information. e. According to the data source type specified in the data source information, the corresponding API method is called to create a data source connection. (3) According to the decrypted data source information, the data collection data is collected according to the returned data source connection, and the data collection system executes the data collection according to the task configuration.

[0059] In the embodiment of the present application, the data storage management system is responsible for management and maintenance by an independent data security administrator (the related functions can be provided by a data security management system), the data storage management system provides a public key based on an asymmetric encryption RSA algorithm to support the encryption of a data source reporting system and the encrypted storage of sensitive information of the data source, and provides an FTP / sftp / jdbc proxy client for data source access to support the data collection system to connect the data source without contacting the plaintext information of the data source. After receiving the data source event message notification, the data collection system adds or updates the collection task, calls the proxy client provided by the data source storage management system to access the data source to establish a connection to collect data, and if a problem occurs in the collection, the reporting module can be notified of the problem details in reverse, and the data source owner can be notified by the reporting module to check and handle. The network management system data collection operation and maintenance personnel can also check the collection task running condition and verify the data source connectivity based on the scheme without contacting the plaintext information of the data source account and password. In the whole process, only the data source owner contacts the plaintext data source information, and the data collection system and the data collection operation and maintenance personnel are shielded, thereby effectively eliminating the risk of leakage of sensitive information of the data source.

[0060] Figure 5 A flowchart of a data collection method provided by the embodiment of the present application is shown, and the method is executed by the data collection system in the data management system of the above-mentioned embodiment. As shown in 5, the method comprises the following steps:

[0061] Step 110: receiving a data source collection event notification sent by a data source reporting system, wherein the data source collection event notification carries a data source ID.

[0062] The data source reporting system generates a corresponding data source ID and event notification after obtaining the data source information input by the user, and sends the data source information and the data source ID after being encrypted by a public key to the data source storage management system. The public key is previously sent by the data source storage management system to the data source reporting system, and the public key can be configured according to the data source type and location. After the data source reporting system receives the encrypted data source information and the data source ID, the encrypted data source information and the data source ID are stored. The data source reporting system also sends the event notification and the data source ID to the data collection system after the data source storage management system stores the encrypted data source information and the data source ID.

[0063] Step 120: obtaining the encrypted data source information from the data source storage management system according to the data source ID and decrypting the encrypted data source information to obtain decrypted data source information; the encrypted data source information is obtained by the data source storage management system by encrypting the data source information and the data source ID sent by the data source reporting system in advance.

[0064] In the embodiment of the present application, the proxy client software tool development kit is also pre-acquired from the data source storage management system and integrated in the code; and the data connection with the data source storage management system is established through the proxy client and the access credential, which includes an identity key and a key key. The identity key is used to verify the identity of the data acquisition system, the key key is used for signing the request data and as an encryption private key, which can prevent the request from being tampered during transmission and can effectively protect the private key from being leaked during transmission.

[0065] The specific process of obtaining the encrypted data source information from the data source storage management system according to the data source ID and decrypting it to obtain the decrypted data source information includes: initiating a data source acquisition request to the data source storage management system according to the data source ID; the data source acquisition request includes the data source ID; obtaining the encrypted data source information and the private key encrypted by the key key from the data source storage management system through the proxy client according to the data source acquisition request; obtaining the private key by decrypting the key key through the proxy client, and decrypting the encrypted data source information according to the private key to obtain the decrypted data source information; the encrypted data source information is the data source information encrypted by the public key. Specifically, the process of obtaining the encrypted data source information and the private key encrypted by the key key from the data source storage management system through the proxy client according to the data source acquisition request is: the proxy client calculates the data signature of the data source acquisition request according to the key key; the proxy client sends the data source ID, the data signature and the access credential to the data source storage management system, so that the data source storage management system verifies the access credential and the data signature, and returns the encrypted data source information and the private key encrypted by the key key.

[0066] Step 130: According to the decrypted data source information, data acquisition is performed.

[0067] After obtaining the decrypted data source information, corresponding data acquisition can be performed according to the acquisition configuration task to obtain the acquired data.

[0068] The embodiment of the present application receives a data source collection event announcement sent by a data source reservation system, the data source collection event announcement carrying a data source ID; acquires encrypted data source information from a data source storage management system according to the data source ID and decrypts the encrypted data source information to obtain decrypted data source information; the encrypted data source information is obtained by pre-encryption of the data source information and the data source ID sent by the data source reservation system according to the data source storage management system; and data collection is performed according to the decrypted data source information, which can effectively protect the data source information from being leaked and protect the data security during data collection.

[0069] Figure 6 A flow chart of a data collection method provided by another embodiment of the present application is shown, which is executed by a data source storage management system in the data management system of the above-mentioned embodiment. As shown in Figure 6 The method comprises the following steps:

[0070] Step 210: receiving encrypted data source information and a corresponding data source ID sent by a data source reservation system. The encrypted data source information is obtained by encryption of the data source storage management system according to a public key sent by the data source storage management system to the data source reservation system in advance.

[0071] Step 220: storing the encrypted data source information and the corresponding data source ID.

[0072] Step 230: receiving a data source acquisition request sent by a data collection system and verifying the data source acquisition request; the data source acquisition request comprises the data source ID.

[0073] The verification process of the data source acquisition request sent by the data collection system is as follows:

[0074] The data source acquisition request is obtained through a proxy client, the data source acquisition request carrying a data signature encrypted by a secret key and an access credential. The data signature encrypted by the secret key is verified, and whether the data collection system has the acquisition right of the data source information is verified according to the access credential.

[0075] Step 240: when the verification is passed, the encrypted data source information and the private key encrypted by the secret key are sent to the data collection system, so that the data collection system decrypts the encrypted data source information to obtain decrypted data source information, and performs data collection according to the decrypted data source information.

[0076] When the verification is passed, the encrypted data source information and the private key encrypted by the key are sent to the data collection system, so that the data collection system decrypts the private key encrypted by the key, obtains the private key, decrypts the encrypted data source information by using the private key, obtains the decrypted data source information, and collects data according to the decrypted data source information.

[0077] The data source collection event notification sent by the data source reservation system is received, and the data source collection event notification carries a data source ID. The encrypted data source information is obtained from the data source storage management system according to the data source ID and is decrypted to obtain decrypted data source information. The encrypted data source information is obtained by the data source storage management system according to the data source information and the data source ID sent by the data source reservation system. According to the decrypted data source information, data collection is performed, which can effectively protect the data source information from being leaked and protect the data security during the data collection process.

[0078] Figure 7 The structure of the data collection device provided by the embodiment of the application is shown. Figure 7 As shown in the figure, the device 300 includes a receiving module 310, a decryption module 320 and a collection module 330.

[0079] The receiving module 310 is configured to receive the data source collection event notification sent by the data source reservation system, and the data source collection event notification carries a data source ID.

[0080] The decryption module 320 is configured to obtain the encrypted data source information from the data source storage management system according to the data source ID and decrypt the encrypted data source information to obtain decrypted data source information. The encrypted data source information is obtained by the data source storage management system according to the data source information and the data source ID sent by the data source reservation system.

[0081] The collection module 330 is configured to collect data according to the decrypted data source information.

[0082] The specific working process of the data collection device of the embodiment of the application is basically the same as the specific working mode of the data collection system in the data management system, and will not be repeated here.

[0083] The embodiment of the present application receives a data source collection event announcement sent by a data source reservation system, the data source collection event announcement carries a data source ID; acquires encrypted data source information according to the data source ID from a data source storage management system and decrypts the encrypted data source information to obtain decrypted data source information; the encrypted data source information is obtained by pre-encryption of the data source storage management system according to data source information and a data source ID sent by the data source reservation system; and data collection is performed according to the decrypted data source information, which can effectively protect data source information from being leaked and guarantee data security in the data collection process.

[0084] Figure 8 The structure schematic diagram of the computer device provided by the embodiment of the present application is shown, and the specific implementation of the computer device is not limited in the embodiment of the present application.

[0085] As shown in Figure 8 , the computer device can include a processor 402, a communications interface 404, a memory 406, and a communications bus 408.

[0086] The processor 402, the communications interface 404, and the memory 406 can communicate with each other through the communications bus 408. The communications interface 404 is configured to communicate with network elements such as clients or other servers. The processor 402 is configured to execute the program 410, and specifically can execute the related steps in the above-mentioned data collection method embodiment.

[0087] Specifically, the program 410 can include program code, and the program code includes computer executable instructions.

[0088] The processor 402 can be a central processing unit CPU, or an application specific integrated circuit ASIC, or one or more integrated circuits configured to implement the embodiments of the present application. The one or more processors included in the computer device can be the same type of processor, such as one or more CPUs; or can be different types of processors, such as one or more CPUs and one or more ASICs.

[0089] The memory 406 is configured to store the program 410. The memory 406 can include a high-speed RAM memory, and can also include a non-volatile memory, for example, at least one disk memory.

[0090] The program 410 can be specifically called by the processor 402 to make the computer device perform the following operations:

[0091] The executable instructions can be used to cause the computer device to perform the following operations:

[0092] receiving a data source collection event notification sent by the data source reservation system, the data source collection event notification carrying a data source ID;

[0093] obtaining encrypted data source information from a data source storage management system according to the data source ID and decrypting the encrypted data source information to obtain decrypted data source information; the encrypted data source information is obtained by the data source storage management system by encrypting data source information and a data source ID sent by the data source reservation system in advance;

[0094] performing data collection according to the decrypted data source information;

[0095] Alternatively,

[0096] receiving encrypted data source information and a corresponding data source ID sent by the data source reservation system;

[0097] storing the encrypted data source information and the corresponding data source ID;

[0098] receiving a data source acquisition request sent by a data collection system and verifying the data source acquisition request; the data source acquisition request includes the data source ID;

[0099] when the verification is passed, sending the encrypted data source information and a private key encrypted by a secret key to the data collection system, so that the data collection system decrypts the encrypted data source information to obtain decrypted data source information, and performs data collection according to the decrypted data source information.

[0100] In an optional manner, before the step of obtaining encrypted data source information from a data source storage management system according to the data source ID and decrypting the encrypted data source information to obtain decrypted data source information, the method further includes: obtaining an agent client software development kit from the data source storage management system in advance and integrating the agent client software development kit in code; establishing a data connection with the data source storage management system through the agent client and an access credential, the access credential including an identity key and a secret key.

[0101] In an alternative manner, the obtaining of the encrypted data source information from the data source storage management system according to the data source ID and the decryption include: initiating a data source obtaining request to the data source storage management system according to the data source ID; the data source obtaining request includes the data source ID; the data source storage management system obtains the encrypted data source information and the private key encrypted by the key key from the data source storage management system according to the data source obtaining request through the proxy client; the private key is obtained by the proxy client using the key key for decryption, and the encrypted data source information is decrypted according to the private key to obtain the decrypted data source information; the encrypted data source information is the data source information encrypted by the public key.

[0102] In an alternative manner, the data source storage management system obtains the encrypted data source information and the private key encrypted by the key key from the data source storage management system according to the data source obtaining request through the proxy client, including: the proxy client calculates the data signature of the data source obtaining request according to the key key; the data source ID, the data signature and the access credential are sent to the data source storage management system by the proxy client, so that the data source storage management system verifies the access credential and the data signature, and returns the encrypted data source information and the private key encrypted by the key key.

[0103] The embodiment of the application receives a data source collection event notification sent by a data source reservation system, the data source collection event notification carries a data source ID; obtains encrypted data source information from a data source storage management system according to the data source ID and decrypts to obtain decrypted data source information; the encrypted data source information is obtained by the data source storage management system in advance according to the data source information and the data source ID sent by the data source reservation system; and data collection is performed according to the decrypted data source information, which can effectively protect the data source information from being leaked and protect the data security in the data collection process.

[0104] The embodiment of the application provides a computer readable storage medium, the storage medium stores at least one executable instruction, the executable instruction is run on a computer device, and the computer device executes the data collection method in any method embodiment.

[0105] The executable instruction can be specifically used for causing the computer device to perform the following operations:

[0106] Receiving a data source collection event notification sent by a data source reservation system, the data source collection event notification carrying a data source ID;

[0107] obtaining the encrypted data source information from the data source storage management system according to the data source ID and decrypting the encrypted data source information to obtain decrypted data source information; the encrypted data source information is obtained by the data source storage management system by encrypting data source information sent by the data source reservation system and the data source ID in advance;

[0108] performing data collection according to the decrypted data source information;

[0109] Alternatively,

[0110] receiving encrypted data source information and a corresponding data source ID sent by a data source reservation system;

[0111] storing the encrypted data source information and the corresponding data source ID;

[0112] receiving a data source acquisition request sent by a data collection system and verifying the data source acquisition request; the data source acquisition request includes the data source ID;

[0113] when the verification is passed, sending the encrypted data source information and a private key encrypted by a key to the data collection system, so that the data collection system decrypts the encrypted data source information to obtain decrypted data source information, and performs data collection according to the decrypted data source information.

[0114] In an optional manner, before the method of obtaining the encrypted data source information from the data source storage management system according to the data source ID and decrypting the encrypted data source information to obtain decrypted data source information, the method further comprises: obtaining a proxy client software tool development package from the data source storage management system in advance and integrating it in the code; establishing a data connection with the data source storage management system through the proxy client and an access credential, the access credential including an identity key and a key key.

[0115] In an optional manner, the method of obtaining the encrypted data source information from the data source storage management system according to the data source ID and decrypting the encrypted data source information to obtain decrypted data source information comprises: initiating a data source acquisition request to the data source storage management system according to the data source ID; the data source acquisition request includes the data source ID; the data source storage management system obtains the encrypted data source information and the private key encrypted by the key key from the data source storage management system according to the data source acquisition request through the proxy client; the private key is obtained by decrypting the key key through the proxy client, and the encrypted data source information is decrypted according to the private key to obtain decrypted data source information; the encrypted data source information is data source information encrypted by a public key.

[0116] In an alternative mode, the data source storage management system obtains the encrypted data source information and the private key encrypted by the key key from the data source storage management system according to the data source acquisition request through the proxy client, including: the proxy client calculates the data signature of the data source acquisition request according to the key key; the proxy client sends the data source ID, the data signature and the access credential to the data source storage management system, so that the data source storage management system verifies the access credential and the data signature, and returns the encrypted data source information and the private key encrypted by the key key.

[0117] The embodiment of the application can receive a data source collection event notification sent by a data source reservation system, the data source collection event notification carrying a data source ID; obtain encrypted data source information from a data source storage management system according to the data source ID and decrypt the encrypted data source information to obtain decrypted data source information; the encrypted data source information is obtained by the data source storage management system by pre-encrypting data source information and a data source ID sent by the data source reservation system; and data collection is performed according to the decrypted data source information, which can effectively protect data source information from being leaked and protect data security during data collection.

[0118] The embodiment of the application provides a data collection device for executing the above data collection method.

[0119] The embodiment of the application provides a computer program, which can be called by a processor to enable a computer device to execute the data collection method in any method embodiment.

[0120] The embodiment of the application provides a computer program product, which comprises a computer program stored on a computer readable storage medium, and the computer program comprises program instructions, which, when executed on a computer, enable the computer to execute the data collection method in any method embodiment.

[0121] The algorithms and displays provided herein are not inherently related to any particular computer, virtual system, or other apparatus. Various general purpose systems can be used with these teachings, or with variations thereof, depending on the implementation. The structure required to be present on such systems to implement the present application is apparent from the above description. In addition, the present application is not described with reference to any particular programming language. It will be appreciated that a variety of programming languages can be used to implement the teachings of the application as described herein, and any references below to specific languages are provided for disclosure of enablement only.

[0122] In the description provided herein, numerous specific details are set forth. However, it is understood that embodiments of the application can be practiced without these specific details. In some instances, well-known methods, structures and techniques have not been described in detail in order not to obscure the understanding of this description.

[0123] Similarly, it is to be understood that the embodiments of the application can be adapted to other applications and that their features can be interchanged in any way. It is to be understood that the phraseology and terminology employed herein are for the purpose of description and not of limitation. Any use of being followed by a list of one or more items is to be understood as meaning one, some, or all of the items on the list are used. Only claims may limit the scope of the application. Any reference to claim 1 is to be interpreted as a reference to the subject-matter of the independent claim for which priority is claimed, and not to the dependent claims.

[0124] Those skilled in the art will appreciate that modules in the apparatuses in the embodiments can be adapted and arranged in one or more apparatuses other than the embodiments. Modules or units or components in the embodiments can be combined into one module or unit or component, and can be divided into a plurality of sub-modules or sub-units or sub-components. Any combination of all the features disclosed in the specification (including the accompanying claims, abstract and drawings), and any method or apparatus so disclosed, can be made unless the contrary is explicitly stated in the specification (including the accompanying claims, abstract and drawings), or is otherwise obvious from the specification (including the accompanying claims, abstract and drawings). Except where expressly indicated otherwise, each feature disclosed in the specification (including the accompanying claims, abstract and drawings) can be replaced by alternative features serving the same, equivalent or a similar purpose.

[0125] It is to be noted that the above-mentioned embodiments illustrate rather than limit the application, and that one skilled in the art will be able to design many alternative embodiments without departing from the scope of the appended claims. In the claims, any reference signs placed between parentheses shall not be construed as limiting the claim. The word "comprising" does not exclude the presence of elements or steps other than those listed in a claim. The word "a" or "an" preceding an element does not exclude the presence of a plurality of such elements. The application can be implemented by means of hardware comprising several distinct elements, and by means of a suitably programmed computer. In the case of a suitably programmed computer, the programming and hardware can comprise elements that are well known in the art. In the claims, the word "first", "second", "third", etc. does not imply any order. The terms "first", "second", "third", etc. are to be interpreted according to the context in which they are used. Steps in the above-described embodiments, other than those that are specifically stated to be limiting, are not to be understood as necessarily requiring the order in which they are presented.

Claims

1. A data acquisition method, characterized by, The method comprises: Receiving a data source collection event notification sent by a data source reservation system, the data source collection event notification carrying a data source ID; Pre-acquiring an agent client software tool development kit from a data source storage management system and integrating it in code; establishing a data connection with the data source storage management system through the agent client and access credentials, the access credentials including an identity key and a secret key; According to the data source ID, obtaining encrypted data source information from the data source storage management system and decrypting it to obtain decrypted data source information; the encrypted data source information is obtained by the data source storage management system by pre-encrypting the data source information sent by the data source reservation system according to the data source ID; According to the data source ID, obtaining encrypted data source information from the data source storage management system and decrypting it to obtain decrypted data source information; the encrypted data source information is obtained by the data source storage management system by pre-encrypting the data source information sent by the data source reservation system according to the data source ID; According to the data source ID, obtaining encrypted data source information from the data source storage management system and decrypting it to obtain decrypted data source information; the encrypted data source information is obtained by the data source storage management system by pre-encrypting the data source information sent by the data source reservation system according to the data source ID; According to the data source ID, obtaining encrypted data source information from the data source storage management system and decrypting it to obtain decrypted data source information; the encrypted data source information is obtained by the data source storage management system by pre-encrypting the data source information sent by the data source reservation system according to the data source ID; According to the decrypted data source information, data collection is performed. According to the data source ID, obtaining encrypted data source information from the data source storage management system and decrypting it to obtain decrypted data source information; the encrypted data source information is obtained by the data source storage management system by pre-encrypting the data source information sent by the data source reservation system according to the data source ID; The method comprises:

2. The method of claim 1, wherein, Receiving encrypted data source information and corresponding data source IDs sent by a data source reservation system; Storing the encrypted data source information and corresponding data source IDs; ​ 3. A data acquisition method characterized by, ​ ​ ​ The data source acquisition request sent by the data collection system is received and verified, and the data source acquisition request includes the data source ID; wherein the data collection system obtains a proxy client software tool development package from the data source storage management system, integrates the proxy client software tool development package in the code, and establishes a data connection with the data source storage management system through the proxy client and access credentials, and the access credentials include an identity key and a key key; The process of verifying the data source acquisition request sent by the data collection system is that the data source acquisition request is obtained through the proxy client, the data source acquisition request carries a data signature encrypted by the key key and access credentials; the data signature encrypted by the key key is verified, and whether the data collection system has the access right of the data source information is verified according to the access credentials; When the verification is passed, the encrypted data source information and the private key encrypted by the key key are sent to the data collection system, so that the data collection system obtains the private key by decrypting the private key using the key key through the proxy client, decrypts the encrypted data source information according to the private key, obtains the decrypted data source information, and collects data according to the decrypted data source information, and the encrypted data source information is data source information encrypted by a public key.

4. A data management system, characterized by The data management system includes a data source reservation system, a data collection system and a data source storage management system; The data source reservation system is configured to obtain data source information input by a user and generate a corresponding data source ID and event notification, and is further configured to encrypt the data source information and the data source ID using a public key and send the encrypted data source information and the data source ID to the data source storage management system; The data source storage management system is configured to receive the encrypted data source information and the data source ID sent by the data source reservation system and store them; The data source reservation system is further configured to send a data source collection event notification to the data collection system, and the data source collection event notification carries a data source ID; The data collection system is configured to obtain encrypted data source information from the data source storage management system according to the data source collection event notification and decrypt the encrypted data source information to obtain decrypted data source information; The data collection system is further configured to collect data according to the decrypted data source information, and is further configured to obtain a proxy client software tool development package from the data source storage management system in advance and integrate the proxy client software tool development package in the code; the data collection system establishes a data connection with the data source storage management system through the proxy client and access credentials, and the access credentials include an identity key and a key key; The data collection system is further configured to initiate a data source acquisition request to the data source storage management system according to the data source ID, and the data source acquisition request includes the data source ID; The data collection system is further configured to obtain encrypted data source information and a private key encrypted by a key key from the data source storage management system according to the data source acquisition request through the proxy client; obtain the private key by decrypting the key key through the proxy client, and decrypt the encrypted data source information according to the private key to obtain decrypted data source information; the encrypted data source information is data source information encrypted by a public key; The data source storage management system obtains a data source acquisition request, the data source acquisition request carrying data signature encrypted by a key key and access credentials; the data signature encrypted by the key key is verified, and whether the data acquisition system has the acquisition permission of the data source information is verified according to the access credentials; when the verification is passed, the encrypted data source information and the private key encrypted by the key key are sent.

5. A data acquisition device, characterized by Comprise: The receiving module is used for receiving the data source collection event notification sent by the data source reservation system, and the data source collection event notification carries a data source ID; The decryption module is used for obtaining encrypted data source information from the data source storage management system according to the data source ID and decrypting to obtain decrypted data source information; the encrypted data source information is obtained by the data source storage management system by encrypting the data source information and the data source ID sent by the data source reservation system in advance; The collection module is used for collecting data according to the decrypted data source information; The device is also used for: obtaining the proxy client software tool development package from the data source storage management system in advance, and integrating in the code; the data connection with the data source storage management system is established through the proxy client and the access credentials, and the access credentials include an identity key and a key key; The decryption module is also used for: initiating a data source acquisition request to the data source storage management system according to the data source ID; the data source acquisition request includes the data source ID; The encrypted data source information and the private key encrypted by the key key are obtained from the data source storage management system by the proxy client according to the data source acquisition request; The private key is obtained by decrypting the key key through the proxy client, and the encrypted data source information is decrypted according to the private key to obtain decrypted data source information; the encrypted data source information is data source information encrypted by a public key; The data source storage management system obtains a data source acquisition request, the data source acquisition request carrying data signature encrypted by a key key and access credentials; the data signature encrypted by the key key is verified, and whether the data acquisition system has the acquisition permission of the data source information is verified according to the access credentials; when the verification is passed, the encrypted data source information and the private key encrypted by the key key are sent.

6. A computer device, comprising: Comprise: The processor, the memory, the communication interface and the communication bus, the processor, the memory and the communication interface complete the communication among each other through the communication bus; The memory is used for storing at least one executable instruction, and the executable instruction makes the processor execute the operations of the data acquisition method in any one of claims 1-3.

7. A computer-readable storage medium, characterized in that, The storage medium stores at least one executable instruction, and the executable instruction, when running on the computer device, causes the computer device to perform the operations of the data collection method according to any one of claims 1-3.

Images