Data acquisition method and device, electronic equipment and storage medium

Abstract

The invention relates to a data acquisition method and device, electronic equipment and a storage medium, and relates to the technical field of computers. The method comprises the following steps: in response to a received data acquisition task, acquiring configuration information of a target interface corresponding to the data acquisition task; determining an authentication mode of the target interface according to the configuration information; in response to the determination that the authentication mode is a dynamic authentication mode, determining an acquisition mode of a dynamic authentication parameter according to the configuration information; and updating the dynamic authentication parameter through the authentication interface before the dynamic authentication parameter expires in response to the determination that the acquisition mode of the dynamic authentication parameter is acquired through the authentication interface, so as to complete the data acquisition task. By applying the method and the device, the security of the target interface is ensured, and the completion efficiency of the data acquisition task is also improved.

Description

Technical Field

[0001] The present application relates to the field of computer technology, and in particular to a data acquisition method, device, electronic device and storage medium. Background Art

[0002] API data integration is a data integration method based on application program interfaces. Different from traditional database and table data integration, it acquires and integrates data in real time by calling API interfaces provided by various data sources, thereby achieving real-time data transmission and sharing. The advantages of API data integration lie in its flexibility and real-time performance. It can achieve real-time data transmission and dynamic update, and is suitable for scenarios that require rapid response and real-time analysis. In addition, API data integration can also be easily integrated with other systems to achieve cross-platform data sharing and collaboration.

[0003] For some important API interfaces, the security of the API interface is also very important. Usually some data providers will use authentication to limit the access and acquisition of data from the API interface. So, in the process of data synchronization, how to obtain data from the API interface safely and efficiently is also a problem that needs to be solved urgently. Summary of the invention

[0004] Embodiments of the present disclosure provide a data acquisition method, device, electronic device, and storage medium.

[0005] In a first aspect, an embodiment of the present disclosure provides a data acquisition method, comprising: in response to receiving a data acquisition task, acquiring configuration information of a target interface corresponding to the data acquisition task; determining an authentication method for the target interface based on the configuration information; in response to determining that the authentication method is a dynamic authentication method, determining an acquisition method for dynamic authentication parameters based on the configuration information; in response to determining that the acquisition method for dynamic authentication parameters is to obtain through an authentication interface, updating the dynamic authentication parameters through the authentication interface before the dynamic authentication parameters expire to complete the data acquisition task.

[0006] In a second aspect, an embodiment of the present disclosure provides a data acquisition device, comprising: an information acquisition unit, configured to, in response to receiving a data acquisition task, acquire configuration information of a target interface corresponding to the data acquisition task; a first determination unit, configured to determine an authentication method of the target interface according to the configuration information; a second determination unit, configured to, in response to determining that the authentication method is a dynamic authentication method, determine an acquisition method of dynamic authentication parameters according to the configuration information; a parameter updating unit, configured to, in response to determining that the acquisition method of the dynamic authentication parameters is to be obtained through an authentication interface, update the dynamic authentication parameters through the authentication interface before the dynamic authentication parameters expire, so as to complete the data acquisition task.

[0007] In a third aspect, an embodiment of the present disclosure provides an electronic device, comprising a memory, a processor, a bus, and a computer program stored in the memory and executable on the processor, wherein the processor implements the data acquisition method described in the first aspect when executing the computer program.

[0008] In a fourth aspect, an embodiment of the present disclosure provides a non-transitory computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements the data acquisition method as described in the first aspect.

[0009] By applying the technical solution disclosed in the present invention, when receiving a data acquisition task, it is possible to determine whether the target interface requires dynamic authentication based on the configuration information of the target interface. When it is determined that the dynamic authentication parameters need to be acquired multiple times to complete the above data acquisition task, the dynamic authentication parameters can be automatically updated before the dynamic authentication parameters expire to ensure that the data acquisition task is completed without feeling, which not only ensures the security of the target interface, but also improves the completion efficiency of the data acquisition task.

[0010] It should be understood that the content described in this section is not intended to identify the key or important features of the embodiments of the present disclosure, nor is it intended to limit the scope of the present disclosure. Other features of the present disclosure will become easily understood through the following description. BRIEF DESCRIPTION OF THE DRAWINGS

[0011] The accompanying drawings are used to better understand the present solution and do not constitute a limitation of the present disclosure.

[0012] Figure 1 is an exemplary system architecture diagram to which an embodiment of the data acquisition method disclosed herein can be applied;

[0013] Figure 2 A flowchart of an embodiment of the data acquisition method disclosed in the present invention;

[0014] Figure 3 A flowchart of another embodiment of the data acquisition method disclosed in the present invention;

[0015] Figure 4 A schematic diagram of a configuration information input page of a target interface in the data acquisition method disclosed in the present invention;

[0016] Figure 5 A schematic diagram of a configuration information input page of an authentication interface in the data acquisition method disclosed in the present invention;

[0017] Figure 6 A schematic diagram of a cache mechanism of the data acquisition method disclosed in the present invention;

[0018] Figure 7A schematic diagram of the structure of an embodiment of a data acquisition device disclosed in the present invention;

[0019] Figure 8 A schematic diagram of the structure of an embodiment of an electronic device disclosed herein. DETAILED DESCRIPTION

[0020] It should be noted that the following detailed descriptions are exemplary and are intended to provide further explanation of the present disclosure. Unless otherwise specified, all technical and scientific terms used herein have the same meanings as those commonly understood by those skilled in the art to which the present disclosure belongs.

[0021] It should be noted that the terms used herein are only for describing specific embodiments and are not intended to limit the exemplary embodiments according to the present disclosure. As used herein, unless the context clearly indicates otherwise, the singular form is also intended to include the plural form. In addition, it should be understood that when the terms "comprising" and / or "including" are used in this specification, it indicates the presence of features, steps, operations, devices, components and / or combinations thereof.

[0022] In the absence of conflict, the embodiments in the present disclosure and the features in the embodiments may be combined with each other.

[0023] In order to make the technical solutions and advantages of the present disclosure more clearly understood, the present disclosure is further described in detail below in conjunction with the accompanying drawings and specific embodiments.

[0024] Figure 1 An exemplary system architecture 100 is shown to which embodiments of the data acquisition method or data acquisition apparatus of the present disclosure can be applied.

[0025] like Figure 1 As shown, the system architecture 100 may include a terminal device 101, a server 102, a database 103, and a network 104. The network 104 is used to provide a medium for a communication link between the terminal device 101, the server 102, and the database 103. The network 104 may include various connection types, such as wired, wireless communication links, or optical fiber cables, etc.

[0026] The user can use the terminal device 101 to interact with the server 103 through the network 104 to receive or send messages, etc. Various communication client applications, such as browser applications, can be installed on the terminal devices 101, 102, and 103. The configuration information setting interface can be accessed through the browser application to configure the target interface.

[0027] The server 102 may be a server providing various services, such as a background server providing configuration information support for the interface. The background server may receive configuration information input by the user through each terminal device 101, and save the configuration information so as to perform authentication according to the configuration information when receiving a request.

[0028] The database 103 may be a database for storing data, and it may store various types of data. The terminal device 101 may access the database 103 through the target interface to read the data in the database 103. When accessing the database through the target interface, authentication is required first. The terminal device 101 may send authentication parameters to the target interface.

[0029] It should be noted that the data acquisition method provided in the embodiment of the present disclosure is generally executed by the server 102. Accordingly, the data acquisition device is generally disposed in the server 102.

[0030] It should be understood that Figure 1 The number of terminal devices, servers, databases and networks in the embodiment is only for illustration. Any number of terminal devices, servers, databases and networks may be provided as required.

[0031] Figure 2 FIG. 2 shows a process 200 of an embodiment of the data acquisition method of the present disclosure. Figure 2 As shown, the data acquisition method of this embodiment may include the following steps:

[0032] Step 201: in response to receiving a data acquisition task, acquiring configuration information of a target interface corresponding to the data acquisition task.

[0033] In this embodiment, the execution subject of the data acquisition method (for example Figure 1 The server 102 shown in the figure can receive data acquisition tasks. Specifically, it can obtain data acquisition tasks from a task distribution device, or it can receive data acquisition tasks sent by a technician through a terminal device. Data acquisition tasks can be distinguished by using identifiers. Here, data acquisition tasks refer to tasks that require access to a database through a target interface to obtain data. When accessing a database through a target interface, authentication is required first.

[0034] After receiving the data acquisition task, the configuration information of the target interface corresponding to the data acquisition task can be obtained. The above configuration information may include information such as the communication address and authentication method of the target interface. The execution subject can obtain the configuration information from a device or platform pre-set for storing the configuration information of the target interface. Alternatively, the configuration information can be obtained from a technician.

[0035] Step 202: Determine the authentication method of the target interface according to the configuration information.

[0036] After obtaining the configuration information, the authentication method of the target interface can be further determined. Specifically, a specific parameter in the configuration information can be analyzed to determine the authentication method. For example, if the parameter value used to identify the authentication method in the configuration information is A, it means that the authentication method is a static authentication method. If the parameter value is B, it means that the authentication method is a dynamic authentication method.

[0037] In this embodiment, the authentication method may include a static authentication method and a dynamic authentication method. The static authentication method refers to using a fixed string or a user name and password for authentication. The dynamic authentication method refers to using a string for authentication within a certain period of time, and using a new string for authentication after the string expires.

[0038] Step 203: In response to determining that the authentication mode is a dynamic authentication mode, a method for obtaining dynamic authentication parameters is determined according to the configuration information.

[0039] If the execution subject determines that the authentication method of the target interface is a dynamic authentication method, the method for obtaining the dynamic authentication parameters can be further determined based on the configuration information. Specifically, the method for obtaining the dynamic authentication parameters can be determined based on a specific parameter in the configuration information. In this embodiment, the acquisition method can include obtaining through the authentication interface and obtaining from a server that provides an authentication parameter generation service. Among them, if it is necessary to obtain the dynamic authentication parameters multiple times to complete the data acquisition task, it can be obtained through the authentication interface. If it is not necessary to obtain the dynamic authentication parameters multiple times to complete the data acquisition task, it can be obtained from a server that provides an authentication parameter generation service.

[0040] Here, the authentication interface can be understood as an interface for obtaining dynamic authentication parameters. The technician can set the authentication interface in advance, and the above settings can include authentication interface protocol and address, request parameter configuration, mapping configuration, etc.

[0041] Step 204, in response to determining that the dynamic authentication parameter is obtained through the authentication interface, the dynamic authentication parameter is updated through the authentication interface before the dynamic authentication parameter expires to complete the data acquisition task.

[0042] In this embodiment, if the dynamic authentication parameter is obtained through an authentication interface, the dynamic authentication parameter can be updated through the authentication interface before the dynamic authentication parameter expires to complete the data acquisition task. Specifically, a timer can be set to start a thread to re-acquire a dynamic authentication parameter through the authentication interface before the dynamic authentication parameter expires. The newly acquired dynamic authentication parameter is used to update the existing dynamic authentication parameter. In this way, the new dynamic authentication parameter can be used to obtain data through the target interface to complete the data acquisition task.

[0043] The data acquisition method provided by the above embodiment of the present disclosure can determine whether the target interface requires dynamic authentication based on the configuration information of the target interface when receiving the data acquisition task. When it is determined that the dynamic authentication parameters need to be acquired multiple times to complete the above data acquisition task, the dynamic authentication parameters can be automatically updated before the dynamic authentication parameters expire to ensure that the data acquisition task is completed without feeling, which not only ensures the security of the target interface, but also improves the completion efficiency of the data acquisition task.

[0044] Continue to see Figure 3 , which shows a process 300 of another embodiment of the data acquisition method according to the present disclosure. Figure 3 As shown, the method in this embodiment may include the following steps:

[0045] Step 301: in response to receiving a data acquisition task, acquiring configuration information of a target interface corresponding to the data acquisition task.

[0046] In this embodiment, the configuration information of the target interface can be obtained from a preset configuration information configuration platform. Here, the preset configuration information configuration platform can be a cloud or server for storing the configuration information of each interface. The configuration information configuration platform here can also provide the user with a configuration information input page (such as Figure 4 As shown in the figure), users can configure the target interface by filling in the information on the above page.

[0047] In some specific practices, the configuration of the target interface may include the following aspects:

[0048] For the API interface, the following configuration is usually required:

[0049] (1) Interface request method: supports GET request and POST request methods.

[0050] (2) Data response format: supports interface response data in JSON and XML formats.

[0051] (3) Interface address: The address of the API interface to be requested. The interface address is usually a URL for http or https requests.

[0052] (4) Request parameters:

[0053] ①K-V type request parameter configuration: supports configuration of Key-Value type request parameters, where you can set the parameter name, parameter type, whether it is a paging parameter, parameter description, and the parameter value corresponding to the parameter name.

[0054] ②Header type request parameter configuration: Header is the header information carried when sending a request. Usually, you can set the authentication information of the request: authentication / cookie, the data format of the response: content-type, etc. Here, it is also supported to set the header information when requesting the interface in the form of Key-Value.

[0055] ③Body type request parameter configuration: When the interface request method is selected as the POST request method, the Body type request parameter can be set, and the Body type request parameter transmission is more flexible. In the present invention, the application / json type parameter type is supported, which is also the most common json format, and its form is as follows: {"param1":"value1","param2":"value2","param3":value3}.

[0056] Step 302: Determine the authentication method of the target interface according to the configuration information.

[0057] In this embodiment, the authentication method of the target interface may include a static authentication method and a dynamic authentication method. Among them, the static authentication method may include API key and Basic Authentication, and the dynamic authentication method may include Token. API key means: the server generates a pair of API Key / API Secret for each client and informs the client. The client needs to carry these keys when requesting the API interface. Only when the resources, API Key and API Secret match can the server's resources be accessed. Basic Authentication is an authentication method based on username and password, usually used for user verification and access control to restricted resources. The "Authorization" field is used in the header of the request to encode the combination of username and password into Base64 encoding format for transmission. Token is a stateless mechanism for user verification. After logging in, the user will generate a Token token, which is used to verify and authorize the identity.

[0058] Step 303, in response to determining that the authentication mode is a static authentication mode, sending static authentication parameters to the target interface to execute the data acquisition task.

[0059] In this embodiment, if the authentication mode of the target interface is a static authentication mode, the static authentication parameters can be directly sent to the target interface. In this way, after the target interface determines that the static authentication parameters are correct, it can allow data reading, so that the data acquisition task can be executed. The static authentication parameters here can be sent to the execution subject in advance by other devices, or obtained by the execution subject interacting with the target interface before executing the data acquisition task.

[0060] In some specific practices, when the authentication method is API key, you need to carry the key when requesting the API data interface, that is, you need to add the two parameters appId and appSecret in the request parameters of the interface (the parameter names may be slightly different depending on the actual interface), and set the corresponding parameter values. When the authentication method is Basic Authentication, you need to add an attribute named Authorization in the request header when requesting the API data interface, and set the corresponding attribute value. The attribute value is generally a Base64 combination encoding of the user name and password. For the same API interface, since the user name and password generally do not expire, you can use a fixed Base64 encoded string for the request.

[0061] Step 304: In response to determining that the authentication mode is a dynamic authentication mode, an acquisition mode is determined according to a parameter in the configuration information for indicating a dynamic authentication parameter acquisition mode.

[0062] If it is determined that the authentication method of the target interface is a dynamic authentication method, the configuration information can be further parsed to determine the method for obtaining the dynamic authentication parameters. Specifically, a parameter can be set in the configuration information to indicate the acquisition method. The specific acquisition method can be determined based on the value of the parameter used to indicate the dynamic authentication parameter acquisition method in the configuration information. For example, when the parameter value is a string A, it indicates that the acquisition method is to obtain the dynamic authentication parameters through the authentication interface. If it is not A, it means that it is not obtained through the authentication interface.

[0063] In some optional implementations of this embodiment, different configuration methods may be used for different authentication parameters.

[0064] (1) API key configuration

[0065] When the authentication method is API key, configure the API key in the request parameter list, add parameters: appId and appSecret, and set their corresponding key values. The key value is generally authorized and provided by the provider of the API interface.

[0066] When requesting the API interface, the key will be carried along as a parameter to request the API data interface. When the appId and appSecret are verified, the API interface will return the required data.

[0067] (2) Basic Authentication Configuration

[0068] When the authentication method is Basic Authentication, configure the Authorization attribute in the Headers request header configuration. The Authorization attribute value string generally needs to be obtained by Base64 encoding the user name and password in advance.

[0069] When requesting an API interface, Authorization will be carried in the form of a request header. When the server receives the request, it will decode the Base64 string and extract the username and password for verification. When the verification is successful, the API interface will return the required data.

[0070] (3) Token configuration

[0071] In data integration, when performing API interface data integration, it is possible to obtain data with a single request, or it is possible to loop through multiple requests to the interface to obtain the corresponding batch data. Tokens usually have an expiration time set, and if this time is exceeded, the request will be intercepted.

[0072] Based on this, this disclosure provides two types of Token configurations:

[0073] ①Fixed string.

[0074] The fixed string configuration method is suitable for single requests or API data integration scenarios where the request does not take a long time, and the API interface that can complete the request within the token validity period.

[0075] When using this method, you need to configure the Token attribute in the Headers request header or the Body request body configuration, and the Token attribute value needs to be authorized and provided in advance by the API interface provider.

[0076] ②Authentication interface.

[0077] When the API interface requires multiple requests, the expiration time of the Token may not be enough to support all the requests. In this case, the authentication interface can be used to configure the task.

[0078] The authentication interface is used to obtain the Token in real time. When the interface is requested using relevant authentication parameters, after the authentication parameters are verified, the authentication interface will generate and return the corresponding Token value, which is within the validity period.

[0079] In some specific practices, the configuration of the authentication interface can also be done through a specified page (such as Figure 5 The implementation may mainly include the following aspects:

[0080] ① Authentication interface protocol and address: supports GET request and POST request methods.

[0081] The authentication interface address is generally a URL for an http or https request.

[0082] ② Request parameter configuration: For GET request method, it supports adding parameter list after request URL. For POST request method, it supports configuring request parameters through request body.

[0083] ③Headers configuration: supports the header information carried when sending requests through Headers.

[0084] ④Authentication mapping configuration: used to specify the mapping relationship between the Token in the result returned by the authentication interface and the parameters carried by the API data interface.

[0085] ⑤Token validity period: Set the expiration time of the Token. Generally, the expiration time of the Token is calculated from the time when the Token is generated by the authentication interface. The unit here is set to hours.

[0086] In some optional implementations of this embodiment, after the authentication method of the target interface is configured, in order to test the correctness of the authentication configuration and the API data interface configuration, the present disclosure can perform a connectivity test. Specifically, the present disclosure can simulate real interface authentication and API data interface requests in real time. When the connectivity test passes, it means that the above configuration items are configured correctly and completely. When the connectivity test fails, the user can adjust the configuration parameters according to the relevant error prompts.

[0087] Step 305, in response to determining that the dynamic authentication parameters are obtained through the authentication interface, the identifier of the data acquisition task and the dynamic authentication parameters are added to the cache queue, and the expiration time of the dynamic authentication parameters is recorded; before the expiration time arrives, the dynamic authentication parameters are updated through the preset authentication interface.

[0088] If the dynamic authentication parameters are obtained through the authentication interface, the identifier of the data acquisition task and the dynamic authentication parameters can be added to the cache queue. At the same time, the expiration time of the dynamic authentication parameters is recorded. Before the expiration time arrives, the dynamic authentication parameters are updated through the preset authentication interface. If there are multiple data acquisition tasks executed at the same time, each data acquisition task can be sorted in the cache queue according to the expiration time of the dynamic authentication parameters. That is, the dynamic authentication parameter at the top of the cache queue has the earliest expiration time.

[0089] In some optional implementations of this embodiment, a time period may be preset before the expiration time arrives, and updated dynamic authentication parameters may be obtained through a preset authentication interface. Here, the preset time period may be set to 5 minutes.

[0090] In some optional implementations of the present embodiment, the cache queue includes a first cache queue and a second cache queue. The first cache queue is used to store the identifier of the data acquisition task and the expiration time of the dynamic authentication parameters. The second cache queue is used to store the identifier of the data acquisition task that is about to update the dynamic authentication parameters. Specifically, when the expiration time is preset, the identifier of the data acquisition task is obtained from the first cache queue and added to the second cache queue. In response to the listening thread listening to a new event generated in the second cache queue, the identifier of the data acquisition task is taken out from the second cache queue, and the updated dynamic authentication parameters are obtained through the preset authentication interface.

[0091] In some optional implementations of this embodiment, after obtaining the updated dynamic authentication parameters, the updated dynamic authentication parameters can be used to replace the corresponding dynamic authentication parameters in the cache queue, and the order of data acquisition tasks in the cache queue can be updated according to the expiration time of the updated dynamic authentication parameters.

[0092] Step 306, in response to determining that the dynamic authentication parameter is not obtained through the authentication interface, the dynamic authentication parameter is obtained, and the dynamic authentication parameter is sent to the target interface.

[0093] In this embodiment, if the authentication method of the target interface is not to obtain through the authentication interface, the dynamic authentication parameters can be directly obtained from the server providing the authentication parameter service, and the dynamic authentication parameters are sent to the target interface.

[0094] The use of Token for interface authentication is becoming more and more widely used due to its advantages such as small size, diverse transmission methods, and support for cross-domain verification. Since Token is time-sensitive, in the process of API interface data integration, it may take a considerable amount of time to call the API request multiple times in a cycle. If before each call to the API data interface, a request is made to the authentication interface to obtain the latest Token, and then the latest Token is used to request the API data interface, although this method can solve the problem of Token validity, it is time-consuming to call the authentication interface. When the number of calls is too many, the efficiency of the entire API interface data integration will be greatly reduced. In order to solve the extra time consumed by multiple calls to the authentication interface to obtain Token, the present disclosure can cache the Token tokens within the validity period so that they can be reused.

[0095] In some specific practices, the cache mechanism process of the present disclosure (such as Figure 6 As shown) as follows:

[0096] (1) After the data acquisition task of the target interface is configured, when the task is run for the first time, the authentication interface will be called to obtain the response result of the target interface and parse it to obtain the Token value.

[0097] (2) With the help of redis (Remote Dictionary Server), the data acquisition task ID is used as the key and the token value of the previous step is used as the value, and the data is saved in redis for caching.

[0098] (3) Before sending a request to the target interface each time, the Token value corresponding to the ID of the data acquisition task will be obtained from the redis cache, and then the Token will be set to the request parameter corresponding to the target interface according to the authentication mapping configuration.

[0099] It is worth mentioning that according to the data on the redis official website, the query performance of redis can reach 100,000 times per second, and its query efficiency is very high. Therefore, using redis to save and query Token values ​​can significantly reduce time consumption and is also suitable for distributed scenarios.

[0100] The above cache mechanism avoids the time-consuming frequent calls to the authentication interface, improves the overall efficiency of API data integration, and ensures the high efficiency of data integration for scenarios that require multiple batches, long periods of time, and large amounts of API interface data collection.

[0101] Since the Token uses an expiration time mechanism to ensure its validity, once it expires, the Token is considered invalid and does not have access rights. In order to ensure that the Token carried each time the target interface is requested is valid in API interface data integration, it is necessary to obtain the latest valid Token before the Token expires and replace the previous Token, so as to achieve automatic refresh of the Token.

[0102] In order to achieve a similar timer effect and accurately control each Token to automatically refresh the validity period before it expires, the Redisson delay queue function is introduced here (Redisson is an advanced distributed coordination redis client).

[0103] Redisson provides two queues, RBlocingQueue and RDelayDeque, to implement the function of delayed queue. RDelayDeque is used to store delayed tasks and set the delay time (such as 2 hours), and RBlocingQueue is used to obtain delayed tasks. The principle is: when the set delay time is reached, RDelayDeque will put the stored delayed tasks into the RBlocingQueue. The system needs to start a thread to monitor the event changes in the RBlocingQueue. When a new event is generated, the delayed task is taken out from the RBlocingQueue for processing.

[0104] The specific process of the Token automatic refresh mechanism is as follows:

[0105] (1) After the data acquisition task of the target interface starts running, when the authentication interface is called for the first time to obtain the Token, the ID of the data acquisition task will be used as a delayed task, and the corresponding Token expiration time (generally five minutes in advance to ensure that the Token does not expire at all) will be used as the delay time and set together in the RDelayDeque queue.

[0106] (2) When the set delay time is reached, the RDelayDeque queue will put the stored delayed tasks into the RBlocingQueue queue.

[0107] (3) The listening thread will monitor whether there are new events generated from the RBlocingQueue queue. When a new event is generated, it will take out the delayed task from the RBlocingQueue queue for processing. At this time, it will obtain the task id corresponding to the Token that is about to expire.

[0108] (4) After the listening thread obtains the ID of the data acquisition task, it will start a new thread to request the authentication interface to obtain the latest valid Token, and use the ID of the data acquisition task as the key and the Token value as the value to save it back to Redis for caching. In this way, since the key value has not changed, it is actually overwritten with the new Token.

[0109] (5) Continue to use the data acquisition task ID of the task as a delayed task, and the corresponding token expiration time (usually five minutes in advance to ensure that the token does not expire at all) as the delay time, and set them together in the RDelayDeque queue;

[0110] (6) Repeat the above steps.

[0111] In some optional implementations of this embodiment, after the data is acquired through the target interface, the acquired data may be integrated.

[0112] The data acquisition method provided by the above-mentioned embodiments of the present disclosure can provide a visual configuration interface to perform authentication configuration of API interface data integration according to the authentication requirements of different API interfaces, which complies with the authentication process of the API interface. In addition, through the connectivity test function provided by the visual interface, the authentication of the API interface can be tested in real time to ensure the accuracy of the configuration; in addition, the present solution supports API interface authentication in the form of API keys, Basic Authentication, Tokens, etc., which can cope with the authentication requirements of most API interfaces and is suitable for API interface data integration with authentication requirements; the cache mechanism designed in the present disclosure can reuse the authentication token, reduce the extra time brought by authentication, improve the efficiency of API interface data integration, and ensure the high efficiency of data integration; the Token automatic refresh mechanism designed in the present solution can automatically refresh the Token expiration time for the Token authentication method to ensure the validity of the Token in the API interface data integration; the present solution also has the characteristics of simple implementation and low cost, and can be applied to most data integration tools or frameworks, with strong applicability.

[0113] Further references Figure 7 As an implementation of the methods shown in the above figures, the present disclosure provides an embodiment of a data acquisition device, which is similar to Figure 2 Corresponding to the method embodiment shown, the device can be specifically applied to various electronic devices.

[0114] like Figure 7As shown, the data acquisition device 700 of this embodiment includes: an information acquisition unit 701 , a first determination unit 702 , a second determination unit 703 and a parameter updating unit 704 .

[0115] The information acquisition unit 701 is configured to, in response to receiving a data acquisition task, acquire configuration information of a target interface corresponding to the data acquisition task.

[0116] The first determining unit 702 is configured to determine the authentication method of the target interface according to the configuration information.

[0117] The second determining unit 703 is configured to, in response to determining that the authentication mode is a dynamic authentication mode, determine a method for acquiring dynamic authentication parameters according to the configuration information.

[0118] The parameter updating unit 704 is configured to, in response to determining that the dynamic authentication parameter is obtained through the authentication interface, update the dynamic authentication parameter through the authentication interface before the dynamic authentication parameter expires to complete the data acquisition task.

[0119] In addition, in the technical solution of the present application, an electronic device is also proposed.

[0120] Figure 8 A schematic structural diagram of an electronic device provided by an embodiment of the present disclosure is shown.

[0121] like Figure 8 As shown, the electronic device may include a processor 801, a memory 802, a bus 803, and a computer program stored in the memory 802 and executable on the processor 801, wherein the processor 801 and the memory 802 communicate with each other through the bus 803. When the processor 801 executes the computer program, the steps of implementing the above method include, for example, in response to receiving a data acquisition task, acquiring configuration information of a target interface corresponding to the data acquisition task; determining an authentication method of the target interface according to the configuration information; in response to determining that the authentication method is a dynamic authentication method, determining an acquisition method of dynamic authentication parameters according to the configuration information; in response to determining that the acquisition method of dynamic authentication parameters is to be acquired through an authentication interface, updating the dynamic authentication parameters through the authentication interface before the dynamic authentication parameters expire to complete the data acquisition task.

[0122] In addition, in one embodiment of the present disclosure, a non-transitory computer-readable storage medium is provided, on which a computer program is stored. When the computer program is executed by a processor, the steps of the above method are implemented, for example, including: in response to receiving a data acquisition task, obtaining configuration information of a target interface corresponding to the data acquisition task; determining an authentication method of the target interface according to the configuration information; in response to determining that the authentication method is a dynamic authentication method, determining an acquisition method of dynamic authentication parameters according to the configuration information; in response to determining that the acquisition method of dynamic authentication parameters is to obtain through an authentication interface, updating the dynamic authentication parameters through the authentication interface before the dynamic authentication parameters expire to complete the data acquisition task. .

[0123] In summary, in the technical solution disclosed in the present invention, when a data acquisition task is received, it can be determined whether the target interface requires dynamic authentication based on the configuration information of the target interface. When it is determined that the dynamic authentication parameters need to be obtained multiple times to complete the above data acquisition task, the dynamic authentication parameters can be automatically updated before the dynamic authentication parameters expire to ensure that the data acquisition task is completed without feeling, which not only ensures the security of the target interface, but also improves the completion efficiency of the data acquisition task.

[0124] The above description is only a preferred embodiment of the present disclosure and is not intended to limit the present disclosure. Any modifications, equivalent substitutions, improvements, etc. made within the spirit and principles of the present disclosure shall be included in the scope of protection of the present disclosure.

Claims

1. A data acquisition method, comprising: In response to receiving the data acquisition task, acquiring configuration information of the target interface corresponding to the data acquisition task; Determining an authentication method for the target interface according to the configuration information; In response to determining that the authentication mode is a dynamic authentication mode, determining a method for obtaining dynamic authentication parameters according to the configuration information; In response to determining that the dynamic authentication parameter is acquired through an authentication interface, the dynamic authentication parameter is updated through the authentication interface before the dynamic authentication parameter expires to complete the data acquisition task.

2. The method according to claim 1, wherein: The method further comprises: In response to determining that the authentication mode is a static authentication mode, a static authentication parameter is sent to the target interface to execute the data acquisition task.

3. The method according to claim 1, wherein: The method further comprises: In response to determining that the dynamic authentication parameter is not acquired through the authentication interface, the dynamic authentication parameter is acquired and sent to the target interface.

4. The method according to claim 1, wherein: The determining, based on the configuration information, whether it is necessary to obtain authentication parameters multiple times to complete the data acquisition task includes: The acquisition method is determined according to the parameter in the configuration information used to indicate the dynamic authentication parameter acquisition method.

5. The method according to claim 4, wherein: The updating of the dynamic authentication parameters before the dynamic authentication parameters expire includes: Adding the identifier of the data acquisition task and the dynamic authentication parameters to a cache queue, and recording the expiration time of the dynamic authentication parameters; Before the expiration time arrives, updated dynamic authentication parameters are obtained through a preset authentication interface according to the identifier of the data acquisition task stored in the cache queue.

6. The method according to claim 5, wherein: The step of obtaining updated dynamic authentication parameters through a preset authentication interface before the expiration time arrives includes: When the preset time period is left before the expiration time, the updated dynamic authentication parameters are obtained through the authentication interface.

7. The method according to claim 6, wherein: The cache queue includes a first cache queue and a second cache queue; and Before the expiration time arrives, according to the identifier of the data acquisition task stored in the cache queue, the dynamic authentication parameter is updated through the preset authentication interface, including: When the preset time is left before the expiration time, obtaining the identifier of the data acquisition task from the first cache queue and adding it to the second cache queue; In response to the monitoring thread monitoring a new event generated in the second cache queue, the identifier of the data acquisition task is taken out from the second cache queue, and the updated dynamic authentication parameter is obtained through a preset authentication interface.

8. The method according to claim 7, wherein: The method further comprises: In response to acquiring the updated dynamic authentication parameter, the updated dynamic authentication parameter replaces the dynamic authentication parameter corresponding to the identifier of the data acquisition task in the cache queue.

9. The method according to claim 1, wherein: The obtaining the configuration information of the target interface corresponding to the data acquisition task includes: The configuration information of the target interface is obtained from a preset configuration information configuration platform.

10. The method according to claim 1, wherein: The method further comprises: Integrate the acquired data.

11. A data acquisition device, comprising: an information acquisition unit, configured to, in response to receiving a data acquisition task, acquire configuration information of a target interface corresponding to the data acquisition task; A first determining unit is configured to determine an authentication method of the target interface according to the configuration information; a second determining unit configured to, in response to determining that the authentication mode is a dynamic authentication mode, determine a method for obtaining dynamic authentication parameters according to the configuration information; The parameter updating unit is configured to, in response to determining that the acquisition method of the dynamic authentication parameter is acquisition through the authentication interface, update the dynamic authentication parameter through the authentication interface before the dynamic authentication parameter expires to complete the data acquisition task.

12. An electronic device comprising a memory, a processor, a bus, and a computer program stored in the memory and executable on the processor, wherein: When the processor executes the computer program, the data acquisition method according to any one of claims 1 to 10 is implemented.

13. A non-transitory computer-readable storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, the data acquisition method according to any one of claims 1 to 10 is implemented.

Images