Data acquisition method, device, electronic device and storage medium

Abstract

The embodiments of the present invention disclose a data acquisition method, device, electronic device, and storage medium. The method includes: determining a text file including at least two data acquisition request parameters; parsing the text file using a data warehouse client tool to determine the data column index to which the at least two data acquisition request parameters belong; organizing the values ​​of the data fields included in the data column index into request parameters, initiating a request to a server to obtain target data, and processing the obtained target data. Using the technical solution of the embodiments of the present invention, a text file with a mapping relationship between the data column index and the data acquisition request parameters is established, the text file is parsed using a data warehouse client tool to determine the required target data; using the text file as a parameter mapping source, the request parameters are dynamically parsed and assembled, thereby realizing automated requests of the HTTP interface.

Description

Technical Field

[0001] The embodiments of the present invention relate to the field of computer application technology, and in particular to a data acquisition method, device, electronic device, and storage medium. Background Art

[0002] In the field of big data collection, data sources exist in the form of Web services, providing a passive data query interface that allows data to be retrieved via HTTP requests. When the amount of data to be retrieved is excessively large or the data types are diverse, a single access to a single Web service interface cannot retrieve all the target data. Multiple requests with different parameters are often required to retrieve data in batches. The HTTP interface for data query limits query subtype parameters, preventing the simultaneous query of all parent types. When requesting to retrieve all data via HTTP, due to the huge amount of data, frequent access to the HTTP interface can cause a significant increase in pressure on the Web service, leading to crashes.

[0003] Therefore, how to obtain a large amount of data is a technical problem that needs to be solved urgently by those skilled in the art. Summary of the Invention

[0004] The embodiments of the present invention provide a data acquisition method, device, electronic device and storage medium, which automatically execute HTTP based on dynamic parameter assembly of text files, solving the problem of relying on HTTP interface to obtain large amounts of data.

[0005] In a first aspect, an embodiment of the present invention provides a data acquisition method, comprising:

[0006] determining a text file including at least two data acquisition request parameters;

[0007] Parsing the text file using a data warehouse client tool to determine the data column index to which the at least two data acquisition request parameters belong;

[0008] The values ​​of the data fields included in the data column index are organized into request parameters, a request is initiated to the server to obtain target data, and the obtained target data is processed.

[0009] In a second aspect, an embodiment of the present invention further provides a data acquisition device, comprising:

[0010] A text file determining module, configured to determine a text file including at least two data acquisition request parameters;

[0011] a data column index determination module, configured to parse the text file using a data warehouse client tool to determine the data column indexes to which the at least two data acquisition request parameters belong;

[0012] The data acquisition request module is used to organize the values ​​of the data fields included in the data column index into request parameters, initiate a request to the server to obtain target data and process the obtained target data.

[0013] In a third aspect, an embodiment of the present invention further provides an electronic device, the electronic device comprising:

[0014] one or more processors;

[0015] a storage device for storing one or more programs;

[0016] When the one or more programs are executed by the one or more processors, the one or more processors implement the data acquisition method described in any embodiment of the present invention.

[0017] In a fourth aspect, an embodiment of the present invention further provides a computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements the data acquisition method described in any embodiment of the present invention.

[0018] Embodiments of the present invention provide a data acquisition method, apparatus, electronic device, and storage medium. These methods involve determining a text file containing at least two data acquisition request parameters; parsing the text file using a data warehouse client tool to determine the data column indexes to which the at least two data acquisition request parameters belong; organizing the values ​​of the data fields included in the data column indexes into request parameters, initiating a request to a server to acquire target data, and processing the acquired target data. Using the technical solution of the embodiments of the present invention, a text file is established with a mapping relationship between the data column indexes and the data acquisition request parameters; the text file is parsed using a data warehouse client tool to determine the required target data; and the text file is used as a parameter mapping source to dynamically parse and assemble the request parameters, thereby implementing automated requests for an HTTP interface. BRIEF DESCRIPTION OF THE DRAWINGS

[0019] Other features, objects, and advantages of the present invention will become more apparent upon reading the detailed description of non-limiting embodiments made with reference to the following drawings. The drawings are for the purpose of illustrating preferred embodiments only and are not to be considered as limiting the present invention. Like reference characters are used throughout the drawings to denote like parts. In the drawings:

[0020] Figure 1A This is a flow chart of a data acquisition method provided by the first embodiment of the present invention;

[0021] Figure 1B This is a schematic diagram of a process and configuration for obtaining data through an HTTP interface request provided by an embodiment of the present invention;

[0022] Figure 1C This is a schematic diagram of the structure of a text file provided by an embodiment of the present invention;

[0023] Figure 2A A flowchart of a data acquisition method provided in the second embodiment of the present invention;

[0024] Figure 2B This is a schematic diagram of a multiple request structure with different URL parameters provided by an embodiment of the present invention;

[0025] Figure 2C This is a schematic diagram of the structure of a parameter value set provided by an embodiment of the present invention;

[0026] Figure 3 This is a structural diagram of a data acquisition device provided by Embodiment 3 of the present invention;

[0027] Figure 4 This is a structural diagram of an electronic device provided in Example 4 of the present application. DETAILED DESCRIPTION

[0028] The present invention will be further described in detail below with reference to the accompanying drawings and examples. It will be understood that the specific embodiments described herein are intended only to illustrate the present invention and are not intended to limit the present invention. It should also be noted that, for ease of description, the accompanying drawings only illustrate portions relevant to the present invention, not all structures.

[0029] Before discussing the exemplary embodiments in more detail, it should be mentioned that some exemplary embodiments are described as processes or methods depicted as flow charts. Although the flow charts describe the various operations (or steps) as sequential processes, many of the operations (or steps) therein can be implemented in parallel, concurrently, or simultaneously. In addition, the order of the various operations can be rearranged. The process can be terminated when its operation is completed, but can also have additional steps not included in the accompanying drawings. The process can correspond to a method, function, procedure, subroutine, subprogram, etc.

[0030] Example 1

[0031] Figure 1A This is a flow chart of a data acquisition method provided in Example 1 of the present invention. This embodiment is applicable to situations where large amounts of data are acquired. The method of this embodiment can be performed by a data acquisition device, which can be implemented using hardware and / or software. The device can be configured in a data acquisition server. The method specifically includes the following steps:

[0032] S110: Determine a text file including at least two data acquisition request parameters.

[0033] In the field of big data, there are many mature and practical tools for data collection and transmission. Most of the data sources they target are databases, log files, and message queue services. There are few tool software that use HTTP interfaces as data sources. Even if there are such tools (such as Flume), their functions are relatively simple and cannot handle complex and changing production environments.

[0034] Figure 1B This is a flow chart and configuration diagram of a HTTP interface request to obtain data provided by an embodiment of the present invention. Figure 1B The HTTP interface includes but is not limited to method type, URL address, configuration parameters, request parameters and request results; among them, interface A is the method type, including but not limited to POST and GET; interface B is the Uniform Resource Locator (URL) address; interface C is the configuration parameters, such as Headers configuration information; interface D is the request parameters, including but not limited to URL parameters, form parameters and Body parameters; interface E is the request result, including but not limited to status code, Headers information and response Body data.

[0035] The data acquisition request parameter may refer to a request parameter for acquiring data, see Figure 1B The data acquisition request parameters may be Body parameters, including but not limited to name and age.

[0036] In an optional solution of the embodiment of the present invention, optionally, determining a text file including at least two data acquisition request parameters includes:

[0037] determining a text file based on at least two data acquisition request parameters;

[0038] The text file includes at least two rows of row data; the row data includes a data field and a column separator; and the at least two data acquisition request parameters are represented by data fields and separated by a column separator.

[0039] Wherein, the text file may refer to a case File file, Figure 1C This is a schematic diagram of the structure of a text file provided by an embodiment of the present invention, see Figure 1C, the text file includes at least two lines of row data. In the case File file, the data in each line is called row data; the row data includes data fields and column separators; the at least two data acquisition request parameters are represented by data fields and separated by column separators. The content of the row data consists of one or more meaningful values, called data fields, and the data fields are also called data columns or column data; if the row data has multiple data fields, each data field is separated by a fixed separator, and the data field separators of all row data should be consistent. The separator forms include but are not limited to English colons, English commas, and English underscores; each data field has a sequence number based on its position in the row data, called a data column index, and the data column index is also called a column index or field index, and the index value starts from 0; for example, see Figure 1C , CCOP is a data field with an index value of 1, and is separated from other data fields by English semicolons.

[0040] S120: Use a data warehouse client tool to parse the text file and determine the data column index to which the at least two data acquisition request parameters belong.

[0041] The data warehouse client tool (DWHCLI) may refer to a tool software for HTTP data source extraction and offline large-scale (for example, GB or above) text file preprocessing. The data warehouse client tool is used to parse the text file, determine the data structure of the text file, and determine the data column index to which the at least two data acquisition request parameters belong based on the data structure of the text file. For example, the data warehouse client tool is used to parse the case File text file, determine the row data structure in the case File text file, and determine the index values ​​to which at least two data fields in the row data belong. For example, see Figure 1C , the data column index value of the data field CCOP is 1.

[0042] S130: Organize the values ​​of the data fields included in the data column index into request parameters, initiate a request to the server to obtain target data, and process the obtained target data. Organizing the values ​​of the data fields included in the data column index into request parameters may mean initiating a data acquisition request to the server based on the target data required by the user. For example, if a user wants to obtain the personal information of a company employee, including the employee's name, age, and registered residence, the client needs to send part of the employee's personal information to the server, where the part of the employee's personal information refers to the request parameters. For example, the employee's name and age are organized into request parameters, and the client sends the request parameters to the server to initiate a data acquisition request to obtain the target data. For example, if the index value corresponding to the name is 1 and the index value corresponding to the age is 2, the values ​​of the data fields included in the index values ​​1 and 2 are obtained. For example, if the obtained name "Zhang San" and age "23" are used as request parameters, a data acquisition request is initiated to the server, so that the server can determine the personal information required by the client based on the employee's name and age.

[0043] If the text file doesn't include the data retrieval request parameters the user needs, they need to be added to the text file and configured accordingly in the data warehouse client tool to retrieve the required data. This automated HTTP execution approach, using case file-based dynamic parameter assembly, effectively solves several issues associated with relying on HTTP interfaces to retrieve large amounts of data.

[0044] The client initiates a data acquisition request to obtain target data and processes the target data. This includes the client initiating a data acquisition request to obtain target data and saving portions of the target data based on the user's desired information. For example, if the user wishes to save the employee's name, age, and registered address, the user simply specifies the key path corresponding to the data to be retained, known as the key-path. The program then automatically parses and saves the data structure.

[0045] An embodiment of the present invention provides a data acquisition method, which comprises determining a text file including at least two data acquisition request parameters; parsing the text file using a data warehouse client tool to determine the data column indexes to which the at least two data acquisition request parameters belong; organizing the values ​​of the data fields included in the data column indexes into request parameters, initiating a request to a server to obtain target data, and processing the obtained target data. Using the technical solution of an embodiment of the present invention, a text file with a mapping relationship between the data column indexes and the data acquisition request parameters is established, parsed using a data warehouse client tool, and the required target data is determined. Using the text file as a parameter mapping source, the parameters are dynamically parsed and assembled into request parameters, thereby implementing automated requests for the HTTP interface.

[0046] Example 2

[0047] Figure 2A This is a flow chart of a data acquisition method provided by the second embodiment of the present invention. The embodiment of the present invention further optimizes the above embodiment on the basis of the above embodiment. The embodiment of the present invention can be combined with various optional solutions in one or more of the above embodiments. Figure 2A As shown, the data acquisition method provided in the embodiment of the present invention may include the following steps:

[0048] S210: Determine a text file including at least two data acquisition request parameters.

[0049] Due to access efficiency and computing efficiency considerations, a large amount of dynamic data and static data are stored in the database using the data structure of a typical relational database. If the data is directly extracted from the database, it is almost impossible to parse it. The data query HTTP interface provided by the middle-office system and business system allows HTTP clients to query and obtain data, and the data structure can be interpreted.

[0050] However, this also has certain problems: the amount of data requested by a single HTTP interface is limited, and it is not suitable for obtaining the full amount of data in one request; some HTTP interfaces for data query limit the query subtype parameters, and do not allow all parent types of data to be queried at once. For example, the interface for querying device records allows specifying a specific device subtype for querying, but does not allow ignoring the device subtype and querying all device records. However, there are thousands of device subtypes, which means that in order to query the data of all devices, thousands of HTTP interfaces need to be accessed; accessing the HTTP interface too frequently may cause request pressure on the HTTP server, causing service degradation or paralysis.

[0051] In summary, it is feasible to use the HTTP interface for data acquisition, but there are many limitations. In order to obtain the full amount of data of a certain type, it is often necessary to make N HTTP interface requests, and the HTTP request parameters are different for each HTTP request. Usually, this data acquisition strategy requires the writing of customized code programs or scripts to implement, and requires the user who extracts data to have certain code writing capabilities. Therefore, an embodiment of the present invention provides a data acquisition method, which determines a text file based on at least two data acquisition request parameters, establishes a mapping relationship between a data column index and a data acquisition request parameter, and uses a data warehouse client tool to acquire target data, thereby reducing code writing and simplifying operation.

[0052] S220: Use a data warehouse client tool to parse the text file to obtain the data structure of the line data in the text file.

[0053] Among them, the data warehouse client tool parses the text file through preset rules, generates HTTP request parameters in units of row data, and then initiates a data request to the HTTP service. In an optional solution of an embodiment of the present invention, optionally, JavaIO technology can be used to read and parse the text file data into a data structure of a two-dimensional set, which is called caseC. Each element of caseC is an array of fixed length, referred to as caseC-A. The elements of caseC-A are the data fields of a row of data in the text file.

[0054] HTTP request parameters (whether URL parameters, form parameters, or Body parameters) can be regarded as key-value pair data structures in KV format, where K (KEY) is fixed and V (value) is variable. Figure 2B This is a schematic diagram of a multiple request structure with different URL parameters provided by an embodiment of the present invention. Figure 2B Taking URL parameters as an example, URL parameters include project and type. The values ​​of these two parameters are a finite set. All data obtained by multiple HTTP requests constitute this set is the target data to be obtained. Figure 2C This is a schematic diagram of a parameter value set provided by an embodiment of the present invention. Figure 2C , this text file stores a set of parameter values.

[0055] In an optional solution of the embodiment of the present invention, optionally, the parsing the text file using a data warehouse client tool to obtain the data structure of the line data in the text file further includes:

[0056] When parsing the text file, if a parsing error occurs and the parsing error is a predicted error, suspending the data acquisition request;

[0057] Alternatively, if a parsing error occurs on the server side, the parsing of the text file is automatically terminated and the parsing error is recorded in a log; when the text file is parsed again, a data acquisition request is continued based on the position recorded in the log.

[0058] Among them, the text file is parsed using a data warehouse client tool. If a parsing error occurs and the parsing error is a predicted error, the data acquisition request is suspended; if a parsing error occurs on the server side, the text file parsing is automatically terminated and the parsing error is recorded in a log; when the text file is parsed again, the data acquisition request is continued to be initiated based on the position recorded in the log.

[0059] S230: Determine, based on the data structure of the row data, the data column indexes to which the at least two data acquisition request parameters belong.

[0060] In an optional solution of the embodiment of the present invention, optionally, determining the data column index to which the at least two data acquisition request parameters belong based on the data structure of the row data includes:

[0061] Parsing the text file using a data warehouse client tool to obtain a data structure of the line data in the text file;

[0062] According to the data structure of the row data, the data column index to which the at least two data acquisition request parameters belong is determined; wherein the data column index represents the location of the at least two data acquisition request parameters.

[0063] Among them, the dynamic parameter connection is established by specifying the correspondence between the data column index in the case File row data and the URL data acquisition request parameter K. The mapping rules specified by DWHCL are: keyName1:indexNum1; keyName2:indexNum2;... The mapping between the data acquisition request parameter K and the data column index in the case File is established using an English colon, and multiple mapping relationships are separated by English semicolons. The established mapping relationship syntax can be: project:0; type:1. After DWHCLI establishes the mapping between the case File and the URL parameters, it realizes the automation of dynamically assembling HTTP parameters and successively acquiring data by looping through the caseC collection and executing HTTP requests. This entire process only requires the user to perform a simple mapping configuration between the case File and the parameter K, and does not require the user who extracts data to have certain coding skills.

[0064] S240: Organize the values ​​of the data fields included in the data column index into request parameters, initiate a request to the server to obtain target data, and process the obtained target data.

[0065] In an optional solution of the embodiment of the present invention, optionally, organizing the values ​​of the data fields included in the data column index into request parameters, initiating a request to the server to obtain target data, and processing the obtained target data includes:

[0066] Determining, based on the data column index to which the at least two data acquisition request parameters belong, a value of a data field included in the data column index;

[0067] The values ​​of the data fields included in the data column index are organized into request parameters, a request is initiated to the server to obtain target data, and the obtained target data is processed.

[0068] Among them, returning HTTP request data through the response body is currently a more common method. DWHCLI has made targeted data structure parsing designs for the JSON structure data returned in this way.

[0069] The response body result example is as follows:

[0070]

[0071] In most cases, the returned results are only of interest to the data array. Therefore, DWHCLI provides a configuration for mapping key values ​​to JSON data structures. Simply specify the path to the key corresponding to the data you want to retain, called a key-path, and the program will automatically parse and save the JSON data structure. For example, to save the data array, configure the key-path to content.data and select the data structure of content.data as "array."

[0072] An embodiment of the present invention provides a data acquisition method, comprising: determining a text file including at least two data acquisition request parameters; parsing the text file using a data warehouse client tool to obtain the data structure of the row data within the text file; determining, based on the data structure of the row data, the data column indexes to which the at least two data acquisition request parameters belong; organizing the values ​​of the data fields included in the data column indexes into request parameters, initiating a request to a server to obtain target data, and processing the obtained target data. Using the technical solution of the embodiment of the present invention, for scenarios requiring multiple HTTP requests with different parameters to acquire data, the data warehouse client tool provides simple configuration content to dynamically assemble parameters, automatically execute request logic, and parse and store returned results.

[0073] Example 3

[0074] Figure 3 3 is a schematic diagram of the structure of a data acquisition device provided by the third embodiment of the present invention, which includes: a text file determination module 310, a data column index determination module 320 and a data acquisition request module 330.

[0075] A text file determining module 310 is configured to determine a text file including at least two data acquisition request parameters;

[0076] A data column index determination module 320 is configured to parse the text file using a data warehouse client tool to determine the data column indexes to which the at least two data acquisition request parameters belong;

[0077] The data acquisition request module 330 is configured to organize the values ​​of the data fields included in the data column index into request parameters, initiate a request to the server to acquire target data, and process the acquired target data.

[0078] Based on the above embodiment, optionally, the text file determination module includes:

[0079] determining a text file based on at least two data acquisition request parameters;

[0080] The text file includes at least two rows of row data; the row data includes a data field and a column separator; and the at least two data acquisition request parameters are represented by data fields and separated by a column separator.

[0081] Based on the above embodiment, optionally, the data column index determination module includes:

[0082] Parsing the text file using a data warehouse client tool to obtain a data structure of the line data in the text file;

[0083] According to the data structure of the row data, the data column index to which the at least two data acquisition request parameters belong is determined; wherein the data column index represents the location of the at least two data acquisition request parameters.

[0084] Based on the above embodiment, optionally, the data acquisition request result returning module includes:

[0085] Determining, based on the data column index to which the at least two data acquisition request parameters belong, a value of a data field included in the data column index;

[0086] The values ​​of the data fields included in the data column index are organized into request parameters, a request is initiated to the server to obtain target data, and the obtained target data is processed.

[0087] Based on the above embodiment, optionally, the data column index determination module further includes:

[0088] When parsing the text file, if a parsing error occurs and the parsing error is a predicted error, suspending the data acquisition request;

[0089] Alternatively, if a parsing error occurs on the server side, the parsing of the text file is automatically terminated and the parsing error is recorded in a log; when the text file is parsed again, a data acquisition request is continued based on the position recorded in the log.

[0090] The above-mentioned device can execute the data acquisition method provided by any embodiment of the present invention, and has the corresponding functional modules and beneficial effects for executing the data acquisition method.

[0091] Example 4

[0092] Figure 4 This is a structural diagram of an electronic device provided in the fourth embodiment of the present application. The present embodiment of the present application provides an electronic device, in which the interactive device for data acquisition provided in the embodiment of the present application can be integrated. Figure 4 As shown, this embodiment provides an electronic device 400, which includes: one or more processors 420; a storage device 410 for storing one or more programs. When the one or more programs are executed by the one or more processors 420, the one or more processors 420 implement the data acquisition method provided in the embodiment of the present application. The method includes:

[0093] determining a text file including at least two data acquisition request parameters;

[0094] Parsing the text file using a data warehouse client tool to determine the data column index to which the at least two data acquisition request parameters belong;

[0095] The values ​​of the data fields included in the data column index are organized into request parameters, a request is initiated to the server to obtain target data, and the obtained target data is processed.

[0096] Of course, those skilled in the art will appreciate that the processor 420 also implements the technical solution of the data acquisition method provided in any embodiment of the present application.

[0097] Figure 4 The electronic device 400 shown is merely an example and should not limit the functions and scope of use of the embodiments of the present application.

[0098] like Figure 4 As shown, the electronic device 400 includes a processor 420, a storage device 410, an input device 430, and an output device 440; the number of processors 420 in the electronic device can be one or more. Figure 4 In the figure, a processor 420 is used as an example; the processor 420, the storage device 410, the input device 430 and the output device 440 in the electronic device can be connected via a bus or other means. Figure 4 The connection via bus 450 is taken as an example.

[0099] The storage device 410 is a computer-readable storage medium that can be used to store software programs, computer executable programs, and module units, such as program instructions corresponding to the data acquisition method in the embodiment of the present application.

[0100] The storage device 410 may primarily include a program storage area and a data storage area. The program storage area may store an operating system and at least one application required for a function; the data storage area may store data created based on the use of the terminal, etc. Furthermore, the storage device 410 may include high-speed random access memory and may also include non-volatile memory, such as at least one disk storage device, flash memory device, or other non-volatile solid-state memory device. In some instances, the storage device 410 may further include a memory remotely located relative to the processor 420, and such remote memory may be connected via a network. Examples of the aforementioned networks include, but are not limited to, the Internet, an intranet, a local area network, a mobile communication network, and combinations thereof.

[0101] The input device 430 may be used to receive input numbers, character information or voice information, and generate key signal input related to user settings and function control of the electronic device. The output device 440 may include electronic devices such as a display screen and a speaker.

[0102] The electronic device provided in the embodiment of the present application can achieve the technical effect of automatically executing HTTP based on the dynamic parameter assembly of text files, effectively solving the problem of relying on the HTTP interface to obtain large amounts of data.

[0103] Example 5

[0104] A fifth embodiment of the present invention further provides a storage medium containing computer-executable instructions. When the computer-executable instructions are executed by a computer processor, the computer-executable instructions are used to perform a data acquisition method. The method includes:

[0105] determining a text file including at least two data acquisition request parameters;

[0106] Parsing the text file using a data warehouse client tool to determine the data column index to which the at least two data acquisition request parameters belong;

[0107] The values ​​of the data fields included in the data column index are organized into request parameters, a request is initiated to the server to obtain target data, and the obtained target data is processed.

[0108] The computer storage medium of the embodiment of the present invention may adopt any combination of one or more computer-readable media. The computer-readable medium may be a computer-readable signal medium or a computer-readable storage medium. The computer-readable storage medium may be, for example, but not limited to, an electrical, magnetic, optical, electromagnetic, infrared, or semiconductor system, device or component, or any combination thereof. More specific examples (a non-exhaustive list) of computer-readable storage media include: an electrical connection with one or more wires, a portable computer disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM), a flash memory, an optical fiber, a portable CD-ROM, an optical storage device, a magnetic storage device, or any suitable combination thereof. The computer-readable storage medium may be any tangible medium containing or storing a program that can be used by or in conjunction with an instruction execution system, device or device.

[0109] A computer-readable signal medium may include a data signal propagated in baseband or as part of a carrier wave, carrying computer-readable program code. Such a propagated data signal may take a variety of forms, including but not limited to electromagnetic signals, optical signals, or any suitable combination thereof. A computer-readable signal medium may also be any computer-readable medium other than a computer-readable storage medium that can transmit, propagate, or transport a program for use by or in conjunction with an instruction execution system, apparatus, or device.

[0110] The program code contained on the computer-readable medium may be transmitted using any appropriate medium, including but not limited to wireless, wire, optical cable, radio frequency (RF), etc., or any suitable combination of the foregoing.

[0111] Computer program code for performing the operations of the present invention may be written in one or more programming languages, or a combination thereof, including object-oriented programming languages ​​such as Java, Smalltalk, C++, and conventional procedural programming languages ​​such as "C" or similar programming languages. The program code may be executed entirely on the user's computer, partially on the user's computer, as a stand-alone software package, partially on the user's computer and partially on a remote computer, or entirely on a remote computer or server. In cases involving a remote computer, the remote computer may be connected to the user's computer through any type of network, including a local area network (LAN) or a wide area network (WAN), or may be connected to an external computer (e.g., through the Internet using an Internet service provider).

[0112] Throughout this specification, reference to terms such as "one embodiment," "some embodiments," "examples," "specific examples," or "some examples" means that a specific feature, structure, material, or characteristic described in conjunction with that embodiment or example is included in at least one embodiment or example of the present invention. In this specification, schematic representations of the above terms do not necessarily refer to the same embodiment or example. Furthermore, the specific features, structures, materials, or characteristics described may be combined in any suitable manner in any one or more embodiments or examples.

[0113] Note that the above are only preferred embodiments of the present invention and the technical principles employed. Those skilled in the art will understand that the present invention is not limited to the specific embodiments described herein, and that various obvious changes, readjustments, and substitutions can be made by those skilled in the art without departing from the scope of protection of the present invention. Therefore, although the present invention has been described in detail through the above embodiments, the present invention is not limited to the above embodiments and may include many other equivalent embodiments without departing from the concept of the present invention. The scope of the present invention is determined by the scope of the appended claims.

Claims

1. A data acquisition method, characterized in that: The method comprises: determining a text file including at least two data acquisition request parameters; Parsing the text file using a data warehouse client tool to determine the data column index to which the at least two data acquisition request parameters belong; Organizing the values ​​of the data fields included in the data column index into request parameters, initiating a request to the server to obtain target data and processing the obtained target data; The step of organizing the values ​​of the data fields included in the data column index into request parameters, initiating a request to a server to obtain target data, and processing the obtained target data includes: Based on the data column index to which the at least two data acquisition request parameters belong, the values ​​of the data fields included in the data column index are determined; the values ​​of the data fields included in the data column index are organized into request parameters, a request is initiated to the server to obtain target data, and the obtained target data is processed; wherein, if the data acquisition request parameters required by the user are not included in the text file, the corresponding data acquisition request parameters required by the user need to be added to the text file, and corresponding settings are made in the data warehouse client tool to obtain the data acquisition request parameters required by the user.

2. The method according to claim 1, characterized in that The determining of a text file including at least two data acquisition request parameters comprises: determining a text file based on at least two data acquisition request parameters; The text file includes at least two rows of row data; the row data includes a data field and a column separator; and the at least two data acquisition request parameters are represented by data fields and separated by a column separator.

3. The method according to claim 1, characterized in that The step of parsing the text file using a data warehouse client tool to determine the data column index to which the at least two data acquisition request parameters belong includes: Parsing the text file using a data warehouse client tool to obtain a data structure of the line data in the text file; According to the data structure of the row data, the data column index to which the at least two data acquisition request parameters belong is determined; wherein the data column index represents the location of the at least two data acquisition request parameters.

4. The method according to claim 1, wherein The step of parsing the text file using a data warehouse client tool to determine the data column index to which the at least two data acquisition request parameters belong further comprises: When parsing the text file, if a parsing error occurs and the parsing error is a predicted error, suspending the data acquisition request; Alternatively, if a parsing error occurs on the server side, the parsing of the text file is automatically terminated and the parsing error is recorded in a log; when the text file is parsed again, a data acquisition request is continued based on the position recorded in the log.

5. A data acquisition device, characterized in that: The device comprises: A text file determining module, configured to determine a text file including at least two data acquisition request parameters; a data column index determination module, configured to parse the text file using a data warehouse client tool to determine the data column indexes to which the at least two data acquisition request parameters belong; A data acquisition request module is used to organize the values ​​of the data fields included in the data column index into request parameters, initiate a request to the server to obtain target data and process the obtained target data; The data acquisition request module is specifically used to: Based on the data column index to which the at least two data acquisition request parameters belong, the values ​​of the data fields included in the data column index are determined; the values ​​of the data fields included in the data column index are organized into request parameters, a request is initiated to the server to obtain target data, and the obtained target data is processed; wherein, if the data acquisition request parameters required by the user are not included in the text file, the corresponding data acquisition request parameters required by the user need to be added to the text file, and corresponding settings are made in the data warehouse client tool to obtain the data acquisition request parameters required by the user.

6. The device according to claim 5, characterized in that The text file determination module includes: determining a text file based on at least two data acquisition request parameters; The text file includes at least two rows of row data; the row data includes a data field and a column separator; and the at least two data acquisition request parameters are represented by data fields and separated by a column separator.

7. The device according to claim 5, characterized in that The data column index determination module includes: Parsing the text file using a data warehouse client tool to obtain a data structure of the line data in the text file; According to the data structure of the row data, the data column index to which the at least two data acquisition request parameters belong is determined; wherein the data column index represents the location of the at least two data acquisition request parameters.

8. An electronic device, characterized in that: include: one or more processors; a storage device for storing one or more programs; When the one or more programs are executed by the one or more processors, the one or more processors implement the data acquisition method according to any one of claims 1 to 4.

9. A storage medium containing computer-executable instructions, characterized in that: When the computer executable instructions are executed by a computer processor, the computer executable instructions are used to perform the data acquisition method according to any one of claims 1 to 4.

Images