Data acquisition method and device, electronic equipment and medium

Abstract

Embodiments of the present application disclose a data acquisition method and device, electronic equipment and medium, which are applied to the field of cloud technology, and can be related to the field of big data technology. The method comprises: obtaining structure information of a to-be-scanned data table, the structure information comprising a field name and a field type of the to-be-scanned data table; obtaining a configuration file corresponding to the field type from a data scanning library according to the field type of the to-be-scanned data table; generating a data scanning tool based on the field name of the to-be-scanned data table and the configuration file; and obtaining target data from the to-be-scanned data table through the data scanning tool. The embodiments of the present application can improve the efficiency of data acquisition. The present application can also be related to the field of blockchain technology, and the obtained target data can be written into a blockchain.

Description

Technical Field

[0001] This application relates to the field of cloud technology, and in particular to a data acquisition method, apparatus, electronic device and medium. Background Technology

[0002] Currently, scanning data tables to retrieve data typically requires developers to manually write a complete data scanning plan to obtain data. However, the inventors realized that because data table structures vary widely, writing data scanning plans for different data tables is time-consuming and resource-intensive. Furthermore, compilation errors during the writing process may render the scanned data unusable, leading to low data retrieval efficiency. Therefore, improving the efficiency of data retrieval when scanning data tables has become a pressing issue. Summary of the Invention

[0003] This application provides a data acquisition method, apparatus, electronic device, and medium, which helps to improve the efficiency of data acquisition.

[0004] On one hand, embodiments of this application provide a data acquisition method, the method comprising:

[0005] Obtain the structure information of the data table to be scanned; the structure information includes the field names and field types of the data table to be scanned;

[0006] Based on the field types of the data table to be scanned, the configuration file corresponding to the field type is obtained from the data scanning library, and a data scanning tool is generated based on the field names of the data table to be scanned and the configuration file.

[0007] The target data is obtained from the data table to be scanned using the data scanning tool.

[0008] On one hand, embodiments of this application provide a data acquisition device, which includes:

[0009] The acquisition module is used to acquire the structural information of the data table to be scanned; the structural information includes the field names and field types of the data table to be scanned.

[0010] The processing module is used to obtain the configuration file corresponding to the field type from the data scanning library according to the field type of the data table to be scanned, and generate a data scanning tool based on the field name of the data table to be scanned and the configuration file;

[0011] The acquisition module is also used to acquire target data from the data table to be scanned using the data scanning tool.

[0012] On one hand, embodiments of this application provide an electronic device including a processor and a memory, wherein the memory is used to store a computer program, the computer program including program instructions, and the processor is configured to invoke the program instructions to execute some or all of the steps in the above method.

[0013] On one hand, embodiments of this application provide a computer-readable storage medium storing a computer program, the computer program including program instructions, which, when executed by a processor, are used to perform some or all of the steps in the above-described method.

[0014] In this embodiment, the structural information of the data table to be scanned, including field names and field types, can be obtained. Then, based on the field types of the data table, the corresponding configuration files for each field type can be retrieved from the data scanning library. A data scanning tool is then generated based on the field names and configuration files of the data table to be scanned, and the target data is retrieved from the data table using this tool. Implementing this scheme helps improve the efficiency of data acquisition. Attached Figure Description

[0015] To more clearly illustrate the technical solutions of the embodiments of the present invention, the drawings used in the following description of the embodiments will be briefly introduced. Obviously, the drawings described below are some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0016] Figure 1 A schematic diagram of an application architecture provided for an embodiment of this application;

[0017] Figure 2 A flowchart illustrating a data acquisition method provided in an embodiment of this application;

[0018] Figure 3 This is a schematic diagram illustrating an application scenario of a data scanning tool provided in an embodiment of this application;

[0019] Figure 4 A flowchart illustrating another data acquisition method provided in an embodiment of this application;

[0020] Figure 5 This is a schematic diagram illustrating an application scenario for acquiring data, provided in an embodiment of this application.

[0021] Figure 6 A flowchart illustrating a data acquisition method provided in an embodiment of this application;

[0022] Figure 7 This application provides an illustration of an application scenario for parallel scanning of a data table.

[0023] Figure 8 This is a schematic diagram of the structure of a data acquisition device provided in an embodiment of this application;

[0024] Figure 9 This is a schematic diagram of the structure of an electronic device provided in an embodiment of this application. Detailed Implementation

[0025] The technical solutions of the present invention will be clearly and completely described below with reference to the accompanying drawings in the embodiments of the present invention.

[0026] The data acquisition method proposed in this application can be implemented in an electronic device, which can be a server or a terminal device. The server can be a standalone physical server, a server cluster or distributed system composed of multiple physical servers, or a cloud server providing basic cloud computing services such as cloud services, cloud databases, cloud computing, cloud functions, cloud storage, network services, cloud communication, middleware services, domain name services, security services, CDN, and big data and artificial intelligence platforms. The terminal device can be a smartphone, tablet, laptop, desktop computer, smart speaker, smartwatch, etc., but is not limited to these.

[0027] This application's embodiments may relate to cloud technology-related technical fields, specifically the field of big data technology. Big data refers to data sets that cannot be captured, managed, and processed within a certain timeframe using conventional software tools. It represents massive, rapidly growing, and diverse information assets that require new processing models to achieve stronger decision-making, insight discovery, and process optimization capabilities. With the advent of the cloud era, big data has attracted increasing attention. Big data requires specialized technologies to effectively process large amounts of data within a tolerable timeframe. Technologies suitable for big data include massively parallel processing databases, data mining, distributed file systems, distributed databases, cloud computing platforms, the Internet, and scalable storage systems. By implementing the technical solutions of this application, it is possible to achieve large-scale parallel or serial scanning of data tables stored in a database to obtain data, and to improve the efficiency of data table scanning and data acquisition.

[0028] Please see Figure 1 , Figure 1 This is a schematic diagram of an application architecture provided for an embodiment of this application, through which the data acquisition method proposed in this application can be executed. For example... Figure 1 As shown, Figure 1The system may include an electronic device and a database storing data tables. The electronic device can generate a data scanning tool by executing the technical solution of this application, and use the data scanning tool to scan the data tables and obtain data from them. Optionally, the database storing the data tables may include a storage client and a storage service provider. The data tables may be stored on the storage service provider, and the storage client may be configured with a data acquisition interface. Through this interface, data can be obtained from the data tables on the storage service provider and sent to the electronic device. The electronic device receives the data sent by the storage client through the data scanning tool.

[0029] Understandable, Figure 1 This is merely an example of an application architecture for the technical solution of this application, and does not limit the specific architecture of the technical solution of this application. That is, the technical solution of this application can also provide other forms of application architecture.

[0030] Optionally, in some embodiments, the electronic device may execute the data acquisition method according to actual business needs to improve data acquisition efficiency.

[0031] For example, the technical solution of this application can be applied to data migration scenarios. In a user data migration scenario for direct debit payments, an electronic device can use the technical solution of this application to generate a data scanning tool for the data table storing user data. This tool can then scan the data table to obtain the user data. After obtaining the user data, it can be processed, such as transferring the user data to a designated location for storage to achieve data migration. It is understood that in data migration application scenarios, the technical solution of this application can be used to obtain the data that needs to be migrated.

[0032] For example, the technical solution of this application can also be applied to data aggregation scenarios. For instance, when merging product A and product B into a single product, it is necessary to aggregate the user authorization data of product A and product B. Electronic devices can use the technical solution of this application to generate a data table storing the user authorization data of product A and a data scanning tool storing the user authorization data of product B. The data scanning tool can then scan the corresponding data tables to obtain the user authorization data of product A and product B, thereby merging the two sets of user authorization data to achieve data aggregation. It is understood that in data aggregation application scenarios, the technical solution of this application can obtain the data that needs to be aggregated.

[0033] Optionally, the data involved in this application, such as the data table to be scanned and the data obtained by scanning the data table, can be stored in a database or in a blockchain. This application does not limit the storage to a blockchain distributed system. For example, the server-side, such as a storage service, can be a node in the blockchain; similarly, the client-side, such as a storage client, can be a node in the blockchain; or both the server and client can be nodes in the blockchain. Further optionally, the electronic equipment involved in this application, such as a server used to process the target data, can be a node in the blockchain.

[0034] Blockchain is a novel application model of computer technologies such as distributed data storage, peer-to-peer transmission, consensus mechanisms, and encryption algorithms. Essentially, a blockchain is a decentralized database, a chain of data blocks linked together using cryptographic methods. Each data block contains information about a batch of network transactions, used to verify the validity of the information (anti-counterfeiting) and generate the next block. A blockchain can include an underlying platform, a platform product service layer, and an application service layer.

[0035] The underlying blockchain platform can include processing modules such as user management, basic services, smart contracts, and operational monitoring. The user management module is responsible for managing the identity information of all blockchain participants, including maintaining public and private key generation (account management), key management, and maintaining the correspondence between user real identities and blockchain addresses (access management). Furthermore, under authorization, it monitors and audits transactions of certain real identities and provides risk control rule configuration (risk control audit). The basic services module is deployed on all blockchain node devices to verify the validity of business requests. After consensus is reached on valid requests, they are recorded in storage. For a new business request, the basic services first perform interface adaptation parsing and authentication (interface adaptation), and then encrypt the business information through a consensus algorithm (consensus management). After encryption, the data is transmitted completely and consistently to the shared ledger (network communication) and recorded and stored. The smart contract module is responsible for contract registration, issuance, triggering, and execution. Developers can define contract logic using a programming language and publish it to the blockchain (contract registration). According to the contract terms, the key or other events are invoked to trigger execution and complete the contract logic. It also provides functions for contract upgrades and cancellations. The operation monitoring module is mainly responsible for deployment, configuration modification, contract settings, cloud adaptation, and real-time status visualization output during product release, such as alarms, monitoring network conditions, and monitoring the health status of node devices.

[0036] The platform's product service layer provides basic capabilities and implementation frameworks for typical applications. Developers can leverage these basic capabilities, add business characteristics, and complete the blockchain implementation of business logic. The application service layer provides application services based on blockchain solutions for use by business stakeholders. For example, in this application, data storage functionality can be provided through blockchain, and the stored data can be made available to developers.

[0037] It is understood that the above scenarios are merely examples and do not constitute a limitation on the application scenarios of the technical solutions provided in the embodiments of this application. The technical solutions of this application can also be applied to other scenarios. For example, as those skilled in the art will know, with the evolution of system architecture and the emergence of new business scenarios, the technical solutions provided in the embodiments of this application are also applicable to similar technical problems.

[0038] Based on the above description, embodiments of this application propose a data acquisition method, which can be executed by the aforementioned electronic device. For example... Figure 2 As shown, the data acquisition method of this application embodiment may include the following flow:

[0039] S201. Obtain the structure information of the data table to be scanned; this structure information includes the field names and field types of the data table to be scanned.

[0040] The data table to be scanned can be stored in a database, meaning it can be retrieved from the database and used as the data table to be scanned; alternatively, it can be a data table transmitted from other devices, or a data table stored locally, etc., and this application does not impose any limitations. Optionally, the data table to be scanned can be any specified data table, or it can be multiple data tables with the same characteristic information. This same characteristic information can refer to having the same structural information and being of the same type. For example, multiple data tables of type storing user payment information, and these multiple data tables have the same structural information, that is, the field names and field types of the fields they contain are all the same.

[0041] In one possible implementation, obtaining the structural information of the data table to be scanned can specifically involve developers analyzing the data table to obtain the field names and field types of multiple (two or more) fields that make up the data table. For example, if the data table to be scanned is a data table storing user information, suppose the obtained structural information of the data table to be scanned includes: 3 fields, with field names of [User ID], [User Name], and [User Mobile Number], and the field type of the field named [User ID] is string, the field type of the field named [User Name] is varchar, and the field type of the field named [User Mobile Number] is int.

[0042] Optionally, in addition to obtaining structural information by analyzing the data table to be scanned, the amount of data in the data table to be scanned, i.e. the size of the memory occupied, can also be obtained. When the electronic device obtains the amount of data in the data table to be scanned, it will pre-allocate a hard disk with an appropriate memory capacity to store the scanned data.

[0043] S202. Based on the field types of the data table to be scanned, obtain the configuration files corresponding to the field types from the data scanning library, and generate a data scanning tool based on the field names of the data table to be scanned and the configuration files.

[0044] In one possible implementation, the data scanning library encapsulates multiple configuration files for different field types. These configuration files are configured with protocols that support type conversion of data corresponding to the respective field types. The protocol can be protobuf, also known as Protocol Buffer (PB). Based on the protobuf description language and reflection mechanism, data can be serialized and deserialized.

[0045] Serialization can be used to convert the raw data obtained from the data to be scanned into binary data, and deserialization can be used to convert the binary data into data in a specified format. The specified format data can exist in the form of a data table, that is, the data table obtained after conversion has the same structure as the data table to be scanned before conversion. The protobuf description language and reflection principle mechanism mean that the serialized data can be deserialized again to obtain valid data information.

[0046] Optionally, since protobuf handles serialization and deserialization differently for data corresponding to different field types, multiple PB files for different field types can be pre-defined and generated based on protobuf. These PB files can then be encapsulated into configuration files and stored in a data scanning library. This data scanning library can be deployed on electronic devices or other devices capable of data storage. The electronic device can retrieve the configuration file corresponding to the field type from the data scanning library based on the field type of the data table to be scanned. For example, configuration file 1 can perform serialization and deserialization on field data of type int.

[0047] For example, see Figure 3 , Figure 3This application provides an illustration of an application scenario for generating a data scanning tool. When an electronic device receives the structural information of a data table to be scanned sent by a developer, it retrieves the configuration file corresponding to the field types of the data table to be scanned from the data scanning library, and compiles and generates a data scanning tool based on the field names of the data table to be scanned and the configuration file.

[0048] In one possible implementation, the electronic device can retrieve the corresponding configuration file from a data scanning library based on the field types of the data table to be scanned, and generate a data scanning tool based on the field names of the data table to be scanned and the configuration file. This data scanning tool has the ability to identify the structure of the data table to be scanned, and also has the ability to serialize and deserialize data corresponding to multiple fields in the scanned data based on different field types.

[0049] For example, an electronic device obtains a row of data from a data table to be scanned using a data scanning tool. The tool then identifies the structure of the row, obtaining the field names and corresponding data for each field, as well as the field types. The data is then serialized using the data scanning tool. For instance, if the row has "Field 1" and "Field 2," with "Field 1" having a field type of type 1 and "Field 2" having a field type of type 2, the data corresponding to "Field 1" is serialized sequentially using the data scanning tool, and then the data corresponding to "Field 2" is serialized as well, resulting in a byte sequence that includes the row of data and the types of the included fields.

[0050] S203. Obtain the target data from the data table to be scanned using a data scanning tool.

[0051] The target data can be data serialized using a data scanning tool, or data deserialized using a data scanning tool. Optionally, the target data can be part or all of the data in the data table to be scanned, organized in the form of a data table.

[0052] In one possible implementation, the electronic device can use the data scanning tool to scan a data table at the row level. The data table can include N rows of data, which are of a first type, i.e., data in a table structure. Furthermore, the N rows of data include the i-th row, where N is a positive integer and i is a positive integer less than or equal to N.

[0053] One specific implementation method for an electronic device to obtain target data from a data table using a data scanning tool can be as follows: The data scanning tool scans the i-th row of the data table to be scanned and obtains the data of the i-th row; then, based on a configuration file in the data scanning tool, the data of the i-th row is converted into a second type of data; and the target data is determined based on the second type of data corresponding to N rows. Specifically, converting the data of the i-th row to the second type of data based on the configuration file in the data scanning tool means that the second type of data is the data after serialization of the data of the i-th row. For example, this second type of data can be binary data.

[0054] Optionally, in one possible implementation, determining the target data based on the second type of data corresponding to the N rows can specifically involve directly using the second type of data corresponding to the N rows as the target data. In this case, the electronic device can process the target data, such as directly storing the second type of target data to reduce data space usage, or using the second type of target data for transmission to improve data transmission efficiency.

[0055] Optionally, determining the target data based on the second type of data corresponding to the N rows can also involve deserializing the second type of data corresponding to the N rows to obtain data of a specified type, which can be the first type of data. In this case, the electronic device can process the target data, such as storing or migrating it. The target data can be restored to the form of a data table, which can visually represent the data table to be scanned. Specifically, deserializing the second type of data corresponding to the N rows can be based on a configuration file in the data scanning tool. That is, the data scanning tool can identify which part of the second type of data in the N rows corresponds to a serialized field, and the field type of that field. Based on the second type of data in the N rows, the data scanning tool sequentially performs deserialization processing, and then restores the deserialized data to the form of a data table using the field names defined in the data scanning tool.

[0056] In this embodiment, the electronic device acquires the structural information of the data table to be scanned. This structural information includes the field names and field types of the data table, and, based on the field types, retrieves the corresponding configuration files from a data scanning library. A data scanning tool is then generated based on the field names and configuration files of the data table, and the target data is obtained from the data table using this tool. By implementing the proposed method, a data scanning tool can be generated using the structural information of the data table to be scanned, ensuring that the data obtained through this tool is accurate and usable. This, in turn, helps improve the efficiency and reliability of scanning data tables to obtain data.

[0057] Please participate Figure 4 , Figure 4 This is a flowchart illustrating another data acquisition method provided in an embodiment of this application, which can be executed by the aforementioned electronic device. Figure 4 As shown, the data acquisition method in this embodiment may include the following steps:

[0058] S401. Obtain the structure information of the data table to be scanned; this structure information includes the field names and field types of the data table to be scanned.

[0059] S402. Based on the field types of the data table to be scanned, obtain the configuration files corresponding to the field types from the data scanning library, and generate a data scanning tool based on the field names of the data table to be scanned and the configuration files.

[0060] The specific implementation of steps S401-S402 can be found in the relevant descriptions of steps S201-S202 in the above embodiments, and will not be repeated here.

[0061] S403. Obtain the storage information of the data table to be scanned; the storage information includes the table name of the data table to be scanned and the storage address of the data table to be scanned.

[0062] In one possible implementation, obtaining the storage information of the data table to be scanned can specifically involve the table name that makes up the data table and the storage address of the data table. For example, the table name and storage address can be obtained by developers when analyzing the data table to be scanned; or they can be obtained by querying the management module used to store the storage information of the data tables. Since the database storing the data table to be scanned contains a large number of data tables, the electronic device can quickly find the data table to be scanned from the database using the table name and storage address, and then scan the data table to be scanned.

[0063] Optionally, the storage address of the data table to be scanned may include the identifier of the storage device where the data table is located and the module name of the storage module it belongs to. For example, if the database storing the data table to be scanned is a distributed storage system, since a distributed storage system contains multiple storage devices, and each storage device uses different storage modules to store the data tables, the identifier of the storage device can be used to quickly find the target storage device where the data table to be scanned is located among multiple storage devices, and the module name of the storage module can be used to quickly find the target storage module where the data table to be scanned is located among different storage modules. Therefore, the identifier of the storage device where the data table to be scanned is located and the module name of the storage module it belongs to can be used as the storage address of the data table to be scanned.

[0064] Alternatively, the distributed storage system can specifically be a distributed key-value pair storage system (also known as a distributed key-value database or distributed KV database). Therefore, the data table to be scanned can be a key-value data table, meaning it includes key data and corresponding value data. When retrieving a specific row of data from the data table, the corresponding value data can be found through the key data in that row, and the specified row of data can be obtained based on the key data and the corresponding value data. For example, if the data table to be scanned is a table storing user information, assuming the obtained structure information of the data table includes three fields named [User ID], [User Name], and [User Mobile Number], then the data corresponding to the field named [User ID] can be used as the key, and the data corresponding to the other fields can be used as the value corresponding to the key.

[0065] S404. Based on the table name and storage address of the data table to be scanned, the target data is obtained from the data table to be scanned using a data scanning tool.

[0066] The electronic device can locate the data table to be scanned based on the table name and storage address, and then scan the data table at the row level using a data scanning tool, and obtain the target data based on the scan results.

[0067] In one possible implementation, the data table to be scanned includes N rows of data, including the i-th row, where N is a positive integer and i is a positive integer less than or equal to N. The N rows of data are of a first type. The electronic device scans the data table to be scanned by row level using a data scanning tool, and obtains the target data based on the scanning results. Specifically, the implementation can be as follows: the data scanning tool scans the i-th row of the data table to be scanned. If the i-th row meets the data acquisition conditions, the i-th row of data is obtained, and the i-th row of data is converted into a second type of data based on the data scanning tool. The target data is determined based on the second type of data corresponding to the R rows that meet the data acquisition conditions, where R is a positive integer less than or equal to N.

[0068] Specifically, the data in the i-th row is converted into the second type of data by the data scanning tool, which is to perform serialization processing on the i-th row of data; the target data is determined based on the second type of data corresponding to the R rows that meet the data acquisition conditions. This can be done by directly using the second type of data corresponding to the R rows that meet the data acquisition conditions as the target data, or by using the data after deserializing the second type of data corresponding to the R rows that meet the data acquisition conditions as the target data.

[0069] Optionally, when the data table to be scanned is a key-value pair data table, and the N rows of data in the data table to be scanned include N key data, and the i-th row of data includes the i-th key data, where i is a positive integer, then the data acquisition condition can be that the i-th key data is a specified key data. Therefore, when scanning the i-th row of the data table to be scanned using a data scanning tool, if the i-th row meets the data acquisition condition, the data of the i-th row can be obtained as follows: when scanning the i-th row of the data table to be scanned using a data scanning tool, the i-th key data corresponding to the i-th row is obtained. If the i-th key data corresponding to the i-th row is a specified key data, the value data corresponding to the i-th key data is obtained, and the data of the i-th row is obtained based on the i-th key data and the corresponding value data. By defining the data acquisition condition, the required target data can be obtained quickly, and the efficiency of data acquisition can be accelerated.

[0070] In one possible implementation, the electronic device scans the data table to be scanned by row level using a data scanning tool, and obtains target data based on the scanning results. A further implementation could be: scanning the i-th row of the data table to be scanned using a data scanning tool, obtaining data that meets the data acquisition conditions from the i-th row, converting the data that meets the data acquisition conditions from the i-th row into a second type of data based on the data scanning tool, and determining the target data based on the second type of data corresponding to the data that meets the data acquisition conditions in N rows of data.

[0071] Specifically, the data in the i-th row of data that meets the data acquisition conditions is converted into the second type of data by the data scanning tool, that is, the data in the i-th row of data that meets the data acquisition conditions is serialized; the target data is determined based on the second type of data corresponding to the data in the N-th row of data that meets the data acquisition conditions. This can be done by directly using the second type of data corresponding to the data in the N-th row of data that meets the data acquisition conditions as the target data, or by using the data in the N-th row of data that meets the data acquisition conditions as the target data after deserialization.

[0072] Optionally, the above data acquisition condition can be to obtain the data corresponding to a specified field in the i-th row of data. This specified field can be one or more fields in the data table to be scanned. Specifically, scanning the i-th row of the data table using a data scanning tool and extracting the data that meets the data acquisition condition from the i-th row can be done as follows: Scanning the i-th row of the data table using a data scanning tool to obtain the i-th row of data; identifying the structure of the i-th row of data using the data scanning tool; and extracting the data corresponding to the specified field from the i-th row of data. It can be understood that when the data table to be scanned is a key-value pair data table, obtaining the i-th row of data can specifically involve obtaining the value data corresponding to the i-th key data based on the i-th key data in the i-th row of data, and then obtaining the i-th row of data based on the i-th key data and its corresponding value data.

[0073] Optionally, the data acquisition condition can also be that when the i-th key data corresponding to the i-th row of data is the specified key data, the data corresponding to the specified field in the i-th row of data is acquired. Specifically, acquiring target data from the data table to be scanned using a data scanning tool can be as follows: when scanning the i-th row of the data table to be scanned using the data scanning tool, if the i-th key data corresponding to the i-th row is the specified key data, then the i-th row of data is acquired. The data acquisition tool is then used to extract the data corresponding to the specified field from the i-th row of data. Furthermore, based on the data extraction tool converting the data corresponding to the specified field extracted from the i-th row of data into a second type of data, the target data is determined according to the second type of data corresponding to the data extracted from R rows of data that satisfy the data acquisition condition, where R is a positive integer less than or equal to N.

[0074] Optionally, in one possible implementation, the data table to be scanned is stored on the storage service side (kv server) of a distributed key-value pair storage system. This distributed key-value pair storage system also includes a storage client (kvclient). The storage client is configured with a data retrieval interface for obtaining data from the storage service side. The storage service side can provide data query services, and the electronic device can query and retrieve data by calling the data retrieval interface in the storage client to request the data query services provided by the storage service side. Specifically, the electronic device can retrieve target data from the data table to be scanned using a data scanning tool by: generating a data retrieval command for the data table to be scanned using the data scanning tool, sending the data retrieval command to the storage client, receiving the data from the data table to be scanned sent by the storage client, and determining the target data based on the data from the data table to be scanned sent by the storage client.

[0075] The data retrieval command can be used to instruct the storage client to retrieve data from the table to be scanned on the storage service side according to the corresponding data retrieval interface. In other words, the data retrieval command can instruct the storage client to find the corresponding data retrieval interface and call it to retrieve data from the table to be scanned on the storage service side.

[0076] Furthermore, when generating the data scanning tool, the electronic device also encapsulates a protocol file for the storage client within the tool. This protocol file enables the data scanning tool to generate data retrieval instructions that the storage client can parse and execute. The data retrieval instructions carry the table name and storage address of the data table to be scanned. After receiving the instructions, the storage client retrieves the corresponding data retrieval interface based on the table name and storage address, and then calls this interface to request a data query service from the storage service provider. The storage client can then use the data query service to retrieve value data based on key data and returns the retrieved key and value data to the electronic device. The storage client sends each row of data retrieved from the table to be scanned to the electronic device. Optionally, the storage client can request the data query service from the storage service provider by calling the data retrieval interface via Remote Procedure Call (RPC). A RPC can be understood as one node requesting a service provided by another node; that is, the storage client can request data query services provided by the storage service provider.

[0077] Based on this, if the electronic device receives data acquisition conditions for the data table to be scanned, and if the data acquisition condition is that the i-th key data included in the i-th row of data is a specified key data, the data acquisition instruction generated by the data scanning tool can carry the data acquisition condition that the i-th key data included in the i-th row of data is a specified key data. When acquiring the i-th row of data through the data acquisition interface of the storage client, the value data is queried based on the key data only if the i-th key data included in the i-th row of data is a specified key data, and the acquired i-th row of data is returned; wherein, the i-th row of data includes the i-th key data and the corresponding value data. When the electronic device receives the i-th row of data sent by the storage client through the data scanning tool, it can perform serialization processing on the i-th row of data based on the data scanning tool, and determine the target data based on the serialized data corresponding to the R rows of data sent by the storage client. Here, the R rows of data can be part or all of the data in the data table to be scanned, and R is a positive integer less than or equal to N.

[0078] Optionally, the data acquisition instruction generated by the data scanning tool may not carry the data acquisition condition that the i-th key data included in the i-th row of data is the specified key data. When the i-th row of data returned by the storage client is received, the electronic device uses the data scanning tool to identify whether the i-th key value data included in the received i-th row of data is the specified key data. If it is the specified key data, the i-th row of data is serialized. If it is not the specified key data, the i-th row of data is deleted.

[0079] Optionally, if the data acquisition condition is to retrieve the data corresponding to the specified field in the i-th row of data, the data acquisition instruction generated by the data scanning tool can carry this condition. When retrieving the i-th row of data through the storage client's data acquisition interface, the corresponding value data is queried based on the i-th key data included in the i-th row, and the data corresponding to the specified field is retrieved from the i-th row of data, and the data corresponding to the specified field retrieved from the i-th row of data is returned. Alternatively, the data acquisition instruction generated by the data scanning tool may not carry the condition of retrieving the data corresponding to the specified field in the i-th row of data. Upon receiving the i-th row of data returned by the storage client, the electronic device identifies the data corresponding to the specified field in the received i-th row of data using the data scanning tool, performs serialization processing on the data corresponding to the specified field in the i-th row of data using the data scanning tool, and determines the target data based on the serialized data corresponding to the specified field in the R-th row of data sent by the storage client.

[0080] In one possible implementation, during the scanning of a data table using a data scanning tool to obtain target data, if a data scanning end command is received for the data scanning tool, the electronic device stops the current scanning operation on the data table. The data scanning end command can be generated via a trigger control, such as a button provided by the electronic device; when the button is triggered, the data scanning end command is generated. Alternatively, it can be generated via a command, such as entering the command "kill -9 process_name", where the process_name is the process name corresponding to the current scanning operation on the data table.

[0081] For example, see Figure 5 , Figure 5 The following is a schematic diagram of an application scenario for data acquisition provided in this application. The process includes: S1, the developer analyzes the data table to be scanned in the database to obtain the structure information and storage information of the data table to be scanned; S2, the electronic device receives the structure information, storage information, and data acquisition conditions of the data table to be scanned sent by the developer, and generates a data scanning tool; S3, the electronic device runs the data scanning tool to scan the data table to be scanned to obtain target data (also known as the result set) that meets the data acquisition conditions; S4, the electronic device performs data processing on the target data, such as data migration or data merging.

[0082] In this embodiment, the electronic device acquires the structural information of the data table to be scanned. This structural information includes the field names and field types of the data table to be scanned. Based on the field types of the data table to be scanned, the device retrieves the configuration file corresponding to the field types from the data scanning library. Based on the field names and configuration file of the data table to be scanned, the device generates a data scanning tool and acquires the storage information of the data table to be scanned. This storage information includes the table name and the storage address of the data table to be scanned. Based on the table name and the storage address of the data table to be scanned, the device retrieves the target data from the data table to be scanned using the data scanning tool. By implementing the method proposed above, a data scanning tool can be generated from the structural information of the data table to be scanned. This ensures that the data obtained by the data scanning tool is accurate and usable, and that the required target data can be obtained based on the data acquisition conditions. This helps to improve the efficiency of data acquisition from the data table, as well as the reliability of data acquisition and the efficiency of data processing. For example, in data aggregation scenarios, the data to be aggregated can be obtained accurately and without error, and data aggregation can be performed directly based on the serialized data. Then, the deserialization function of the data scanning tool can be used to quickly and stably restore the aggregated data to the table structure form.

[0083] Please see Figure 6 , Figure 6This is a flowchart illustrating a data acquisition method provided in an embodiment of this application. This method can be executed by the aforementioned electronic device and can be combined with the above embodiments to acquire data from a data table using a data scanning tool. In this embodiment, the aforementioned data table to be scanned may include at least two sub-tables to be scanned. For example... Figure 6 As shown, the data acquisition method in this embodiment may include the following steps:

[0084] S601. Obtain the number of data tables to be scanned and the specified number of threads.

[0085] In one possible implementation, if the data table to be scanned includes at least two sub-tables, meaning the data volume in the data table to be scanned is too large, the data is stored in at least two sub-tables to be scanned, and these at least two sub-tables to be scanned have the same structure. Therefore, scanning the data table to be scanned can be understood as scanning these at least two sub-tables to be scanned. Each of these at least two sub-tables to be scanned has a unique table ID. When the electronic device obtains the table IDs of these at least two sub-tables to be scanned, it can use a data scanning tool to scan them sequentially according to the table IDs in ascending order to obtain the data.

[0086] Optionally, to further improve data acquisition efficiency, when the electronic device obtains the number of data shards to be scanned, their table IDs, and the specified number of threads, multiple threads can be started to achieve concurrent scanning of multiple data shards. The specified number of threads can be set by developers based on experience, or by developers analyzing the database to obtain the key concurrency limit and setting it accordingly. The aforementioned key concurrency limit refers to the limit on the number of concurrent calls to the data query interface corresponding to the key. That is, scanning a data shard in a single thread involves retrieving the corresponding value based on the key of each row sequentially. Specifically, this can be done by calling the data query interface corresponding to the key of each row to retrieve the corresponding value. One data query interface can correspond to one or more keys. Therefore, when scanning multiple data shards in multiple threads, there may be concurrent calls to a data query interface to retrieve the value corresponding to a key. Therefore, determining the specified number of threads based on the key concurrency limit can make the number of threads created more reasonable. For example, if a thread queries the value corresponding to a key once per second on average, that is, calls the data query interface once per second, the concurrency limit for a key means that the number of concurrent calls to the data query interface corresponding to the key is limited to 5 times / key / second. Therefore, the number of threads can be specified as 5.

[0087] S602. Create a specified number of threads and assign at least one data partition to be scanned to each of the specified number of threads.

[0088] In one possible implementation, allocating at least one data table to be scanned to each thread within a specified number of threads can specifically be done by sequentially assigning at least one data table to be scanned to each thread based on the table ID of each data table to be scanned. For example, if there are 10 data tables to be scanned with table IDs from 01 to 10, and the specified number of threads is 3, then the data tables to be scanned with table IDs 01-03 can be assigned to the first thread, the data tables to be scanned with table IDs 04-07 can be assigned to the second thread, and the data tables to be scanned with table IDs 08-10 can be assigned to the third thread.

[0089] S603. In each of the specified number of threads, the target data is obtained from at least one assigned data table to be scanned using a data scanning tool.

[0090] In one possible implementation, when multiple data are obtained from the corresponding assigned data table in each thread, these multiple data need to be summarized. Specifically, the summarization can be based on the table IDs of the data tables assigned to each thread in ascending order.

[0091] For example, see Figure 7 , Figure 7 This application provides a schematic diagram of a parallel scanning data table application scenario. The table IDs of the data sub-tables to be scanned assigned to the first thread are 01-03, the table IDs of the data sub-tables to be scanned assigned to the second thread are 04-07, and the table IDs of the data sub-tables to be scanned assigned to the third thread are 08-10. Therefore, the electronic device can scan the data sub-tables to be scanned with table IDs 01-03 in the first thread, scan the data sub-tables to be scanned with table IDs 04-07 in the second thread, and scan the data sub-tables to be scanned with table IDs 08-10 in the third thread. Let the data obtained by scanning the corresponding assigned data sub-tables to be scanned in the first thread be data 1, the data obtained by scanning the corresponding assigned data sub-tables to be scanned in the second thread be data 2, and the data obtained by scanning the corresponding assigned data sub-tables to be scanned in the third thread be data 3. Therefore, the order of summarizing multiple data is data 1-data 2-data 3.

[0092] In one possible implementation, to ensure the electronic device maintains a reasonable load while scanning the data partitions, load information of the electronic device can be acquired in real time, including CPU utilization. When CPU utilization is greater than or equal to a threshold, at least one thread is paused. This threshold can be set by the developers based on experience. Based on this, the electronic device, in each of a specified number of threads, uses a data scanning tool to obtain target data from at least one assigned data partition. Specifically, in each unpaused thread, the data scanning tool obtains first data from the assigned data partition. After obtaining the first data, the paused thread is restarted, and in the restarted thread, the data scanning tool obtains second data from the assigned data partition. It is understood that the target data includes both the first and second data.

[0093] Optionally, the load information may also include the hard disk utilization rate of the stored data. When the hard disk utilization rate is greater than or equal to the maximum utilization rate, at least one thread can be paused, and the data on the hard disk can be processed, such as deleting or transferring the data (excluding data in the data sub-table to be scanned) to another location, before restarting the at least one thread. The maximum utilization rate can be set by the developers based on experience.

[0094] Further optionally, the load information may include both CPU utilization and disk utilization. When CPU utilization is greater than or equal to a threshold or disk utilization is greater than or equal to the maximum utilization, at least one thread is paused and an alarm is generated.

[0095] Specifically, the load information of the electronic device can be obtained at any time during the scanning process of the data table to be scanned. The specific method of obtaining the load information of the electronic device can be that the electronic device obtains the load information when it detects the load information acquisition operation, or the electronic device can automatically detect its own load, and when it detects that its own load exceeds a reasonable range, it obtains the load information and generates an alarm prompt.

[0096] In this embodiment, if the data table to be scanned includes at least two sub-tables, the electronic device obtains the number of sub-tables and the specified number of threads, creates the specified number of threads, and assigns at least one sub-table to be scanned to each thread. Then, within each thread of the specified number of threads, a data scanning tool is used to obtain target data from the corresponding assigned at least one sub-table. By implementing the method proposed above, multiple sub-tables can be scanned concurrently using multiple threads, improving data acquisition efficiency. Furthermore, during the scanning process, the electronic device can maintain a reasonable load to ensure stable data acquisition.

[0097] Please see Figure 8 , Figure 8 This is a schematic diagram of a data acquisition device provided in this application. It should be noted that... Figure 8 The data acquisition device shown is used to execute this application. Figure 2 , Figure 4 and Figure 6 The methods in the illustrated embodiments are shown only in the parts relevant to the embodiments of this application for ease of explanation; specific technical details are not disclosed. Reference to this application is required. Figure 2 , Figure 4 and Figure 6 The illustrated embodiment. The data acquisition device 800 may include: an acquisition module 801 and a processing module 802. Wherein:

[0098] The acquisition module 801 is used to acquire the structure information of the data table to be scanned; the structure information includes the field names and field types of the data table to be scanned.

[0099] The processing module 802 is used to obtain the configuration file corresponding to the field type from the data scanning library according to the field type of the data table to be scanned, and generate a data scanning tool based on the field name of the data table to be scanned and the configuration file;

[0100] The acquisition module 801 is also used to acquire target data from the data table to be scanned through the data scanning tool.

[0101] In one possible implementation, the acquisition module 801 is further configured to:

[0102] Obtain the storage information of the data table to be scanned; the storage information includes the table name of the data table to be scanned and the storage address of the data table to be scanned;

[0103] When the acquisition module 801 is used to acquire target data from the data table to be scanned using the data scanning tool, it is specifically used for:

[0104] Based on the table name and storage address of the data table to be scanned, the target data is obtained from the data table to be scanned using the data scanning tool.

[0105] In one possible implementation, the data table to be scanned includes N rows of data, the N rows of data including the i-th row of data, where N is a positive integer and i is a positive integer less than or equal to N; the N rows of data are data of a first type.

[0106] When the acquisition module 801 is used to acquire target data from the data table to be scanned using the data scanning tool, it is further used to:

[0107] The data scanning tool scans the i-th row of the data table to be scanned. If the i-th row meets the data acquisition conditions, the data of the i-th row is acquired, and the data of the i-th row is converted into a second type of data based on the data scanning tool.

[0108] The target data is determined based on the second type of data corresponding to the R rows that satisfy the data acquisition conditions; R is a positive integer less than or equal to N.

[0109] In one possible implementation, the N rows of data include N key data, and the i-th row of data includes the i-th key data; the data acquisition condition includes that the i-th key data is specified key data;

[0110] The acquisition module 801, when scanning the i-th row of the data table to be scanned using the data scanning tool, and acquiring the data of the i-th row if the i-th row meets the data acquisition conditions, is specifically used for:

[0111] When the data scanning tool scans the i-th row of the data table to be scanned, if the i-th key data corresponding to the i-th row is the specified key data, then the data of the i-th row is obtained.

[0112] In one possible implementation, the data table to be scanned is stored in a storage service terminal of a storage system, and the storage system further includes a storage client, which is configured with a data acquisition interface for obtaining data from the storage service terminal;

[0113] The acquisition module 801 is used to acquire target data from the data table to be scanned using the data scanning tool, specifically for:

[0114] The data scanning tool generates a data retrieval instruction for the data table to be scanned and sends the data retrieval instruction to the storage client. The data retrieval instruction is used to instruct the storage client to retrieve data from the data table to be scanned on the storage service terminal according to the data retrieval interface corresponding to the data retrieval instruction.

[0115] Receive data from the data table to be scanned sent by the storage client;

[0116] The target data is determined based on the data in the data table to be scanned sent by the storage client.

[0117] In one possible implementation, the data table to be scanned includes at least two sub-tables; the acquisition module 801 is further configured to:

[0118] Obtain the number of data tables to be scanned, and the specified number of threads;

[0119] Create the specified number of threads, and assign at least one of the data sub-tables to be scanned to each of the specified number of threads;

[0120] When the acquisition module 801 is used to acquire target data from the data table to be scanned using the data scanning tool, it is specifically used for:

[0121] In each of the specified number of threads, the target data is obtained from at least one of the corresponding assigned data sub-tables to be scanned using the data scanning tool.

[0122] In one possible implementation, the acquisition module is further configured to:

[0123] Obtain load information; the load information includes the central processing unit (CPU) utilization rate;

[0124] If the CPU utilization rate is greater than or equal to the threshold, at least one of the threads is paused.

[0125] When the acquisition module 801 acquires the target data from at least one of the correspondingly assigned data sub-tables in each of the specified number of threads using the data scanning tool, it is specifically used for:

[0126] In each of the unpaused threads, the data scanning tool is used to obtain the first data from the data table to be scanned assigned to each of the unpaused threads;

[0127] After obtaining the first data, the paused thread is restarted, and in the restarted thread, the data scanning tool is used to obtain the second data from the data table to be scanned assigned to the restarted thread; the target data includes the first data and the second data.

[0128] In this embodiment, the electronic device acquires the structural information of the data table to be scanned. This structural information includes the field names and field types of the data table, and, based on the field types, retrieves the corresponding configuration files from a data scanning library. A data scanning tool is then generated based on the field names and configuration files of the data table, and the target data is obtained from the data table using this tool. By implementing the aforementioned apparatus, a data scanning tool can be generated using the structural information of the data table to be scanned, ensuring that the data obtained through this tool is accurate and usable. This, in turn, helps improve the efficiency and reliability of scanning data tables to obtain data.

[0129] In the various embodiments of this application, the functional modules can be integrated into one processing module, or each module can exist physically separately, or two or more modules can be integrated into one module. The integrated modules described above can be implemented in hardware or as software functional modules, and this application does not impose any limitations on this.

[0130] Please see Figure 9 , Figure 9 This is a schematic diagram of the structure of an electronic device provided in an embodiment of this application. Figure 9 As shown, the electronic device 900 includes at least one processor 901 and a memory 902. Optionally, the electronic device may also include a network interface 903. The processor 901, memory 902, and network interface 903 can exchange data. The network interface 903, controlled by the processor 901, is used to send and receive messages. The memory 902 stores a computer program, which includes program instructions. The processor 901 executes the program instructions stored in the memory 902. The processor 901 is configured to invoke the program instructions to execute the aforementioned method.

[0131] The memory 902 may include volatile memory, such as random-access memory (RAM); the memory 902 may also include non-volatile memory, such as flash memory, solid-state drive (SSD), etc.; the memory 902 may also include a combination of the above types of memory.

[0132] The processor 901 may be a central processing unit (CPU). In one embodiment, the processor 901 may also be a graphics processing unit (GPU). The processor 901 may also be a combination of a CPU and a GPU.

[0133] In one possible implementation, the memory 902 is used to store program instructions. The processor 901 can invoke the program instructions to perform the following steps:

[0134] Obtain the structure information of the data table to be scanned; the structure information includes the field names and field types of the data table to be scanned;

[0135] Based on the field types of the data table to be scanned, the configuration file corresponding to the field type is obtained from the data scanning library, and a data scanning tool is generated based on the field names of the data table to be scanned and the configuration file.

[0136] The target data is obtained from the data table to be scanned using the data scanning tool.

[0137] In one possible implementation, the processor 901 is further configured to:

[0138] Obtain the storage information of the data table to be scanned; the storage information includes the table name of the data table to be scanned and the storage address of the data table to be scanned;

[0139] When the processor 901 is used to obtain target data from the data table to be scanned via the data scanning tool, it is specifically used for:

[0140] Based on the table name and storage address of the data table to be scanned, the target data is obtained from the data table to be scanned using the data scanning tool.

[0141] In one possible implementation, the data table to be scanned includes N rows of data, the N rows of data including the i-th row of data, where N is a positive integer and i is a positive integer less than or equal to N; the N rows of data are data of a first type; when the processor 901 is used to obtain target data from the data table to be scanned through the data scanning tool, it is specifically used for:

[0142] The data scanning tool scans the i-th row of the data table to be scanned. If the i-th row meets the data acquisition conditions, the data of the i-th row is acquired, and the data of the i-th row is converted into a second type of data based on the data scanning tool.

[0143] The target data is determined based on the second type of data corresponding to the R rows that satisfy the data acquisition conditions; R is a positive integer less than or equal to N.

[0144] In one possible implementation, the N rows of data include N key data, and the i-th row of data includes the i-th key data; the data acquisition condition includes that the i-th key data is specified key data; when the processor 901 scans the i-th row of the data table to be scanned using the data scanning tool, and if the i-th row meets the data acquisition condition, then acquires the data of the i-th row, it is specifically used for:

[0145] When the data scanning tool scans the i-th row of the data table to be scanned, if the i-th key data corresponding to the i-th row is the specified key data, then the data of the i-th row is obtained.

[0146] In one possible implementation, the data table to be scanned is stored on a storage service terminal of a storage system, the storage system further including a storage client, the storage client being configured with a data acquisition interface for retrieving data from the storage service terminal; when the processor 901 is used to retrieve target data from the data table to be scanned via the data scanning tool, it is specifically used for:

[0147] The data scanning tool generates a data retrieval instruction for the data table to be scanned and sends the data retrieval instruction to the storage client. The data retrieval instruction is used to instruct the storage client to retrieve data from the data table to be scanned on the storage service terminal according to the data retrieval interface corresponding to the data retrieval instruction.

[0148] Receive data from the data table to be scanned sent by the storage client;

[0149] The target data is determined based on the data in the data table to be scanned sent by the storage client.

[0150] In one possible implementation, the data table to be scanned includes at least two sub-tables; the processor 901 is further configured to:

[0151] Obtain the number of data tables to be scanned, and the specified number of threads;

[0152] Create the specified number of threads, and assign at least one of the data sub-tables to be scanned to each of the specified number of threads;

[0153] When the processor 901 is used to obtain target data from the data table to be scanned via the data scanning tool, it is specifically used for:

[0154] In each of the specified number of threads, the target data is obtained from at least one of the corresponding assigned data sub-tables to be scanned using the data scanning tool.

[0155] In one possible implementation, the processor 901 is further configured to:

[0156] Obtain load information; the load information includes the central processing unit (CPU) utilization rate;

[0157] If the CPU utilization rate is greater than or equal to the threshold, at least one of the threads is paused.

[0158] When the processor 901 is used to obtain the target data from at least one of the corresponding allocated data sub-tables to be scanned in each of the specified number of threads using the data scanning tool, it is also used to obtain first data from the corresponding allocated data sub-tables to be scanned in each of the unpaused threads using the data scanning tool.

[0159] After obtaining the first data, the paused thread is restarted, and in the restarted thread, the data scanning tool is used to obtain the second data from the data table to be scanned assigned to the restarted thread; the target data includes the first data and the second data.

[0160] In specific implementations, the data acquisition device 800, processor 901, memory 902, etc., described in the embodiments of this application can execute the implementation methods described in the above method embodiments, or they can execute the implementation methods described in the embodiments of this application, which will not be repeated here.

[0161] This application also provides a computer-readable storage medium storing a computer program. The computer program includes program instructions, which, when executed by a processor, enable the processor to perform some or all of the steps described in the above method embodiments. Optionally, the computer storage medium can be volatile or non-volatile. The computer-readable storage medium may primarily include a program storage area and a data storage area. The program storage area may store an operating system, at least one application program required for a given function, etc.; the data storage area may store data created based on the use of blockchain nodes, etc.

[0162] This application also provides a computer program product or computer program, which includes program instructions that, when executed by a processor, implement some or all of the steps in the above-described method. Optionally, the program instructions are stored in a computer-readable storage medium, and the processor of a computer device can read the program instructions from the computer-readable storage medium to make the processor executable, causing the computer device to perform some or all of the steps in the above-described method.

[0163] Those skilled in the art will understand that all or part of the processes in the above embodiments can be implemented by a computer program instructing related hardware. The program can be stored in a computer storage medium, which can be a computer-readable storage medium. When executed, the program can include the processes of the embodiments of the above methods. The storage medium can be a magnetic disk, optical disk, read-only memory (ROM), or random access memory (RAM), etc.

[0164] The above-disclosed embodiments are merely some of the embodiments of this application, and should not be construed as limiting the scope of this application. Those skilled in the art can understand that all or part of the processes for implementing the above embodiments, and equivalent changes made in accordance with the claims of this application, still fall within the scope of this application.

Claims

1. A data acquisition method, characterized by, The method includes: Obtain the structure information of the data table to be scanned and the amount of data in the data table to be scanned; the structure information includes the field names and field types of the data table to be scanned; Based on the amount of data in the data table to be scanned, pre-allocate hard disk resources with a memory capacity matching the amount of data. The hard disk resources are used to store the data scanned from the data table to be scanned. Based on the field types of the data table to be scanned, the configuration file corresponding to each field type is retrieved from the data scanning library, and a data scanning tool is compiled based on the field names of the data table to be scanned and the configuration file. The data scanning library encapsulates multiple configuration files for different field types, each configuration file having a protocol that supports type conversion of data corresponding to the corresponding field type. Different configuration files have different data processing methods. The data scanning tool has the ability to identify the structure of the data table to be scanned, and the ability to serialize and deserialize data corresponding to multiple fields in the scanned data based on different field types. The data table to be scanned includes key data and corresponding value data. The process of obtaining target data from the data table to be scanned using the data scanning tool includes: obtaining concurrency limit information of the data query interface corresponding to the key data; determining a specified number of threads based on the concurrency limit information and the call frequency characteristics of a thread calling the data query interface; and, in each of the specified number of threads, using the data scanning tool to obtain target data from at least one data sub-table to be scanned corresponding to each thread. Specifically, scanning the data sub-table to be scanned in a single thread involves sequentially calling the data query interface corresponding to the key data of each row to obtain the corresponding value data based on the key data of each row. One data query interface corresponds to one or more key data sets. Get the hard drive usage rate for storing data; If the hard disk occupancy rate is greater than or equal to the maximum occupancy rate, at least one thread is paused, and the data on the hard disk is processed. The processing includes deleting or relocating the data on the hard disk except for the data in the data sub-table to be scanned, and restarting the at least one thread. If the target data is of the first type, then the target data is stored or migrated, and the target data is restored to the form of a data table. The first type of data is obtained by deserializing the second type of data. If the target data is of the second type, then the target data of the second type will be stored or transmitted.

2. The method of claim 1, wherein, The method further includes: Obtain the storage information of the data table to be scanned; the storage information includes the table name of the data table to be scanned and the storage address of the data table to be scanned; The step of obtaining target data from the data table to be scanned using the data scanning tool includes: Based on the table name and storage address of the data table to be scanned, the target data is obtained from the data table to be scanned using the data scanning tool.

3. The method of claim 1, wherein, The data table to be scanned includes N rows of data, and the N rows of data include the i-th row of data, where N is a positive integer and i is a positive integer less than or equal to N; The N rows of data are of the first type; The step of obtaining target data from the data table to be scanned using the data scanning tool includes: The data scanning tool scans the i-th row of the data table to be scanned. If the i-th row meets the data acquisition conditions, the data of the i-th row is acquired, and the data of the i-th row is converted into a second type of data based on the data scanning tool. The target data is determined based on the second type of data corresponding to the R rows that satisfy the data acquisition conditions; R is a positive integer less than or equal to N.

4. The method according to claim 3, characterized in that, The N rows of data include N key data, and the i-th row of data includes the i-th key data; the data acquisition condition includes that the i-th key data is a specified key data; The step of scanning the i-th row of the data table to be scanned using the data scanning tool, and obtaining the data of the i-th row if the i-th row meets the data acquisition conditions, includes: When the data scanning tool scans the i-th row of the data table to be scanned, if the i-th key data corresponding to the i-th row is the specified key data, then the data of the i-th row is obtained.

5. The method according to claim 1, characterized in that, The data table to be scanned is stored in the storage service terminal of the storage system. The storage system also includes a storage client, which is configured with a data acquisition interface for obtaining data from the storage service terminal.

6. The method according to any one of claims 1-5, characterized in that, The data table to be scanned includes at least two sub-tables; the method further includes: Obtain the number of data tables to be scanned, and the specified number of threads; Create the specified number of threads, and assign at least one of the data sub-tables to be scanned to each of the specified number of threads.

7. The method according to claim 6, characterized in that, The method further includes: Obtain load information; the load information includes the central processing unit (CPU) utilization rate; If the CPU utilization rate is greater than or equal to the threshold, at least one of the threads is paused. The step of obtaining target data from at least one data table to be scanned corresponding to each thread within the specified number of threads, using the data scanning tool, includes: In each of the unpaused threads, the data scanning tool is used to obtain the first data from the data table to be scanned assigned to each of the unpaused threads; After obtaining the first data, the paused thread is restarted, and in the restarted thread, the data scanning tool is used to obtain the second data from the data table to be scanned assigned to the restarted thread; the target data includes the first data and the second data.

8. A data acquisition device, characterized in that, The device includes: The acquisition module is used to acquire the structure information of the data table to be scanned and the amount of data in the data table to be scanned; the structure information includes the field names and field types of the data table to be scanned; The processing module is used to pre-allocate hard disk resources with a memory capacity matching the data volume in the data table to be scanned, and the hard disk resources are used to store the data scanned from the data table to be scanned. The processing module is further configured to retrieve the configuration file corresponding to the field type of the data table to be scanned from the data scanning library, and compile and generate a data scanning tool based on the field names of the data table to be scanned and the configuration file; the data scanning library encapsulates multiple configuration files for different field types, each configuration file is configured with a protocol that supports type conversion of data corresponding to the corresponding field type, and different configuration files have different data processing methods; the data scanning tool has the ability to identify the structure of the data table to be scanned, and has the ability to serialize and deserialize data corresponding to multiple fields in the scanned data based on different field types; the data table to be scanned includes key data and corresponding value data; The acquisition module is further configured to acquire target data from the data table to be scanned using the data scanning tool, including: acquiring concurrency limit information of the data query interface corresponding to the key data; determining a specified number of threads based on the concurrency limit information and the call frequency characteristics of a thread calling the data query interface; and acquiring target data from at least one data sub-table to be scanned corresponding to each thread within the specified number of threads using the data scanning tool; wherein, scanning the data sub-table to be scanned in a thread involves sequentially calling the data query interface corresponding to the key data of each row to acquire the corresponding value data based on the key data of each row; one data query interface corresponds to one or more key data; The acquisition module is also used to acquire the hard disk occupancy rate of the stored data; if the hard disk occupancy rate is greater than or equal to the maximum occupancy rate, at least one thread is paused and the data in the hard disk is processed. The processing includes deleting or relocating the data in the hard disk except for the data in the data sub-table to be scanned, and restarting the at least one thread. The acquisition module is further configured to, if the target data is of the first type, store or migrate the target data and restore the target data to the form of a data table, wherein the first type of data is obtained by deserializing the second type of data; and if the target data is of the second type, store or transmit the second type of target data.

9. An electronic device, characterized in that, The system includes a processor and a memory, wherein the memory is used to store a computer program, the computer program including program instructions, and the processor is configured to invoke the program instructions to perform the method as described in any one of claims 1-7.

10. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores a computer program, the computer program including program instructions, which, when executed by a processor, are used to perform the method as described in any one of claims 1-7.

11. A computer program product, characterized in that, The computer program product includes program instructions that, when executed by a processor, implement the method as described in any one of claims 1-7.

Images