A data acquisition method, device, system, equipment and storage medium

Abstract

The present disclosure relates to a data acquisition method, device, system, equipment and storage medium, and relates to the field of computer technology, which can improve data acquisition efficiency. The data acquisition method includes: when the lag time of acquiring data in a message queue from a server is greater than a preset time threshold, acquiring the location information of an electronic device storing the data in the message queue; storing the data in the message queue in the electronic device through a distributed file system; and acquiring the data to be read in the message queue from the electronic device based on the location information of the electronic device.

Description

Technical Field

[0001] The present disclosure relates to the field of computer technology, and in particular to a data acquisition method, device, system, equipment and storage medium. Background Art

[0002] The message queue is a container for storing messages during the transmission of the message. The message queue is not suitable for retaining data for a long time. Therefore, when receiving a large number of data read requests sent by clients, the reading speed of the message queue in the server may be greatly reduced, and the busyness of the disk drive may increase sharply.

[0003] In order to solve the above problems, the prior art usually stores the message data in the message queue on a distributed file system such as a simple storage service when the disk drive is too busy, so that the storage function and the computing function in the server are processed separately (also known as a "storage-computing separation architecture"). However, distributed file systems such as simple storage services can only provide simple storage services, and the server still needs to provide computing services during the data reading process. As a result, this "storage-computing separation architecture" will still consume the server's bandwidth and increase the server's load, reducing the efficiency of data acquisition. Summary of the invention

[0004] The present disclosure provides a data acquisition method, device, system, equipment and storage medium, which can improve the efficiency of data acquisition.

[0005] The technical solution of the embodiment of the present disclosure is as follows:

[0006] According to a first aspect of an embodiment of the present disclosure, a data acquisition method is provided, which can be applied to a client. The method may include: when the lag time of acquiring data in a message queue from a server is greater than a preset time threshold, acquiring location information of an electronic device storing data in the message queue; storing the data in the message queue in the electronic device through a distributed file system; and acquiring the data to be read in the message queue from the electronic device based on the location information of the electronic device.

[0007] Optionally, based on the location information of the electronic device, the data to be read in the message queue is obtained from the electronic device, including: when the file in the directory of the storage partition in the distributed file system is complete, and the data to be read belongs to the first M segments of data in the message queue, based on the location information of the electronic device, the data to be read in the message queue is obtained from the electronic device; M is an integer greater than 0 and less than N; N is the number of data segments in the message queue.

[0008] Optionally, based on the location information of the electronic device, the data to be read in the message queue is obtained from the electronic device, including: based on the location information of the electronic device, an index file is obtained from the electronic device; the index file is used to read the file in the directory of the storage partition of the data to be read in the distributed file system; the index file includes a starting offset and an offset of the data in the message queue; a first read data request including the starting offset and the offset of the data to be read is sent to the electronic device; and the data to be read sent by the electronic device in response to the first read data request is received.

[0009] Optionally, the data to be read includes a data address check bit for verifying the data integrity of the data to be read; the data address check bit corresponds to the file storage start position and the file storage end position of the data to be read; after receiving the data to be read sent by the electronic device in response to the first read data request, it also includes: based on the file storage start position and the file storage end position, obtaining the data file in the data to be read; according to the data file, verifying the data integrity of the data to be read and the continuity of the offset of the data to be read; when the data to be read is incomplete, or the offset of the data to be read is discontinuous, sending a second read data request for obtaining the data to be read to the server to which the message queue belongs.

[0010] Optionally, the data acquisition method also includes: constructing a log object of the storage partition to which the data to be read belongs; the log object is used to determine whether the files under the directory of the storage partition to which the data to be read belongs are complete; sending a first data verification request including the log object to the electronic device; and receiving data verification information sent by the electronic device in response to the first data verification request.

[0011] Optionally, the data acquisition method further includes: when the data verification information is used to indicate that the files under the directory of the storage partition to which the data to be read belongs are incomplete, sending a second data read request for acquiring the data to be read to the server to which the message queue belongs.

[0012] Optionally, the data acquisition method also includes: sending a third read data request to the electronic device; the third read data request is used to request to obtain other data in the message queue except the data to be read in sequence; the first segment of other data is the next segment of data adjacent to the data to be read; the third read data request includes the offset of the data to be read and the offset of the next segment of data; the third read data request includes a data request identifier for requesting to obtain other data; and receiving the next segment of data sent by the electronic device in response to the third read data request.

[0013] Optionally, before receiving the next segment of data sent by the electronic device in response to the third read data request, the data acquisition method also includes: sending a data prefetch request to the electronic device; the data prefetch request is used to instruct the electronic device to perform a data prefetch operation on the next segment of data; the data prefetch request includes a data prefetch request identifier used to instruct the electronic device to perform a data prefetch operation.

[0014] Optionally, the data acquisition method also includes: when the data to be read is the Jth segment of data to be read in the message queue, sending a second data verification request to the electronic device; the second data verification request is used to determine whether new data to be read is generated in the message queue within a target time period; the target time period is a time period for reading the Jth segment of data to be read; J is an integer greater than 0 and less than N; receiving a data verification result sent by the electronic device in response to the second data verification request; when the data verification result is used to indicate that new data to be read is generated in the message queue within the target time period, sending a fourth read data request to the electronic device for obtaining the Jth segment of data to be read.

[0015] Optionally, the data acquisition method further includes: when an abnormality occurs in acquiring data in a message queue from the electronic device, outputting an alarm message; the alarm message includes log information of the abnormal data.

[0016] Optionally, obtaining location information of an electronic device storing data in a message queue includes: sending a location information acquisition request to a server; the location information acquisition request is used to request obtaining location information of the electronic device storing data in the message queue; and receiving the location information sent by the server.

[0017] According to the second aspect of the embodiment of the present disclosure, a data acquisition method is provided, which can be applied to a server. The method may include: receiving a location information acquisition request sent by a client; the location information acquisition request is used to request the location information of an electronic device storing data in a message queue; the location information acquisition request is sent when the lag time for the client to obtain the data in the message queue from the server is greater than a preset time threshold; the data in the message queue is stored in the electronic device by deploying a distributed file system; sending location information to the client so that the client obtains the data to be read in the message queue that meets the preset conditions from the electronic device based on the location information of the electronic device; the preset conditions include: the files under the directory of the storage partition of the distributed file system for the data to be read are complete, and the data to be read belongs to the first M segments of data in the message queue; M is an integer greater than 0 and less than N; N is the number of data segments in the message queue.

[0018] According to a third aspect of an embodiment of the present disclosure, a data acquisition device is provided, which can be applied to a client. The device may include: an acquisition unit; an acquisition unit configured to acquire location information of an electronic device storing data in a message queue when a delay time for acquiring data in a message queue from a server is greater than a preset time threshold; the data in the message queue is stored in the electronic device through a distributed file system; and the acquisition unit is further configured to acquire data to be read in the message queue from the electronic device based on the location information of the electronic device.

[0019] Optionally, the acquisition unit is specifically used to: when the file in the directory of the storage partition in the distributed file system of the data to be read is complete, and the data to be read belongs to the first M segments of data in the message queue, based on the location information of the electronic device, obtain the data to be read in the message queue from the electronic device; M is an integer greater than 0 and less than N; N is the number of data segments in the message queue.

[0020] Optionally, the acquisition unit is specifically used to: acquire an index file from the electronic device based on the location information of the electronic device; the index file is used to read the file in the directory of the storage partition of the data to be read in the distributed file system; the index file includes a starting offset and an offset of the data in the message queue; send a first read data request including the starting offset and the offset of the data to be read to the electronic device; receive the data to be read sent by the electronic device in response to the first read data request.

[0021] Optionally, the data to be read includes a data address check bit for verifying the data integrity of the data to be read; the data address check bit corresponds to the file storage start position and the file storage end position of the data to be read; the acquisition unit is also used to obtain the data file in the data to be read based on the file storage start position and the file storage end position; the data acquisition device also includes: a processing unit and a sending unit; the processing unit is used to verify the data integrity of the data to be read and the continuity of the offset of the data to be read according to the data file; the sending unit is used to send a second read data request for obtaining the data to be read to the server to which the message queue belongs when the data to be read is incomplete or the offset of the data to be read is discontinuous.

[0022] Optionally, the processing unit is also used to construct a log object of the storage partition to which the data to be read belongs; the log object is used to determine whether the files under the directory of the storage partition to which the data to be read belongs are complete; the sending unit is also used to send a first data verification request including the log object to the electronic device; the data acquisition device also includes a receiving unit; the receiving unit is used to receive data verification information sent by the electronic device in response to the first data verification request.

[0023] Optionally, the sending unit is further configured to send a second data read request for obtaining the data to be read to the server to which the message queue belongs when the data verification information is used to indicate that the file under the directory of the storage partition to which the data to be read belongs is incomplete.

[0024] Optionally, the sending unit is also used to send a third read data request to the electronic device; the third read data request is used to request to obtain other data in the message queue except the data to be read in sequence; the first segment of other data is the next segment of data adjacent to the data to be read; the third read data request includes the offset of the data to be read and the offset of the next segment of data; the third read data request includes a data request identifier for requesting to obtain other data; the receiving unit is also used to receive the next segment of data sent by the electronic device in response to the third read data request.

[0025] Optionally, the sending unit is further used to send a data prefetch request to the electronic device; the data prefetch request is used to instruct the electronic device to perform a data prefetch operation on the next segment of data; the data prefetch request includes a data prefetch request identifier used to instruct the electronic device to perform a data prefetch operation.

[0026] Optionally, the sending unit is also used to send a second data verification request to the electronic device when the data to be read is the Jth segment of data to be read in the message queue; the second data verification request is used to determine whether new data to be read is generated in the message queue within a target time period; the target time period is the time period for reading the Jth segment of data to be read; J is an integer greater than 0 and less than N; the receiving unit is also used to receive a data verification result sent by the electronic device in response to the second data verification request; the sending unit is also used to send a fourth read data request to the electronic device for obtaining the Jth segment of data to be read when the data verification result is used to indicate that new data to be read is generated in the message queue within the target time period.

[0027] Optionally, it also includes: an output unit; the output unit is used to output alarm information when an abnormality occurs in the data in the message queue obtained from the electronic device; the alarm information includes log information of the abnormal data.

[0028] Optionally, the acquisition unit is specifically used to: send a location information acquisition request to a server; the location information acquisition request is used to request to obtain the location information of an electronic device storing data in the message queue; and receive the location information sent by the server.

[0029] According to the fourth aspect of the embodiment of the present disclosure, a data acquisition device is provided, which can be applied to a server. The device may include: a receiving unit and a sending unit; the receiving unit is used to receive a location information acquisition request sent by a client; the location information acquisition request is used to request to obtain the location information of an electronic device storing data in a message queue; the location information acquisition request is sent when the lag time for the client to obtain the data in the message queue from the server is greater than a preset time threshold; the data in the message queue is stored in the electronic device through a distributed file system; the sending unit is used to send location information to the client, so that the client obtains the data to be read in the message queue that meets the preset conditions from the electronic device based on the location information of the electronic device; the preset conditions include: the files under the directory of the storage partition of the distributed file system for the data to be read are complete, and the data to be read belongs to the first M segments of data in the message queue; M is an integer greater than 0 and less than N; N is the number of data segments in the message queue.

[0030] According to a fifth aspect of an embodiment of the present disclosure, a client is provided, which may include: a processor and a memory for storing processor executable instructions; wherein the processor is configured to execute the instructions to implement any one of the optional data acquisition methods in the above-mentioned first aspect.

[0031] According to a sixth aspect of an embodiment of the present disclosure, a server is provided, which may include: a processor and a memory for storing processor executable instructions; wherein the processor is configured to execute the instructions to implement the data acquisition method of the second aspect above.

[0032] According to a seventh aspect of an embodiment of the present disclosure, a data acquisition system is provided, comprising: a client and a server; the client is used to execute any optional data acquisition method in the above-mentioned first aspect; and the server is used to execute the above-mentioned data acquisition method in the second aspect.

[0033] According to an eighth aspect of an embodiment of the present disclosure, a computer-readable storage medium is provided, on which instructions are stored. When the instructions in the computer-readable storage medium are executed by a processor of a client, the client is enabled to execute any one of the optional data acquisition methods in the first aspect above, or, when the instructions in the computer-readable storage medium are executed by a processor of a server, the client is enabled to execute the data acquisition method in the second aspect above.

[0034] According to a ninth aspect of an embodiment of the present disclosure, a computer program product is provided, which includes computer instructions. When the computer instructions are executed on a client, the client executes the data acquisition method as described in any optional implementation of the first aspect, or, when the computer instructions are executed on a server, the server executes the data acquisition method of the second aspect.

[0035] It is to be understood that the foregoing general description and the following detailed description are exemplary and explanatory only and are not restrictive of the present disclosure.

[0036] The technical solution provided by the embodiments of the present disclosure brings at least the following beneficial effects:

[0037] Based on any of the above aspects, in the present disclosure, when the lag time for obtaining data in the message queue from the server is greater than a preset time threshold, the location information of the electronic device storing the data in the message queue can be obtained, and based on the location information of the electronic device, the data to be read in the message queue can be obtained from the electronic device. Since the data in the message queue is stored in the electronic device through a distributed file system, the client can bypass the server and directly obtain the data to be read from the electronic device, thereby reducing the server load and network bandwidth consumption and improving the data acquisition efficiency of the client. BRIEF DESCRIPTION OF THE DRAWINGS

[0038] The drawings herein are incorporated into and constitute a part of the specification, illustrate embodiments consistent with the present disclosure, and together with the description are used to explain the principles of the present disclosure, and do not constitute improper limitations on the present disclosure.

[0039] Figure 1 A structural schematic diagram of a data acquisition system provided by an embodiment of the present disclosure is shown;

[0040] Figure 2 A schematic diagram of a data acquisition method provided by an embodiment of the present disclosure is shown;

[0041] Figure 3 A schematic diagram showing a flow chart of another data acquisition method provided by an embodiment of the present disclosure;

[0042] Figure 4 A schematic diagram showing a flow chart of another data acquisition method provided by an embodiment of the present disclosure;

[0043] Figure 5 A schematic diagram showing a flow chart of another data acquisition method provided by an embodiment of the present disclosure;

[0044] Figure 6 A schematic diagram showing a flow chart of another data acquisition method provided by an embodiment of the present disclosure;

[0045] Figure 7 A schematic diagram showing a flow chart of another data acquisition method provided by an embodiment of the present disclosure;

[0046] Figure 8 A schematic diagram showing a flow chart of another data acquisition method provided by an embodiment of the present disclosure;

[0047] Fig. 9 A schematic diagram showing a flow chart of another data acquisition method provided by an embodiment of the present disclosure;

[0048] Fig.10 A schematic diagram showing a flow chart of another data acquisition method provided by an embodiment of the present disclosure;

[0049] Fig.11 A schematic diagram of the structure of a data acquisition device provided by an embodiment of the present disclosure is shown;

[0050] Fig.12 A structural schematic diagram of another data acquisition device provided by an embodiment of the present disclosure is shown;

[0051] Fig.13 A schematic diagram of the structure of a client provided by an embodiment of the present disclosure is shown;

[0052] Fig.14 A schematic diagram of the structure of a server provided by an embodiment of the present disclosure is shown. DETAILED DESCRIPTION

[0053] In order to enable ordinary persons in the art to better understand the technical solutions of the present disclosure, the technical solutions in the embodiments of the present disclosure will be clearly and completely described below in conjunction with the accompanying drawings.

[0054] It should be noted that the terms "first", "second", etc. in the specification and claims of the present disclosure and the above-mentioned drawings are used to distinguish similar objects, and are not necessarily used to describe a specific order or sequence. It should be understood that the data used in this way can be interchanged where appropriate, so that the embodiments of the present disclosure described herein can be implemented in an order other than those illustrated or described herein. The embodiments described in the following exemplary embodiments do not represent all embodiments consistent with the present disclosure. Instead, they are merely examples of devices and methods consistent with some aspects of the present disclosure as detailed in the appended claims.

[0055] It should also be understood that the term “comprising” indicates the presence of described features, integers, steps, operations, elements and / or components, but does not exclude the presence or addition of one or more other features, integers, steps, operations, elements and / or components.

[0056] The data involved in the present disclosure may be data authorized by the user or fully authorized by all parties.

[0057] As described in the background technology, the prior art usually stores the message data in the message queue on a distributed file system such as a simple storage service when the disk drive is too busy, so that the storage function and the computing function in the server are processed separately (also known as a "storage-computing separation architecture"). However, distributed file systems such as simple storage services can only provide simple storage services, and the server still needs to provide computing services during the data reading process. In this way, this "storage-computing separation architecture" will still consume the server's bandwidth and increase the server's load, reducing the efficiency of data acquisition.

[0058] Based on this, the embodiment of the present disclosure provides a data acquisition method. When the lag time for acquiring data in a message queue from a server is greater than a preset time threshold, the location information of an electronic device storing the data in the message queue can be acquired, and based on the location information of the electronic device, the data to be read in the message queue can be acquired from the electronic device. Since the data in the message queue is stored in the electronic device through a distributed file system, the client can bypass the server and directly acquire the data to be read from the electronic device, thereby reducing the server load and network bandwidth consumption and improving the data acquisition efficiency of the client.

[0059] The following is an exemplary description of the data acquisition method provided in the embodiment of the present disclosure:

[0060] like Figure 1 As shown, the data acquisition method provided by the present disclosure can be applied to a data acquisition system including a client 101, a server 102 and an electronic device 103. The client 101, the server 102 and the electronic device 103 are connected to each other in communication.

[0061] In the embodiment of the present application, the client 101 is used to obtain data from the server 102 or the electronic device 103. A distributed file system, such as Hadoop Distributed File System (HDFS), is deployed on the electronic device 103. The server 102 is one or more servers in the Kafka cluster.

[0062] Alternatively, Kafka is a high-throughput distributed publish-subscribe messaging system that can handle streaming data of all actions of consumers on a website. This action (web browsing, searching and other user actions) is a key factor in many social functions on the modern web. This data is usually solved by processing logs and log aggregation due to throughput requirements. For log data and offline analysis systems like Hadoop, but with the constraints of real-time processing, this is a viable solution. Kafka aims to unify online and offline message processing through Hadoop's parallel loading mechanism, and also to provide real-time messages through clusters.

[0063] The client can be a mobile phone, tablet computer, desktop, laptop, handheld computer, notebook computer, ultra-mobile personal computer (UMPC), netbook, as well as cellular phone, personal digital assistant (PDA), augmented reality (AR) and virtual reality (VR) devices that can install and use content community applications (such as Kuaishou). The present disclosure does not impose any special restrictions on the specific form of the electronic device. It can interact with the user through one or more methods such as keyboard, touchpad, touch screen, remote control, voice interaction or handwriting device.

[0064] In some embodiments, the electronic device may be the server 102, or may be another server independent of the server 102, or may be other electronic devices used for data acquisition, which is not limited in the present disclosure.

[0065] The server may be a single server, or a server cluster consisting of multiple servers. In some implementations, the server cluster may also be a distributed cluster. The present disclosure does not limit the specific implementation of the server.

[0066] The data acquisition method provided in the embodiments of the present application is described in detail below with reference to the accompanying drawings.

[0067] like Figure 2 As shown, when the data acquisition method is applied to the above Figure 1 When the client is in, the data acquisition method may include:

[0068] S201: When the delay time for obtaining data in the message queue from the server is greater than a preset time threshold, the client obtains location information of the electronic device storing the data in the message queue.

[0069] The data in the message queue is stored in the electronic device through a distributed file system, that is, the electronic device stores all the data in the message queue by deploying a distributed file system.

[0070] Optionally, when the distributed file system is deployed on a client, the electronic device may be the client or a functional module in the client.

[0071] In the prior art, a message queue is stored in a server. When a client requests to obtain data in a message queue, the client can obtain the data through the server. However, when a large number of clients send data read requests to the server at the same time, the reading speed of the message queue in the server may be greatly reduced, and the busyness of the disk drive may increase sharply. In this case, the lag time for the client to obtain the data in the message queue will be greatly increased. When the lag time for obtaining the data in the message queue is greater than a preset time threshold, the client obtains the location information of the electronic device that stores all the data in the message queue.

[0072] For example, a server is used to deploy a kafka system (a Kafka cluster includes one or more servers, which are called brokers) and an electronic device is used to deploy an HDFS system. The kafka system itself is a message queue, which is not suitable for retaining data for a long time. At the same time, dragging historical data will pollute the page cache memory, greatly reducing the reading speed and increasing the busyness of the disk drive (utils). In this case, the operation and maintenance personnel first move the data of the kafka system to the HDFS system in the kafka format. Due to the characteristics of the HDFS system, the data on the HDFS system can be retained for a longer time.

[0073] Subsequently, based on the lag of the client (consumer) reading messages from the Kafka broker, it is determined to consume from different storage systems (such as the Kafka system or the HDFS system).

[0074] Optionally, when the lag is greater than a preset time threshold, the consumer obtains the location information of the electronic device where the HDFS system is deployed. Correspondingly, when the lag is less than or equal to the preset time threshold, the consumer obtains the data in the message queue through the broker.

[0075] S202: The client obtains the data to be read in the message queue from the electronic device based on the location information of the electronic device.

[0076] Specifically, after obtaining the location information of the electronic device storing the data in the message queue, the data to be read in the message queue may be obtained from the electronic device based on the location information of the electronic device.

[0077] For example, the consumer can determine to read data from the Kafka system or the HDFS system according to user needs. When it is determined to read data from the HDFS system, the consumer requests the HDFS system metadata information from the broker and extracts the address information of the HDFS system (such as IP address, etc.) from the metadata information. Then, based on the address information of the HDFS system, the consumer obtains the data to be read in the message queue from the electronic device.

[0078] As can be seen above, when the consumer obtains the data to be read, it can switch between the Kafka system and the HDFS system at will. The business does not need to be aware of different backend storage systems. Since the data is retained in the HDFS system for a longer time, the business consumption pressure is greatly reduced, which greatly saves the business's computing resources and the network bandwidth and load of the Kafka broker. It also solves the problem of excessive disk drive busyness (utils) when reading historical data or large-volume data.

[0079] The technical solution provided by the above embodiment brings at least the following beneficial effects: As can be seen from S201-S202, when the lag time for the client to obtain the data in the message queue from the server is greater than the preset time threshold, the location information of the electronic device storing the data in the message queue can be obtained, and based on the location information of the electronic device, the data to be read in the message queue can be obtained from the electronic device. Since the data in the message queue is stored in the electronic device through the distributed file system, the client can bypass the server and directly obtain the data to be read from the electronic device, which reduces the server load and network bandwidth consumption and improves the data acquisition efficiency of the client.

[0080] In one achievable manner, in the above S202, the method in which the client obtains the to-be-read data in the message queue from the electronic device based on the location information of the electronic device specifically includes:

[0081] When the files of the data to be read in the directory of the storage partition in the distributed file system are complete, and the data to be read belongs to the first M segments of data in the message queue, the client obtains the data to be read in the message queue from the electronic device based on the location information of the electronic device. In this way, the client can directly obtain the data to be read from the electronic device according to the characteristics of the distributed file system, which reduces the server load and network bandwidth consumption and improves the data acquisition efficiency of the client.

[0082] Wherein, M is an integer greater than 0 and less than N; N is the number of data segments in the message queue.

[0083] Optionally, M can be equal to N-1, that is, when the files under the directory of the storage partition to which the data to be read belongs are complete and the data to be read is not the last piece of data to be read in the message queue, the client can obtain the data to be read in the message queue from the electronic device based on the location information of the electronic device.

[0084] It should be understood that due to the characteristics of the distributed file system, when the files under the directory of the storage partition to which the data to be read belongs are incomplete, it means that the distributed file system has not yet completed loading the data in the message queue or other abnormalities. In this case, in order to avoid incomplete or abnormal data to be read, the client does not obtain the data to be read in the message queue from the electronic device, but obtains the data to be read from the server in the original way.

[0085] In addition, when the data to be read is the last segment of data to be read in the message queue, the distributed file system needs to reopen the file stream, which will waste a lot of human resources and network resources. Therefore, the client does not obtain the data to be read in the message queue from the electronic device, but obtains the data to be read from the server in the original way.

[0086] In one achievable approach, combining Figure 2 ,like Figure 3 As shown, in the above S202, the method for the client to obtain the data to be read in the message queue from the electronic device based on the location information of the electronic device specifically includes:

[0087] S301. The client obtains an index file from the electronic device based on the location information of the electronic device.

[0088] The index file is used to read the file in the directory of the storage partition of the data to be read in the distributed file system; the index file includes the starting offset and the offset of the data in the message queue.

[0089] Specifically, when the client obtains the to-be-read data in the message queue from the electronic device based on the location information of the electronic device, the client may obtain the index file from the electronic device based on the location information of the electronic device.

[0090] It should be noted that when obtaining index files, the index files can be obtained on demand according to the needs of the user, without obtaining all the index files in the electronic device, so as to avoid a large number of index files seriously consuming the memory of the client.

[0091] In addition, the index file obtained by the client from the electronic device based on the location information of the electronic device is an offset index file, and there is no need to load a time index file, thereby avoiding a large number of index files from seriously consuming the client's memory.

[0092] Combined with the above example, each consumer only needs to load the partition directory it subscribes to. It does not actually read the file during the loading process, but mainly builds metadata information such as the segment map (segmentsMap). When the consumer obtains the index file from the HDFS system based on the location information of the electronic device, it only loads the offset index file, does not need to load the time index file, and loads on demand, and only actually loads the index file corresponding to the segment mapped to the offset passed in during the first read.

[0093] S302: The client sends a first data read request including a start offset and an offset of data to be read to the electronic device.

[0094] Specifically, after obtaining the index file from the electronic device based on the location information of the electronic device, since the index file includes the starting offset and the offset of the data in the message queue, the client sends a first read data request including the starting offset and the offset of the data to be read to the electronic device.

[0095] Optionally, the starting offset is a unique starting offset corresponding to the data in the message queue.

[0096] S303: The client receives the data to be read sent by the electronic device in response to the first data read request.

[0097] Specifically, after the client sends a first data read request including a starting offset and an offset of the data to be read to the electronic device, the electronic device responds to the first data read request, determines a data file in a storage starting position of the data file to be read and a file storage ending position based on the starting offset and the offset of the data to be read, and determines the data file as the data to be read, and sends the data to be read to the client. Correspondingly, the client receives the data to be read sent by the electronic device in response to the first data read request.

[0098] Combined with the above example, after the consumer loads the metadata information, it starts to read data from the HDFS system. Here, the consumer uses the HDFS Fetcher to pull data from the HDFS system. The HDFS Fetcher and the native Fetcher share the SubscriptionState object instance to ensure the continuity of the read offset and facilitate the switching between reading data from the Kafka system and the HDFS system.

[0099] The technical solution provided by the above embodiment brings at least the following beneficial effects: It can be known from S301-S303 that when the client obtains the data to be read in the message queue from the electronic device based on the location information of the electronic device, the index file can be obtained from the electronic device based on the location information of the electronic device. Since the index file includes the starting offset and the offset of all the data in the message queue, the client sends a first read data request including the starting offset and the offset of the data to be read to the electronic device. Subsequently, the client receives the data to be read sent by the electronic device in response to the first read data request. In this way, the client can obtain the data to be read according to the starting offset and the offset of the data to be read in the index file, which provides a method for quickly and accurately obtaining the data to be read. In addition, since the client only needs to obtain the index file to obtain the data to be read, compared with the prior art, there is no need to obtain more index files, while improving the efficiency of data acquisition, it also avoids serious consumption of client memory.

[0100] In one achievable approach, combining Figure 3 ,like Figure 4 As shown, the data to be read includes a data address check bit for checking the data integrity of the data to be read; the data address check bit corresponds to the file storage start position and the file storage end position of the data to be read; after the client receives the data to be read sent by the electronic device in response to the first data read request, it also includes:

[0101] S401: The client obtains a data file in the data to be read based on a storage start position and a file storage end position.

[0102] Specifically, after the client receives the data to be read sent by the electronic device in response to the first read data request, since the data to be read includes a data address check bit for verifying the data integrity of the data to be read; the data address check bit corresponds to the file storage starting position and the file storage ending position of the data to be read, the client can obtain the data file in the data to be read based on the storage starting position and the file storage ending position.

[0103] S402: The client verifies the data integrity of the data to be read and the continuity of the offset of the data to be read according to the data file.

[0104] Specifically, after obtaining the data files in the data to be read based on the storage start position and the file storage end position, the client can verify the data integrity of the data to be read and the continuity of the offset of the data to be read according to all data files to ensure the security of the data to be read.

[0105] S403: When the data to be read is incomplete or the offset of the data to be read is discontinuous, the client sends a second data read request for obtaining the data to be read to the server to which the message queue belongs.

[0106] Specifically, after verifying the data integrity of the data to be read and the continuity of the offset of the data to be read based on all data files, when the data to be read is incomplete or the offset of the data to be read is discontinuous, the client sends a second read data request to the server to which the message queue belongs to obtain the data to be read, so as to ensure that the complete data to be read is obtained from the server.

[0107] For example, if the data to be read includes 100 data files, the client obtains 99 data files of the data to be read from the electronic device. In this case, the client determines that the data of the data to be read is incomplete. Subsequently, in order to obtain the complete target file, the client sends a second read data request for obtaining the data to be read to the server to which the message queue belongs.

[0108] Optionally, the second read data request and the first read data request may be the same read data request or different read data requests, which is not limited in the present disclosure.

[0109] The technical solution provided by the above embodiment brings at least the following beneficial effects: It can be seen from S401-S402 that after the client receives the data to be read sent by the electronic device in response to the first read data request, since the data to be read includes a data address check bit for verifying the data integrity of the data to be read; the data address check bit corresponds to the file storage start position and the file storage end position of the data to be read, therefore, the client can obtain the data file in the data to be read based on the storage start position and the file storage end position. Subsequently, the data integrity of the data to be read and the continuity of the offset of the data to be read can be verified based on all data files to ensure the security of the data to be read. When the data to be read is incomplete, or the offset of the data to be read is discontinuous, the client sends a second read data request for obtaining the data to be read to the server to which the message queue belongs, so that the client obtains the complete data to be read from the server.

[0110] In one possible implementation, Figure 5 As shown, the data acquisition method also includes:

[0111] S501: The client constructs a log object of the storage partition to which the data to be read belongs.

[0112] Specifically, when obtaining the data to be read from the electronic device, the client can construct a log object of the storage partition to which the data to be read belongs, wherein the log object is used to determine whether the files under the directory of the storage partition to which the data to be read belongs are complete.

[0113] S502: The client sends a first data verification request including a log object to the electronic device.

[0114] Specifically, after constructing the log object of the storage partition to which the data to be read belongs, the client sends a first data verification request including the log object to the electronic device, so that the electronic device responds to the first data verification request to determine whether the files under the directory of the storage partition to which the data to be read belongs are complete.

[0115] In combination with the above example, the consumer can create an HDFS log object for each partition to verify whether the files in the partition directory are complete. Then, the consumer sends a first data verification request including the log object to the HDFS system. After receiving the first data verification request, the HDFS system responds to the first data verification request to determine whether the files in the directory of the storage partition to which the data to be read belongs are complete. For example, in the directory of the storage partition to which the data to be read belongs, the index file and the log file must exist at the same time, and there should be no deleted files with the suffix .delete, etc.

[0116] S503: The client receives data verification information sent by the electronic device in response to the first data verification request.

[0117] Specifically, after receiving the first data verification request, the electronic device responds to the first data verification request and determines whether the files under the directory of the storage partition to which the data to be read belongs are complete. Subsequently, the electronic device sends data verification information including the judgment result to the client. Correspondingly, the client receives the data verification information sent by the electronic device in response to the first data verification request.

[0118] The technical solution provided by the above embodiment brings at least the following beneficial effects: It can be seen from S501-S503 that the client can determine whether the files under the directory of the storage partition to which the data to be read belongs are complete by constructing a log object for determining whether the files under the directory of the storage partition to which the data to be read belongs are complete, thereby avoiding obtaining incomplete data to be read and ensuring the integrity of the data to be read.

[0119] In one achievable approach, combining Figure 5 ,like Figure 6 As shown, the data acquisition method also includes:

[0120] S601: When the data verification information indicates that the file under the directory of the storage partition to which the data to be read belongs is incomplete, the client sends a second data read request for obtaining the data to be read to the server to which the message queue belongs.

[0121] Specifically, when the data verification information indicates that the file under the directory of the storage partition to which the data to be read belongs is incomplete, the client sends a second read data request to the server to which the message queue belongs to obtain the data to be read, so as to ensure that the complete data to be read is obtained from the server.

[0122] The technical solution provided by the above embodiment brings at least the following beneficial effects: It can be seen from S601 that when the data verification information is used to indicate that the file under the directory of the storage partition to which the data to be read belongs is incomplete, the client sends a second read data request to the server to which the message queue belongs for obtaining the data to be read, so as to ensure that the client obtains the complete data to be read from the server.

[0123] In one possible implementation, Figure 7 As shown, the data acquisition method also includes:

[0124] S701. The client sends a third data read request to the electronic device.

[0125] Among them, the third read data request is used to request to obtain other data in the message queue except the data to be read in sequence; the first segment of other data is the next segment of data adjacent to the data to be read; the third read data request includes the offset of the data to be read and the offset of the next segment of data; the third read data request includes a data request identifier for requesting to obtain other data.

[0126] Specifically, the message queue may include multiple segments of data. After obtaining the data to be read, the client may also need to obtain other data in the message queue. In this case, the client sends a third data read request to the electronic device.

[0127] The third data read request is used to request to obtain the next segment of data adjacent to the data to be read in the message queue. The third data read request includes the offset of the data to be read and the offset of the next segment of data.

[0128] It should be understood that, since the message queue corresponding to the index file obtained in advance by the client, the client can obtain the offset of the data to be read and the offset of the next segment of data from the index file. In this case, the client can add the offset of the data to be read and the offset of the next segment of data to the third read data request, so that the electronic device can quickly obtain the next segment of data according to the offset of the data to be read and the offset of the next segment of data.

[0129] In addition, when the client obtains multiple segments of data from multiple partitions, a thread pool can be used to achieve a certain degree of parallelism. However, the number of thread pools should not be too large to avoid excessive operating pressure on the client.

[0130] Combined with the above example, the consumer goes to the corresponding HDFS file object according to the incoming partition. The first read locates the log file position corresponding to the offset according to the offset index file. Subsequent reads directly record the file position, which is accumulated sequentially each time without the need to use the index file for positioning.

[0131] S702: The client receives the next segment of data sent by the electronic device in response to the third data read request.

[0132] Specifically, after receiving the third data read request, the electronic device can respond to the third data read request, find the next segment of data from the storage space based on the offset of the data to be read and the offset of the next segment of data, and send the next segment of data to the client. Correspondingly, the client receives the next segment of data sent by the electronic device in response to the third data read request.

[0133] The technical solution provided by the above embodiment brings at least the following beneficial effects: As can be seen from S701-S702, when the client obtains the next segment of data to be read, it can obtain the offset of the data to be read and the offset of the next segment of data from the index file. In this case, the client can add the offset of the data to be read and the offset of the next segment of data to the third read data request, so that the electronic device can quickly obtain the next segment of data based on the offset of the data to be read and the offset of the next segment of data. In this way, the client can obtain other data in the message queue based on the offset of each segment of data without reloading the index file, thereby improving the efficiency of data acquisition.

[0134] In one achievable approach, combining Figure 7 ,like Figure 8 As shown, before receiving the next segment of data sent by the electronic device in response to the third data read request, the data acquisition method further includes:

[0135] S801. The client sends a data pre-fetch request to the electronic device.

[0136] The data pre-fetch request is used to instruct the electronic device to perform a data pre-fetch operation on the next segment of data; the data pre-fetch request includes a data pre-fetch request identifier used to instruct the electronic device to perform the data pre-fetch operation.

[0137] Specifically, when the client obtains the data to be read, it can also send a data pre-fetching request to the electronic device, so that the electronic device performs a data pre-fetching operation on the next segment of data. Subsequently, when the client sends a read data request to the electronic device for obtaining the next segment of data, the electronic device can quickly return the next segment of data to the client, thereby improving the efficiency of data acquisition.

[0138] In combination with the above example, the mode of consumers pulling data from the HDFS system can refer to the pipeline mode (pipeline) composed of native consumer modes to avoid user thread blocking as much as possible.

[0139] Specifically, the consumer can convert the offset into the starting position of the file and locate the end position according to the maximum number of messages pulled each time. Each read is pre-read to form a pipeline to avoid waiting until the data is needed to read, thereby improving the efficiency of data acquisition.

[0140] The technical solution provided by the above embodiment brings at least the following beneficial effects: As can be seen from S801, when the client obtains the data to be read, it can also send a data pre-fetching request to the electronic device, so that the electronic device performs a data pre-fetching operation on the next segment of data. Subsequently, when the client sends a read data request to the electronic device for obtaining the next segment of data, the electronic device can quickly return the next segment of data to the client, thereby improving the efficiency of data acquisition by the client.

[0141] In one possible implementation, Fig. 9 As shown, the data acquisition method also includes:

[0142] S901. When the data to be read is the Jth segment of data to be read in the message queue, the client sends a second data verification request to the electronic device.

[0143] The second data verification request is used to determine whether new data to be read is generated in the message queue within a target time period; the target time period is a time period for reading the Jth segment of data to be read; and J is an integer greater than 0 and less than N.

[0144] Optionally, the Jth segment of data to be read may be the last segment of data to be read in the message queue.

[0145] Specifically, when the data to be read is the last segment of data to be read in the message queue, the distributed file system needs to reopen the file stream, which will waste a lot of human resources and network resources. Therefore, the client does not obtain the data to be read in the message queue from the electronic device, but obtains the data to be read from the server in the original way. However, in the process of reading the last segment of data to be read, the electronic device may generate new data to be read. In this case, the client sends a second data verification request to the electronic device.

[0146] S902: The client receives a data verification result sent by the electronic device in response to the second data verification request.

[0147] S903: When the data verification result indicates that new data to be read is generated in the message queue within the target time period, the client sends a fourth data read request for obtaining the Jth segment of data to be read to the electronic device.

[0148] In combination with the above example, when the Jth segment of data to be read is the last segment of data to be read in the message queue, the electronic device determines whether new data to be read is generated in the process of reading the last segment of data to be read after receiving the second data verification request. When new data to be read is generated in the process of reading the last segment of data to be read, the current last segment of data to be read is no longer the last segment of data to be read. In this case, the client sends a fourth read data request to the electronic device for obtaining the last segment of data to be read.

[0149] The technical solution provided by the above embodiment brings at least the following beneficial effects: It can be seen from S901-S903 that after receiving the second data verification request, the electronic device determines whether new data to be read is generated in the process of reading the Jth segment of data to be read. When new data to be read is generated in the process of reading the Jth segment of data to be read, the current Jth segment of data to be read is no longer the last segment of data to be read. In this case, the client sends a fourth read data request to the electronic device for obtaining the Jth segment of data to be read. In this way, the client can bypass the server and directly obtain the data in the message queue from the electronic device, which reduces the data processing pressure of the server and improves the data acquisition efficiency of the client.

[0150] In one achievable approach, combining Figure 2 ,like Fig.10 As shown, the data acquisition method also includes:

[0151] S1001. When an abnormality occurs in obtaining data in a message queue from an electronic device, the client outputs an alarm message.

[0152] The alarm information includes log information of abnormal data.

[0153] In the above example, when an exception occurs when the consumer reads data from the HDFS system, the consumer will re-read the data. The number of times the data is re-read can be set manually. When the number of re-reads meets the preset number of reads, and an exception still occurs when reading the data, an alarm message is output to prompt the operation and maintenance personnel that a failure has occurred. At the same time, the consumer has a point monitoring for each exception, and the log information of the exception is added to the alarm information.

[0154] The technical solution provided by the above embodiment brings at least the following beneficial effects: As can be seen from S1001, the client can monitor whether the process of obtaining the data to be read is abnormal, and issue an alarm in time when an abnormality occurs, thereby improving the accuracy of obtaining the data to be read.

[0155] In one possible implementation, the above-mentioned acquisition of the location information of the electronic device storing the data in the message queue includes: sending a location information acquisition request to a server; the location information acquisition request is used to request the acquisition of the location information of the electronic device storing the data in the message queue; and receiving the location information sent by the server. In this way, the client can obtain the location information of the electronic device from the server, so that the data to be read in the electronic device can be obtained according to the location information later, thereby improving the efficiency of data acquisition.

[0156] In one achievable manner, the data acquisition method provided by the present disclosure may also be applied to a server. When the data acquisition method may also be applied to a server, it includes:

[0157] Receive the location information acquisition request sent by the client.

[0158] Among them, the location information acquisition request is used to request the location information of the electronic device storing the data in the message queue; the location information acquisition request is sent when the delay time for the client to obtain the data in the message queue from the server is greater than the preset time threshold; the data in the message queue is stored in the electronic device by deploying a distributed file system.

[0159] The location information is sent to the client, so that the client obtains the to-be-read data that meets the preset conditions in the message queue from the electronic device based on the location information of the electronic device.

[0160] Among them, the preset conditions include: the file to be read in the directory of the storage partition in the distributed file system is complete, and the data to be read belongs to the first M segments of data in the message queue; M is an integer greater than 0 and less than N; N is the number of data segments in the message queue.

[0161] It is understandable that, in actual implementation, the client / server described in the embodiment of the present disclosure may include one or more hardware structures and / or software modules for implementing the aforementioned corresponding data acquisition method, and these execution hardware structures and / or software modules may constitute an electronic device. It should be easily appreciated by those skilled in the art that, in combination with the algorithm steps of each example described in the embodiment disclosed herein, the present disclosure can be implemented in the form of hardware or a combination of hardware and computer software. Whether a function is executed in the form of hardware or computer software driving hardware depends on the specific application and design constraints of the technical solution. Professional and technical personnel can use different methods to implement the described functions for each specific application, but such implementation should not be considered to be beyond the scope of the present disclosure.

[0162] Based on such understanding, the embodiments of the present disclosure also provide a data acquisition device that can be applied to a client. Fig.11 FIG. 1 is a schematic diagram showing the structure of a data acquisition device provided by an embodiment of the present disclosure. Fig.11 As shown, the data acquisition device may include: an acquisition unit 1101;

[0163] The acquisition unit 1101 is used to acquire the location information of the electronic device storing the data in the message queue when the delay time of acquiring the data in the message queue from the server is greater than a preset time threshold; the data in the message queue is stored in the electronic device through a distributed file system;

[0164] The acquisition unit 1101 is further configured to acquire the to-be-read data in the message queue from the electronic device based on the location information of the electronic device.

[0165] Optionally, the acquiring unit 1101 is specifically configured to:

[0166] When the file to be read in the directory of the storage partition in the distributed file system is complete, and the data to be read belongs to the first M segments of data in the message queue, the data to be read in the message queue is obtained from the electronic device based on the location information of the electronic device; M is an integer greater than 0 and less than N; N is the number of data segments in the message queue.

[0167] Optionally, the acquiring unit 1101 is specifically configured to:

[0168] Based on the location information of the electronic device, an index file is obtained from the electronic device; the index file is used to read a file under a directory of a storage partition of the data to be read in the distributed file system; the index file includes a starting offset and an offset of the data in the message queue;

[0169] Sending a first data read request including a start offset and an offset of data to be read to the electronic device;

[0170] The receiving electronic device sends data to be read in response to the first data read request.

[0171] Optionally, the data to be read includes a data address check bit for checking the data integrity of the data to be read; the data address check bit corresponds to the file storage start position and the file storage end position of the data to be read;

[0172] The acquisition unit 1101 is further used to acquire the data file in the data to be read based on the file storage start position and the file storage end position;

[0173] The data acquisition device further includes: a processing unit 1102 and a sending unit 1103;

[0174] The processing unit 1102 is used to verify the data integrity of the data to be read and the continuity of the offset of the data to be read according to the data file;

[0175] The sending unit 1103 is configured to send a second data read request for acquiring the data to be read to the server to which the message queue belongs when the data to be read is incomplete or the offset of the data to be read is discontinuous.

[0176] Optionally, the processing unit 1102 is further used to construct a log object of the storage partition to which the data to be read belongs; the log object is used to determine whether the files under the directory of the storage partition to which the data to be read belongs are complete;

[0177] The sending unit 1103 is further configured to send a first data verification request including a log object to the electronic device;

[0178] The data acquisition device further includes a receiving unit 1104;

[0179] The receiving unit 1104 is configured to receive data verification information sent by the electronic device in response to the first data verification request.

[0180] Optionally, the sending unit 1103 is further configured to send a second data read request for obtaining the data to be read to the server to which the message queue belongs when the data verification information is used to indicate that the file under the directory of the storage partition to which the data to be read belongs is incomplete.

[0181] Optionally, the sending unit 1103 is further used to send a third data read request to the electronic device; the third data read request is used to request to obtain other data in the message queue except the data to be read in sequence; the first segment of other data is the next segment of data adjacent to the data to be read; the third data read request includes an offset of the data to be read and an offset of the next segment of data; the third data read request includes a data request identifier for requesting to obtain other data;

[0182] The receiving unit 1104 is further configured to receive a next segment of data sent by the electronic device in response to the third data read request.

[0183] Optionally, the sending unit 1103 is further used to send a data prefetch request to the electronic device; the data prefetch request is used to instruct the electronic device to perform a data prefetch operation on the next segment of data; the data prefetch request includes a data prefetch request identifier used to instruct the electronic device to perform a data prefetch operation.

[0184] Optionally, the sending unit 1103 is further configured to send a second data verification request to the electronic device when the data to be read is the Jth segment of data to be read in the message queue; the second data verification request is used to determine whether new data to be read is generated in the message queue within a target time period; the target time period is a time period for reading the Jth segment of data to be read; J is an integer greater than 0 and less than N;

[0185] The receiving unit 1104 is further configured to receive a data verification result sent by the electronic device in response to the second data verification request;

[0186] The sending unit 1103 is further configured to send a fourth data read request for acquiring the Jth segment of data to be read to the electronic device when the data verification result indicates that new data to be read is generated in the message queue within the target time period.

[0187] Optionally, it further includes: an output unit 1105;

[0188] The output unit 1105 is used to output alarm information when an abnormality occurs in the data in the message queue obtained from the electronic device; the alarm information includes log information of the abnormal data.

[0189] Optionally, the acquiring unit 1101 is specifically configured to:

[0190] Sending a location information acquisition request to the server; the location information acquisition request is used to request to obtain the location information of the electronic device storing the data in the message queue;

[0191] Receive location information sent by the server.

[0192] The embodiment of the present disclosure also provides a data acquisition device, which can be applied to a server. Fig.12 FIG. 1 is a schematic diagram showing the structure of a data acquisition device provided by an embodiment of the present disclosure. Fig.12 As shown, the data acquisition device may include: a receiving unit 1201 and a sending unit 1202;

[0193] The receiving unit 1201 is used to receive a location information acquisition request sent by a client; the location information acquisition request is used to request to obtain the location information of the electronic device storing the data in the message queue; the location information acquisition request is sent when the delay time for the client to obtain the data in the message queue from the server is greater than a preset time threshold; the data in the message queue is stored in the electronic device through a distributed file system;

[0194] The sending unit 1202 is used to send location information to the client so that the client obtains the to-be-read data that meets the preset conditions in the message queue from the electronic device based on the location information of the electronic device; the preset conditions include: the file under the directory of the storage partition in the distributed file system of the to-be-read data is complete, and the to-be-read data belongs to the first M segments of data in the message queue; M is an integer greater than 0 and less than N; N is the number of data segments in the message queue.

[0195] As described above, the embodiment of the present disclosure can divide the functional modules of the client according to the above method example. Among them, the above integrated modules can be implemented in the form of hardware or in the form of software functional modules. In addition, it should be noted that the division of modules in the embodiment of the present disclosure is schematic and is only a logical function division. There may be other division methods in actual implementation. For example, each functional module can be divided corresponding to each function, or two or more functions can be integrated into one processing module.

[0196] Regarding the data acquisition device in the above embodiment, the specific manner in which each module performs operations and the beneficial effects thereof have been described in detail in the above method embodiment and will not be repeated here.

[0197] The disclosed embodiment also provides a client, which may be a user client such as a mobile phone or a computer. Fig.13 The schematic diagram of the structure of the client provided by the embodiment of the present disclosure is shown. The client may be a data acquisition device and may include at least one processor 61 , a communication bus 62 , a memory 63 and at least one communication interface 64 .

[0198] The processor 61 may be a processor (central processing unit, CPU), a microprocessor unit, an ASIC, or one or more integrated circuits for controlling the execution of the program of the disclosed solution. Fig.11 The functions implemented by the processing unit 1102 in the client are similar to Fig.13 The functions implemented by the processor 61 in are the same.

[0199] The communication bus 62 may include a pathway for transmitting information between the above-mentioned components.

[0200] The communication interface 64 uses any transceiver-like device for communicating with other devices or communication networks, such as servers, Ethernet, radio access networks (RAN), wireless local area networks (WLAN), etc. As an example,

[0201] The memory 63 may be a read-only memory (ROM) or other types of static storage devices that can store static information and instructions, a random access memory (RAM) or other types of dynamic storage devices that can store information and instructions, or an electrically erasable programmable read-only memory (EEPROM), a compact disc read-only memory (CD-ROM) or other optical disc storage, optical disc storage (including compressed optical disc, laser disc, optical disc, digital versatile disc, Blu-ray disc, etc.), a magnetic disk storage medium or other magnetic storage device, or any other medium that can be used to carry or store the desired program code in the form of instructions or data structures and can be accessed by a computer, but is not limited thereto. The memory may exist independently and be connected to the processing unit through a bus. The memory may also be integrated with the processing unit.

[0202] The memory 63 is used to store application code for executing the solution of the present disclosure, and the execution is controlled by the processor 61. The processor 61 is used to execute the application code stored in the memory 63, so as to realize the functions in the method of the present disclosure.

[0203] In a specific implementation, as an embodiment, the processor 61 may include one or more CPUs, such as Fig.13 CPU0 and CPU1 in.

[0204] In a specific implementation, as an embodiment, the client may include multiple processors, such as Fig.13 61 and 65 in the embodiment of the present invention. Each of these processors may be a single-CPU processor or a multi-CPU processor. The processor herein may refer to one or more devices, circuits, and / or processing cores for processing data (e.g., computer program instructions).

[0205] In a specific implementation, as an embodiment, the client may further include an input device 66 and an output device 67. The input device 66 communicates with the output device 67 and may receive user input in a variety of ways. For example, the input device 66 may be a mouse, a keyboard, a touch screen device, or a sensor device. The output device 67 communicates with the processor 61 and may display information in a variety of ways. For example, the output device 61 may be a liquid crystal display (LCD), a light emitting diode (LED) display device, or the like.

[0206] Those skilled in the art will understand that Fig.13 The structure shown in the figure does not constitute a limitation on the client, and may include more or less components than those shown in the figure, or combine certain components, or adopt a different component arrangement.

[0207] The embodiment of the present disclosure also provides a server. Fig.14 The schematic diagram of the structure of the server provided by the embodiment of the present disclosure is shown. The server may be a data acquisition device. The server may have relatively large differences due to different configurations or performances, and may include one or more processors 71 and one or more memories 72. Among them, at least one instruction is stored in the memory 72, and at least one instruction is loaded and executed by the processor 71 to implement the data acquisition method provided by the above-mentioned various method embodiments. Of course, the server may also have components such as a wired or wireless network interface, a keyboard, and an input and output interface for input and output. The server may also include other components for implementing device functions, which will not be repeated here.

[0208] The present disclosure also provides a data acquisition system, which includes a client and a server. The client is used to execute the above Figure 2-Figure 10 The data acquisition method executed by the client. The server is used to execute the data acquisition method executed by the above server.

[0209] The present disclosure also provides a computer-readable storage medium including instructions, and the computer-readable storage medium stores instructions. When the instructions in the computer-readable storage medium are executed by the processor of a computer device, the computer can execute the data acquisition method provided by the above-mentioned embodiment. For example, the computer-readable storage medium can be a memory 63 including instructions, and the above instructions can be executed by the processor 61 of the client to complete the above method. For another example, the computer-readable storage medium can be a memory 72 including instructions, and the above instructions can be executed by the processor 71 of the server to complete the above method. Optionally, the computer-readable storage medium can be a non-temporary computer-readable storage medium, for example, the non-temporary computer-readable storage medium can be a ROM, RAM, CD-ROM, magnetic tape, floppy disk, optical data storage device, etc.

[0210] The present disclosure also provides a computer program product, which includes computer instructions. When the computer instructions are executed on an electronic device, the electronic device executes the above Figure 1-Figure 10 The data acquisition method shown in any one of the accompanying drawings.

[0211] Those skilled in the art will readily appreciate other embodiments of the present disclosure after considering the specification and practicing the invention disclosed herein. The present disclosure is intended to cover any variations, uses or adaptations of the present disclosure that follow the general principles of the present disclosure and include common knowledge or customary techniques in the art that are not disclosed in the present disclosure. The description and examples are to be considered exemplary only, and the true scope and spirit of the present disclosure are indicated by the following claims.

[0212] It should be understood that the present disclosure is not limited to the exact structures that have been described above and shown in the drawings, and that various modifications and changes may be made without departing from the scope thereof. The scope of the present disclosure is limited only by the appended claims.

Claims

1. A data acquisition method, characterized in that: include: When the delay time of obtaining data in the message queue from the server is greater than a preset time threshold, sending a location information acquisition request to the server; The location information acquisition request is used to request to acquire the location information of the electronic device storing the data in the message queue; the data in the message queue is stored in the electronic device through a distributed file system; receiving the location information sent by the server; Based on the location information of the electronic device, an index file is obtained from the electronic device; the index file is used to read a file under a directory of a storage partition of the data to be read in the distributed file system; the index file includes a starting offset and an offset of the data in the message queue; Sending a first data read request including the starting offset and the offset of the data to be read to the electronic device; The to-be-read data sent by the electronic device in response to the first read data request is received.

2. The data acquisition method according to claim 1, characterized in that: The data to be read meets the following preset conditions: The file of the data to be read under the directory of the storage partition in the distributed file system is complete, and the data to be read belongs to the first M segments of data in the message queue; M is an integer greater than 0 and less than N; N is the number of data segments in the message queue.

3. The data acquisition method according to claim 2, characterized in that: The data to be read includes a data address check bit for checking the data integrity of the data to be read; the data address check bit corresponds to the file storage start position and the file storage end position of the data to be read; After receiving the to-be-read data sent by the electronic device in response to the first data read request, the method further includes: Based on the file storage starting position and the file storage ending position, obtaining a data file in the data to be read; According to the data file, verifying the data integrity of the data to be read and the continuity of the offset of the data to be read; When the data to be read is incomplete, or the offset of the data to be read is discontinuous, a second data read request for acquiring the data to be read is sent to the server to which the message queue belongs.

4. The data acquisition method according to claim 1, characterized in that: Also includes: Constructing a log object of the storage partition to which the data to be read belongs; the log object is used to determine whether the files under the directory of the storage partition to which the data to be read belongs are complete; Sending a first data verification request including the log object to the electronic device; Receive data verification information sent by the electronic device in response to the first data verification request.

5. The data acquisition method according to claim 4, characterized in that: Also includes: When the data verification information is used to indicate that the file under the directory of the storage partition to which the data to be read belongs is incomplete, a second data read request for acquiring the data to be read is sent to the server to which the message queue belongs.

6. The data acquisition method according to claim 1, characterized in that: Also includes: Sending a third data read request to the electronic device; the third data read request is used to request to sequentially obtain other data in the message queue except the data to be read; the first segment of the other data is the next segment of data adjacent to the data to be read; the third data read request includes the offset of the data to be read and the offset of the next segment of data; the third data read request includes a data request identifier for requesting to obtain the other data; The next segment of data sent by the electronic device in response to the third data read request is received.

7. The data acquisition method according to claim 6, characterized in that: Before receiving the next segment of data sent by the electronic device in response to the third data read request, the method further includes: A data pre-fetch request is sent to the electronic device; the data pre-fetch request is used to instruct the electronic device to perform a data pre-fetch operation on the next segment of data; the data pre-fetch request includes a data pre-fetch request identifier used to instruct the electronic device to perform the data pre-fetch operation.

8. The data acquisition method according to claim 1, characterized in that: Also includes: When the data to be read is the Jth segment of data to be read in the message queue, sending a second data verification request to the electronic device; The second data verification request is used to determine whether new data to be read is generated in the message queue within a target time period; the target time period is a time period for reading the Jth segment of data to be read; J is an integer greater than 0 and less than N; N is the number of data segments in the message queue; receiving a data verification result sent by the electronic device in response to the second data verification request; When the data verification result is used to indicate that the new data to be read is generated in the message queue within the target time period, a fourth read data request for obtaining the Jth segment of data to be read is sent to the electronic device.

9. The data acquisition method according to any one of claims 1 to 8, characterized in that: Also includes: When an abnormality occurs in obtaining data in the message queue from the electronic device, outputting an alarm message; The alarm information includes log information of abnormal data.

10. A data acquisition method, characterized in that: include: Receive the location information acquisition request sent by the client; The location information acquisition request is used to request to acquire the location information of the electronic device storing the data in the message queue; The location information acquisition request is sent when the delay time for the client to obtain the data in the message queue from the server is greater than a preset time threshold; the data in the message queue is stored in the electronic device by deploying a distributed file system; The location information is sent to the client so that the client obtains an index file from the electronic device based on the location information of the electronic device; the index file is used to read a file under a directory of a storage partition of the distributed file system containing data to be read; the index file includes a starting offset and an offset of data in the message queue; and the client sends a first read data request including the starting offset and the offset of the data to be read to the electronic device, and receives the data to be read sent by the electronic device in response to the first read data request.

11. The data acquisition method according to claim 10, characterized in that: The data to be read meets the following preset conditions: the file of the data to be read under the directory of the storage partition in the distributed file system is complete, and the data to be read belongs to the first M segments of data in the message queue; M is an integer greater than 0 and less than N; N is the number of data segments in the message queue.

12. A data acquisition device, characterized in that: include: Get unit; The acquisition unit is used to send a location information acquisition request to the server when the delay time of acquiring the data in the message queue from the server is greater than a preset time threshold; the location information acquisition request is used to request to acquire the location information of the electronic device storing the data in the message queue; the data in the message queue is stored in the electronic device through a distributed file system; The acquisition unit is further configured to receive the location information sent by the server; The acquisition unit is further used to acquire an index file from the electronic device based on the location information of the electronic device; the index file is used to read a file under a directory of a storage partition of the data to be read in the distributed file system; the index file includes a starting offset and an offset of the data in the message queue; The acquisition unit is further used to send a first data read request including the starting offset and the offset of the data to be read to the electronic device; The acquisition unit is further configured to receive the to-be-read data sent by the electronic device in response to the first data read request.

13. The data acquisition device according to claim 12, characterized in that: The data to be read meets the following preset conditions: The file of the data to be read under the directory of the storage partition in the distributed file system is complete, and the data to be read belongs to the first M segments of data in the message queue; M is an integer greater than 0 and less than N; N is the number of data segments in the message queue.

14. The data acquisition device according to claim 13, characterized in that: The data to be read includes a data address check bit for checking the data integrity of the data to be read; the data address check bit corresponds to the file storage start position and the file storage end position of the data to be read; The acquisition unit is further used to acquire the data file in the data to be read based on the file storage starting position and the file storage ending position; The data acquisition device further includes: a processing unit and a sending unit; The processing unit is used to verify the data integrity of the data to be read and the continuity of the offset of the data to be read according to the data file; The sending unit is used to send a second data read request for acquiring the data to be read to the server to which the message queue belongs when the data to be read is incomplete or the offset of the data to be read is discontinuous.

15. The data acquisition device according to claim 12, characterized in that: The data acquisition device further includes: a processing unit and a sending unit; The processing unit is further used to construct a log object of the storage partition to which the data to be read belongs; the log object is used to determine whether the files under the directory of the storage partition to which the data to be read belongs are complete; The sending unit is further used to send a first data verification request including the log object to the electronic device; The data acquisition device also includes a receiving unit; The receiving unit is used to receive data verification information sent by the electronic device in response to the first data verification request.

16. The data acquisition device according to claim 15, characterized in that: The sending unit is further configured to send a second data read request for acquiring the data to be read to the server to which the message queue belongs when the data verification information is used to indicate that the file under the directory of the storage partition to which the data to be read belongs is incomplete.

17. The data acquisition device according to claim 12, characterized in that: The data acquisition device further includes: a sending unit and a receiving unit; The sending unit is further used to send a third read data request to the electronic device; the third read data request is used to request to sequentially obtain other data in the message queue except the data to be read in a certain order; the first segment of the other data is the next segment of data adjacent to the data to be read; the third read data request includes an offset of the data to be read and an offset of the next segment of data; the third read data request includes a data request identifier for requesting to obtain the other data; The receiving unit is further configured to receive the next segment of data sent by the electronic device in response to the third data read request.

18. The data acquisition device according to claim 17, characterized in that: The sending unit is further used to send a data prefetch request to the electronic device; the data prefetch request is used to instruct the electronic device to perform a data prefetch operation on the next segment of data; the data prefetch request includes a data prefetch request identifier used to instruct the electronic device to perform the data prefetch operation.

19. The data acquisition device according to claim 12, characterized in that: The data acquisition device further includes: a sending unit and a receiving unit; The sending unit is further used to send a second data verification request to the electronic device when the data to be read is the Jth segment of data to be read in the message queue; the second data verification request is used to determine whether new data to be read is generated in the message queue within a target time period; the target time period is a time period for reading the Jth segment of data to be read; J is an integer greater than 0 and less than N; N is the number of data segments in the message queue; The receiving unit is further configured to receive a data verification result sent by the electronic device in response to the second data verification request; The sending unit is further used to send a fourth data read request for obtaining the Jth segment of data to be read to the electronic device when the data verification result is used to indicate that the new data to be read is generated in the message queue within the target time period.

20. The data acquisition device according to any one of claims 12 to 19, characterized in that: Also includes: Output unit; The output unit is used to output alarm information when an abnormality occurs in obtaining data in the message queue from the electronic device; The alarm information includes log information of abnormal data.

21. A data acquisition device, characterized in that: include: A receiving unit and a sending unit; The receiving unit is used to receive a location information acquisition request sent by a client; The location information acquisition request is used to request to acquire the location information of the electronic device storing the data in the message queue; The location information acquisition request is sent when the delay time for the client to obtain the data in the message queue from the server is greater than a preset time threshold; the data in the message queue is stored in the electronic device through a distributed file system; The sending unit is used to send the location information to the client, so that the client obtains an index file from the electronic device based on the location information of the electronic device; the index file is used to read a file under a directory of a storage partition of the data to be read in the distributed file system; the index file includes a starting offset and an offset of the data in the message queue; and the client sends a first read data request including the starting offset and the offset of the data to be read to the electronic device, and receives the data to be read sent by the electronic device in response to the first read data request.

22. The data acquisition device according to claim 21, characterized in that: The data to be read meets the following preset conditions: the file of the data to be read under the directory of the storage partition in the distributed file system is complete, and the data to be read belongs to the first M segments of data in the message queue; M is an integer greater than 0 and less than N; N is the number of data segments in the message queue.

23. A client, characterized in that: The client comprises: processor; a memory for storing instructions executable by the processor; The processor is configured to execute the instructions to implement the data acquisition method according to any one of claims 1 to 9.

24. A server, characterized in that: The server comprises: processor; a memory for storing instructions executable by the processor; The processor is configured to execute the instructions to implement the data acquisition method as claimed in claim 10 or 11.

25. A computer-readable storage medium having instructions stored thereon, characterized in that: When the instructions in the computer-readable storage medium are executed by a processor of a client, the client is enabled to execute the data acquisition method according to any one of claims 1 to 9; Alternatively, when the instructions in the computer-readable storage medium are executed by a processor of a server, the server is enabled to execute the data acquisition method as claimed in claim 10 or 11.

26. A computer program product comprising instructions, characterized in that When the instruction is executed on the client, the client executes the data acquisition method according to any one of claims 1 to 9; Alternatively, when the instruction is executed on a server, the server is enabled to execute the data acquisition method as claimed in claim 10 or 11.

27. A data acquisition system, characterized in that: The data acquisition system comprises: a client and a server; The client is used to execute the data acquisition method according to any one of claims 1 to 9; The server is used to execute the data acquisition method as described in claim 10 or 11.

Images