Data acquisition method and device, electronic equipment and storage medium

Abstract

The invention discloses a data acquisition method and device, electronic equipment and a storage medium. The method comprises the following steps: determining a unique identifier of first data and a data identifier of the first data; if the data identifier of the first data is a first type of data identifier, determining second data in the data lake and a to-be-executed processing operation of the second data according to the unique identifier of the first data; and processing the second data according to the to-be-executed processing operation of the second data to obtain the first data. By adopting the technical scheme of the invention, the problem that the use experience of a user is influenced due to the fact that the second data in the data lake is not more than the first data and the acquisition of the first data needs to wait for a long time is avoided; in addition, the data lake can have sufficient time to carry out preprocessing work such as storage architecture and the like while timely acquisition of the data which is being updated or waits to be updated in the data lake by the user is not influenced, and the subsequent retrieval capability of the data lake is ensured.

Description

Technical Field

[0001] The present invention relates to the technical field of data distribution storage, and in particular to a data acquisition method, device, electronic equipment and storage medium. Background Art

[0002] The current streaming data lake platform supports inserting or updating incremental data on a distributed file system and provides ACID functionality.

[0003] The reason why the data lake supports updates is that the data is stored in small files after segmentation and indexed. Therefore, the original file needs to be rewritten for the modified data, which means that the data lake needs to be written in batches, and the data can only be queried after each batch is completed. Therefore, when querying the latest data, you need to wait for the data lake to complete the data update, but the waiting time is too long, which affects user use. Summary of the invention

[0004] The present invention provides a data acquisition method, device, electronic device and storage medium to solve the problem that when data is updated in a data lake, updated data cannot be obtained in time.

[0005] According to one aspect of the present invention, a data acquisition method is provided, the method comprising:

[0006] Determine a unique identifier of the first data and a data identifier of the first data;

[0007] If the data identifier of the first data is a first-category data identifier, the second data in the data lake and the processing operation to be executed of the second data are determined according to the unique identifier of the first data; the first data is the processing result after the second data is processed according to the processing operation to be executed; the first-category data identifier is used to identify that the data corresponding to the data identifier has not been updated in the data lake;

[0008] According to the processing operation to be performed on the second data, the second data is processed to obtain the first data.

[0009] According to another aspect of the present invention, there is provided a data acquisition device, the device comprising:

[0010] An identification acquisition module, used to determine a unique identification of the first data and a data identification of the first data;

[0011] A processing operation acquisition module, for determining, if the data identifier of the first data is a first-type data identifier, the second data in the data lake and the processing operation to be executed of the second data according to the unique identifier of the first data; the first data is a processing result after the second data is processed according to the processing operation to be executed; the first-type data identifier is used to identify that the data corresponding to the data identifier has not been updated in the data lake;

[0012] The first data acquisition module is used to process the second data according to the processing operation to be executed on the second data to obtain the first data.

[0013] According to another aspect of the present invention, an electronic device is provided, the electronic device comprising:

[0014] at least one processor; and

[0015] a memory communicatively connected to at least one processor; wherein,

[0016] The memory stores a computer program that can be executed by at least one processor. The computer program is executed by at least one processor so that the at least one processor can execute the data acquisition method of any embodiment of the present invention.

[0017] According to another aspect of the present invention, a computer-readable storage medium is provided. The computer-readable storage medium stores computer instructions, and the computer instructions are used to enable a processor to implement the data acquisition method of any embodiment of the present invention when executed.

[0018] The technical solution of the embodiment of the present invention determines the unique identifier of the first data and the data identifier of the first data; if the data identifier of the first data is a first-category data identifier, the second data in the data lake and the pending processing operations of the second data are determined according to the unique identifier of the first data, and finally the second data is processed according to the pending processing operations of the second data to obtain the first data, thereby avoiding the problem that the acquisition of the first data requires a long waiting time because the second data in the data lake has not yet been updated to the first data, thereby affecting the user experience. Through the combined use of the above steps, while not affecting the user's timely acquisition of the data being updated or waiting to be updated in the data lake, the data lake can have sufficient time to perform pre-processing work such as storage architecture, thereby ensuring the subsequent retrieval capability of the data lake.

[0019] It should be understood that the contents described in this section are not intended to identify the key or important features of the embodiments of the present invention, nor are they intended to limit the scope of the present invention. Other features of the present invention will become easily understood through the following description. BRIEF DESCRIPTION OF THE DRAWINGS

[0020] In order to more clearly illustrate the technical solutions in the embodiments of the present invention, the drawings required for use in the description of the embodiments will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without creative work.

[0021] Figure 1is a flow chart of a data acquisition method provided according to Embodiment 1 of the present invention;

[0022] Figure 2 is a flow chart of another data acquisition method provided according to Embodiment 2 of the present invention;

[0023] Figure 3 is a structural schematic diagram of a data acquisition device provided according to Embodiment 3 of the present invention;

[0024] Figure 4 It is a structural schematic diagram of an electronic device for implementing the data acquisition method according to an embodiment of the present invention. DETAILED DESCRIPTION

[0025] In order to enable those skilled in the art to better understand the scheme of the present invention, the technical scheme in the embodiments of the present invention will be clearly and completely described below in conjunction with the drawings in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work should fall within the scope of protection of the present invention.

[0026] It should be noted that the terms "first", "second", etc. in the specification and claims of the present invention and the above-mentioned drawings are used to distinguish similar objects, and are not necessarily used to describe a specific order or sequence. It should be understood that the data used in this way can be interchanged where appropriate, so that the embodiments of the present invention described herein can be implemented in an order other than those illustrated or described herein. In addition, the terms "including" and "having" and any variations are intended to cover non-exclusive inclusions, for example, a process, method, system, product or device that includes a series of steps or units is not necessarily limited to those steps or units clearly listed, but may include other steps or units that are not clearly listed or inherent to these processes, methods, products or devices.

[0027] Embodiment 1

[0028] Figure 1 A flowchart of a data acquisition method is provided for the first embodiment of the present invention. This embodiment is applicable to the case where data that has not been updated or is being updated in the data lake is acquired when the data is updated in the data lake. The method can be executed by a data acquisition device, which can be implemented in the form of hardware and / or software, and can be configured in an electronic device with data processing capabilities. Figure 1 As shown, the method includes:

[0029] S110: Determine a unique identifier of the first data and a data identifier of the first data.

[0030] The first data may be data that needs to be obtained. The data identifier may be an identifier used to indicate whether the first data has completed cache update in the data lake.

[0031] When obtaining data through a data lake, due to the characteristics of the data lake itself, the data update process is slow. Therefore, for queries on data that is waiting to be updated or being updated, it often takes a long time to get the updated results, which makes the use of the data lake for new data flawed. In this regard, existing methods will avoid the above problems by reducing the time it takes for the data lake to update data. However, when updating data, the data lake will involve the optimization of data architecture and other contents. Therefore, although the existing methods can improve the efficiency of updating data in the data lake, they will reduce the optimization time of the data lake in searching data, thereby affecting the user experience of the data lake.

[0032] For the determined unique identifier and data identifier corresponding to the first data, the unique identifier is used to search for the first data or the second data corresponding to the first data in the data lake, and the data identifier is used to indicate whether the first data has been updated.

[0033] In an optional solution, before determining the unique identifier of the first data and the data identifier of the first data, steps A1-A2 are further included:

[0034] Step A1: Determine an update log corresponding to at least one piece of data to be updated in the data lake.

[0035] Step A2: Update each data to be updated in the data lake according to the update log corresponding to each data to be updated.

[0036] When the data lake is updated, the source database will first update the data to be updated and register the change log of the source database. The change log records the detailed information of the data change in binary form, including the type of change (addition, modification, deletion), time, and data object. When the data lake is updating data, it will register the change log of the source database through the source database and update each data to be updated in the data lake.

[0037] In an optional solution, each data to be updated in the data lake is updated according to the update log corresponding to each data to be updated, which may include steps B1-B2:

[0038] Step B1: Determine the generation order of the update logs corresponding to the data to be updated.

[0039] Step B2: According to the update logs corresponding to the data to be updated, and in accordance with the generation order of the update logs corresponding to the data to be updated, the data to be updated corresponding to the update logs are updated.

[0040] To ensure the accuracy of the updated data, when updating each data to be updated in the data lake, the generation order of the update logs corresponding to each data to be updated will be determined first, and then the data in the data lake will be updated according to the generation order of the update logs corresponding to each data to be updated, so as to ensure the orderliness of the update process, thereby indirectly ensuring the accuracy of the update results and reducing the possibility of missing update data during the update process.

[0041] In an optional solution, after updating the data to be updated corresponding to each update log according to the generation order of the update logs corresponding to each data to be updated, the method further includes steps C1-C4:

[0042] Step C1: Generate a timestamp for the data to be updated corresponding to each update log according to the update time of the data to be updated corresponding to each update log.

[0043] Step C2: Generate an offset for each update log corresponding to each data to be updated according to the generation order of the update logs corresponding to each data to be updated.

[0044] Step C3: If the data to be updated corresponding to the update log has been updated, it is determined that the data identifier of the data to be updated is a second type of data identifier.

[0045] Step C4: If the data to be updated corresponding to the update log has not been updated, determine that the data identifier of the data to be updated is a first type of data identifier.

[0046] When updating data in a data lake, in order to ensure that the data can be found, it is often necessary to provide the data with a unique identifier. Since the data in the data lake is updated at different times, the time when the data in the data lake is updated can be used as a unique identifier. Therefore, based on the update time of each update log corresponding to the data to be updated, a timestamp can be generated for the data to be updated corresponding to each update log, and the timestamp can be used instead of the unique identifier of the data to be updated.

[0047] To ensure the orderliness of the update log, each update log may be assigned an offset. To this end, the offset may be generated for each update log corresponding to each data to be updated according to the generation order of the update log corresponding to each data to be updated.

[0048] When updating data in the data lake, different data identifiers can be assigned to the data to be updated according to the processing progress of the data to be updated, so that when the data is subsequently mobilized, it can be directly determined by the data identifier whether the data has been updated.

[0049] In this regard, the data identifier of the data that has been updated can be changed to the second type of data identifier, and the data identifier of the data that has not been updated can be changed to the first type of data identifier.

[0050] In addition to the above two possibilities, some data lakes may also classify the data to be updated into multiple types, such as scheduled update, updating, and completed update. For this, data identifiers can be set according to the type of data lake and actual needs.

[0051] S120: If the data identifier of the first data is a first-category data identifier, determine the second data in the data lake and the processing operation to be executed on the second data according to the unique identifier of the first data.

[0052] The first data is a processing result of processing the second data according to the processing operation to be executed; the first type of data identifier is used to identify that the data corresponding to the data identifier has not been updated in the data lake.

[0053] When the data identifier of the first data is a first-class data identifier, it can be determined that the first data needs to be updated in the data lake, but has not been updated yet. In this case, in order to ensure that the first data can be obtained as soon as possible, the second data corresponding to the first data and the pending processing operations of the second data can be obtained from the data lake first, thereby skipping the time required for the data lake to update the second data to the first data. The pending processing operations can be operations to be performed when the second data is updated to the first data.

[0054] In an optional solution, if the data identifier of the first data is a first-category data identifier, determining the second data in the data lake and the processing operation to be performed on the second data according to the unique identifier of the first data may include steps D1-D4:

[0055] Step D1: If the data identifier of the first data is a first type of data identifier, determine the offset corresponding to the data with the maximum timestamp in the data lake as the first offset.

[0056] Step D2: According to the first offset, the update logs corresponding to the data to be updated are screened to obtain at least one target update log; the offset of the target update log is greater than the offset corresponding to the data with the maximum timestamp in the data lake.

[0057] Step D3: Screen each target update log to obtain the update log corresponding to the first data.

[0058] Step D4: Determine the second data according to the timestamp of the first data.

[0059] When obtaining the second data in the data lake and the processing operations to be executed on the second data, since each data to be updated corresponds to an update log, the data to be updated corresponding to each update log is updated according to the generation order of the update logs corresponding to each data to be updated. Therefore, the unused update log can be determined as the target update log by determining the offset corresponding to the data with the largest timestamp in the data lake, that is, the offset of the update log corresponding to the latest updated data, thereby reducing the range of update logs to be searched when determining the update log of the first data, thereby improving the search efficiency. After obtaining at least one target update log, each target update log is screened to obtain the update log corresponding to the first data. The second data can be determined in the data lake based on the timestamp of the first data.

[0060] S130: Process the second data according to the processing operation to be executed on the second data to obtain the first data.

[0061] After obtaining the pending processing operations of the second data and the second data, since the pending processing operations of the second data are the specific operations for updating the second data to the first data, the second data can be directly processed through the pending processing operations of the second data to obtain the first data, thereby avoiding the problem that the acquisition of the first data requires a long waiting time because the second data in the data lake has not yet been updated to the first data, thereby affecting the user experience.

[0062] According to the technical solution of the embodiment of the present invention, by determining the unique identifier of the first data and the data identifier of the first data; if the data identifier of the first data is a first-category data identifier, the second data in the data lake and the pending processing operations of the second data are determined according to the unique identifier of the first data, and finally the second data is processed according to the pending processing operations of the second data to obtain the first data, thereby avoiding the problem that the acquisition of the first data requires a long waiting time due to the fact that the second data in the data lake has not yet been updated to the first data, thereby affecting the user experience. Through the combined use of the above steps, while not affecting the user's timely acquisition of the data being updated or waiting to be updated in the data lake, the data lake can have sufficient time to perform pre-processing work such as storage architecture, thereby ensuring the subsequent retrieval capability of the data lake.

[0063] Embodiment 2

[0064] Figure 2A flowchart of another data acquisition method is provided for an embodiment of the present invention. Based on the above embodiment, this embodiment further optimizes the process after the second data is processed according to the processing operation to be executed on the second data to obtain the first data in the above embodiment. This embodiment can be combined with various optional solutions in one or more of the above embodiments. Figure 2 As shown, the data acquisition method of this embodiment may include the following steps:

[0065] S210: Determine a unique identifier of the first data and a data identifier of the first data.

[0066] S220. If the data identifier of the first data is a first-category data identifier, determine the second data in the data lake and the processing operation to be executed for the second data according to the unique identifier of the first data; the first data is a processing result after processing the second data according to the processing operation to be executed; the first-category data identifier is used to identify that the data corresponding to the data identifier has not been updated in the data lake.

[0067] S230: Process the second data according to the processing operation to be executed on the second data to obtain the first data.

[0068] S240: If the data identifier of the first data is a second-category data identifier, determine the first data in the data lake according to the unique identifier of the first data.

[0069] The second type of data identifier is used to identify that the data corresponding to the data identifier has been updated in the data lake.

[0070] In addition to the possibility that the data identifier of the first data is a first-category data identifier, it is also possible that the data identifier of the first data is a second-category data identifier, that is, the first data in the data lake has been updated. At this time, the first data in the data lake can be directly determined by the unique identifier of the first data.

[0071] Optionally, if the data identifier of the first data is a second-category data identifier, determining the first data in the data according to the unique identifier of the first data includes:

[0072] If the data identifier of the first data is a first type of data identifier, the first data is determined based on the timestamp of the first data.

[0073] Since the data identifier of the first data is a timestamp, the first data can be determined according to the timestamp of the first data.

[0074] According to the technical solution of the embodiment of the present invention, if the data identifier of the first data is a second-category data identifier, the first data in the data lake is determined according to the unique identifier of the first data, so that when the data identifier of the first data is a second-category data identifier, the first data in the data lake can be directly called, thereby improving data acquisition efficiency and enhancing user experience.

[0075] Embodiment 3

[0076] Figure 3 The present invention provides a structural block diagram of a data acquisition device, which can be applied to the case where data in a data lake that has not been updated or is being updated is acquired when data is updated in the data lake. The data acquisition device can be implemented in the form of hardware and / or software, and the data acquisition device can be configured in an electronic device with data processing capabilities. Figure 3 As shown, the data acquisition device of this embodiment may include: an identification acquisition module 310, a processing operation acquisition module 320 and a first data acquisition module 330. Among them:

[0077] An identification acquisition module 310, used to determine a unique identification of the first data and a data identification of the first data;

[0078] The processing operation acquisition module 320 is used to determine the second data in the data lake and the processing operation to be executed of the second data according to the unique identifier of the first data if the data identifier of the first data is a first-type data identifier; the first data is the processing result after the second data is processed according to the processing operation to be executed; the first-type data identifier is used to identify that the data corresponding to the data identifier has not been updated in the data lake;

[0079] The first data acquisition module 330 is used to process the second data according to the processing operation to be executed on the second data to obtain the first data.

[0080] Based on the above embodiment, optionally, after the first data acquisition module 330, the following further includes:

[0081] If the data identifier of the first data is a second-type data identifier, the first data in the data lake is determined according to the unique identifier of the first data; the second-type data identifier is used to identify that the data corresponding to the data identifier has been updated in the data lake.

[0082] Based on the above embodiment, optionally, before the identification acquisition module 310, the following is further included:

[0083] Determine an update log corresponding to at least one piece of data to be updated in the data lake;

[0084] According to the update logs corresponding to each data to be updated, each data to be updated in the data lake is updated.

[0085] Based on the above embodiment, optionally, each data to be updated in the data lake is updated according to the update log corresponding to each data to be updated, including:

[0086] Determine the generation order of update logs corresponding to each data to be updated;

[0087] According to the update logs corresponding to the data to be updated, and in the generation order of the update logs corresponding to the data to be updated, the data to be updated corresponding to the update logs are updated.

[0088] Based on the above embodiment, optionally, after updating the data to be updated corresponding to each update log according to the update logs corresponding to each data to be updated and in the generation order of the update logs corresponding to each data to be updated, the method further includes:

[0089] Generate a timestamp for the data to be updated corresponding to each update log according to the update time of the data to be updated corresponding to each update log;

[0090] Generate an offset for each update log corresponding to each data to be updated according to the generation order of the update logs corresponding to each data to be updated;

[0091] If the data to be updated corresponding to the update log has been updated, determining that the data identifier of the data to be updated is a second type of data identifier;

[0092] If the data to be updated corresponding to the update log has not been updated, it is determined that the data identifier of the data to be updated is a first type of data identifier.

[0093] Based on the above embodiment, optionally, the unique identifier of the first data is a timestamp of the last update of the first data in the data lake; the processing operation to be executed is an update log;

[0094] Correspondingly, if the data identifier of the first data is a first-category data identifier, determining the second data in the data and the processing operation to be performed on the second data according to the unique identifier of the first data includes:

[0095] If the data identifier of the first data is a first-category data identifier, determining an offset corresponding to the data with the maximum timestamp in the data lake as a first offset;

[0096] According to the first offset, the update logs corresponding to the data to be updated are respectively screened to obtain at least one target update log; the offset of the target update log is greater than the offset corresponding to the data with the maximum timestamp in the data lake;

[0097] Screening each target update log to obtain an update log corresponding to the first data;

[0098] The second data is determined according to the timestamp of the first data.

[0099] Based on the above embodiment, optionally, the unique identifier of the first data is a timestamp of the last update of the first data in the data lake; the processing operation to be executed is an update log;

[0100] Correspondingly, if the data identifier of the first data is a second-category data identifier, determining the first data in the data according to the unique identifier of the first data includes:

[0101] If the data identifier of the first data is a first type of data identifier, the first data is determined based on the timestamp of the first data.

[0102] The data acquisition device provided in the embodiment of the present invention can execute the data acquisition method provided in any embodiment of the present invention, and has the corresponding functional modules and beneficial effects of the execution method.

[0103] Embodiment 4

[0104] Figure 4 A schematic diagram of the structure of an electronic device 10 that can be used to implement an embodiment of the present invention is shown. The electronic device is intended to represent various forms of digital computers, such as laptop computers, desktop computers, workstations, personal digital assistants, servers, blade servers, mainframe computers, and other suitable computers. The electronic device can also represent various forms of mobile devices, such as personal digital processing, cellular phones, smart phones, wearable devices (such as helmets, glasses, watches, etc.) and other similar computing devices. The components shown herein, their connections and relationships, and their functions are merely examples and are not intended to limit the implementation of the present invention described and / or required herein.

[0105] like Figure 4 As shown, the electronic device 10 includes at least one processor 11, and a memory connected to the at least one processor 11, such as a read-only memory (ROM) 12, a random access memory (RAM) 13, etc., wherein the memory stores a computer program that can be executed by at least one processor, and the processor 11 can perform various appropriate actions and processes according to the computer program stored in the read-only memory (ROM) 12 or the computer program loaded from the storage unit 18 to the random access memory (RAM) 13. In the RAM 13, various programs and data required for the operation of the electronic device 10 can also be stored. The processor 11, the ROM 12, and the RAM 13 are connected to each other through a bus 14. An input / output (I / O) interface 15 is also connected to the bus 14.

[0106] A number of components in the electronic device 10 are connected to the I / O interface 15, including: an input unit 16, such as a keyboard, a mouse, etc.; an output unit 17, such as various types of displays, speakers, etc.; a storage unit 18, such as a disk, an optical disk, etc.; and a communication unit 19, such as a network card, a modem, a wireless communication transceiver, etc. The communication unit 19 allows the electronic device 10 to exchange information / data with other devices through a computer network such as the Internet and / or various telecommunication networks.

[0107] The processor 11 may be a variety of general and / or special processing components with processing and computing capabilities. Some examples of the processor 11 include, but are not limited to, a central processing unit (CPU), a graphics processing unit (GPU), various special artificial intelligence (AI) computing chips, various processors running machine learning model algorithms, a digital signal processor (DSP), and any appropriate processor, controller, microcontroller, etc. The processor 11 executes the various methods and processes described above, such as a data acquisition method.

[0108] In some embodiments, the data acquisition method may be implemented as a computer program, which is tangibly contained in a computer-readable storage medium, such as a storage unit 18. In some embodiments, part or all of the computer program may be loaded and / or installed on the electronic device 10 via the ROM 12 and / or the communication unit 19. When the computer program is loaded into the RAM 13 and executed by the processor 11, one or more steps of the data acquisition method described above may be performed. Alternatively, in other embodiments, the processor 11 may be configured to perform the data acquisition method in any other appropriate manner (e.g., by means of firmware).

[0109] Various implementations of the systems and techniques described above herein can be implemented in digital electronic circuit systems, integrated circuit systems, field programmable gate arrays (FPGAs), application specific integrated circuits (ASICs), application specific standard products (ASSPs), systems on chips (SOCs), load programmable logic devices (CPLDs), computer hardware, firmware, software, and / or combinations thereof. These various implementations can include: being implemented in one or more computer programs that can be executed and / or interpreted on a programmable system including at least one programmable processor, which can be a special purpose or general purpose programmable processor that can receive data and instructions from a storage system, at least one input device, and at least one output device, and transmit data and instructions to the storage system, the at least one input device, and the at least one output device.

[0110] Computer programs for implementing the methods of the present invention may be written in any combination of one or more programming languages. These computer programs may be provided to a processor of a general-purpose computer, a special-purpose computer, or other programmable data processing device, so that when the computer program is executed by the processor, the functions / operations specified in the flow chart and / or block diagram are implemented. The computer program may be executed entirely on the machine, partially on the machine, partially on the machine and partially on a remote machine as a stand-alone software package, or entirely on a remote machine or server.

[0111] In the context of the present invention, a computer-readable storage medium may be a tangible medium that may contain or store a computer program for use by or in combination with an instruction execution system, device or equipment. A computer-readable storage medium may include, but is not limited to, electronic, magnetic, optical, electromagnetic, infrared, or semiconductor systems, devices or equipment, or any suitable combination of the foregoing. Alternatively, a computer-readable storage medium may be a machine-readable signal medium. A more specific example of a machine-readable storage medium may include an electrical connection based on one or more lines, a portable computer disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the foregoing.

[0112] To provide interaction with a user, the systems and techniques described herein may be implemented on an electronic device having: a display device (e.g., a CRT (cathode ray tube) or LCD (liquid crystal display) monitor) for displaying information to the user; and a keyboard and a pointing device (e.g., a mouse or trackball) through which the user can provide input to the electronic device. Other types of devices may also be used to provide interaction with the user; for example, the feedback provided to the user may be any form of sensory feedback (e.g., visual feedback, auditory feedback, or tactile feedback); and input from the user may be received in any form (including acoustic input, voice input, or tactile input).

[0113] The systems and techniques described herein may be implemented in a computing system that includes backend components (e.g., as a data server), or a computing system that includes middleware components (e.g., an application server), or a computing system that includes frontend components (e.g., a user computer with a graphical user interface or a web browser through which a user can interact with implementations of the systems and techniques described herein), or a computing system that includes any combination of such backend components, middleware components, or frontend components. The components of the system may be interconnected by any form or medium of digital data communication (e.g., a communication network). Examples of communication networks include: a local area network (LAN), a wide area network (WAN), a blockchain network, and the Internet.

[0114] A computing system may include a client and a server. The client and the server are generally remote from each other and usually interact through a communication network. The client and server relationship is generated by computer programs running on the corresponding computers and having a client-server relationship with each other. The server may be a cloud server, also known as a cloud computing server or cloud host, which is a host product in the cloud computing service system to solve the defects of difficult management and weak business scalability in traditional physical hosts and VPS services.

[0115] It should be understood that the various forms of processes shown above can be used to reorder, add or delete steps. For example, the steps described in the present invention can be executed in parallel, sequentially or in different orders, as long as the desired results of the technical solution of the present invention can be achieved, and this document does not limit this.

[0116] The above specific implementations do not constitute a limitation on the protection scope of the present invention. It should be understood by those skilled in the art that various modifications, combinations, sub-combinations and substitutions can be made according to design requirements and other factors. Any modification, equivalent substitution and improvement made within the spirit and principle of the present invention should be included in the protection scope of the present invention.

Claims

1. A data acquisition method, characterized in that: include: Determine a unique identifier of the first data and a data identifier of the first data; If the data identifier of the first data is a first-category data identifier, determining the second data in the data lake and the processing operation to be performed on the second data according to the unique identifier of the first data; The first data is a processing result after the second data is processed according to the processing operation to be performed; The first type of data identifier is used to identify that the data corresponding to the data identifier has not been updated in the data lake; The second data is processed according to the processing operation to be executed on the second data to obtain the first data.

2. The method according to claim 1, characterized in that After the second data is processed according to the to-be-performed processing operation of the second data to obtain the first data, the method further includes: If the data identifier of the first data is a second-type data identifier, the first data in the data lake is determined according to the unique identifier of the first data; the second-type data identifier is used to identify that the data corresponding to the data identifier has been updated in the data lake.

3. The method according to claim 1, characterized in that Before determining the unique identifier of the first data and the data identifier of the first data, the method further includes: Determine an update log corresponding to at least one piece of data to be updated in the data lake; According to the update logs corresponding to each data to be updated, each data to be updated in the data lake is updated.

4. The method according to claim 3, characterized in that According to the update logs corresponding to each data to be updated, each data to be updated in the data lake is updated, including: Determine the generation order of update logs corresponding to each data to be updated; According to the update logs corresponding to the data to be updated, and in the generation order of the update logs corresponding to the data to be updated, the data to be updated corresponding to the update logs are updated.

5. The method according to claim 4, characterized in that After updating the data to be updated corresponding to each update log according to the update logs corresponding to each data to be updated and in the generation order of the update logs corresponding to each data to be updated, the method further includes: Generate a timestamp for the data to be updated corresponding to each update log according to the update time of the data to be updated corresponding to each update log; Generate an offset for each update log corresponding to each data to be updated according to the generation order of the update logs corresponding to each data to be updated; If the data to be updated corresponding to the update log has been updated, determining that the data identifier of the data to be updated is a second type of data identifier; If the data to be updated corresponding to the update log has not been updated, it is determined that the data identifier of the data to be updated is a first type of data identifier.

6. The method according to claim 5, characterized in that The unique identifier of the first data is a timestamp of the last update of the first data in the data lake; the processing operation to be executed is an update log; Correspondingly, if the data identifier of the first data is a first-category data identifier, determining the second data in the data and the processing operation to be performed on the second data according to the unique identifier of the first data includes: If the data identifier of the first data is a first type of data identifier, determining an offset corresponding to the data with the maximum timestamp in the data lake as a first offset; According to the first offset, the update logs corresponding to the data to be updated are respectively screened to obtain at least one target update log; the offset of the target update log is greater than the offset corresponding to the data with the maximum timestamp in the data lake; Screening each target update log to obtain an update log corresponding to the first data; The second data is determined according to the timestamp of the first data.

7. The method according to claim 5, characterized in that The unique identifier of the first data is a timestamp of the last update of the first data in the data lake; the processing operation to be executed is an update log; Correspondingly, if the data identifier of the first data is a second-category data identifier, determining the first data in the data according to the unique identifier of the first data includes: If the data identifier of the first data is a first type of data identifier, the first data is determined according to the timestamp of the first data.

8. A data acquisition device, characterized in that: include: An identification acquisition module, used to determine a unique identification of the first data and a data identification of the first data; a processing operation acquisition module, configured to determine, if the data identifier of the first data is a first type of data identifier, second data in the data lake and a processing operation to be executed on the second data according to the unique identifier of the first data; The first data is a processing result after the second data is processed according to the processing operation to be performed; The first type of data identifier is used to identify that the data corresponding to the data identifier has not been updated in the data lake; The first data acquisition module is used to process the second data according to the processing operation to be executed on the second data to obtain the first data.

9. An electronic device, characterized in that: The electronic device comprises: at least one processor; and a memory communicatively connected to the at least one processor; wherein, The memory stores a computer program executable by the at least one processor, and the computer program is executed by the at least one processor so that the at least one processor can perform the data acquisition method according to any one of claims 1 to 7.

10. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores computer instructions, and the computer instructions are used to enable a processor to implement the data acquisition method according to any one of claims 1 to 7 when executed.

Images