Data processing method and device, equipment, storage medium and program product

Abstract

The embodiment of the invention relates to a data processing method and device, equipment, a storage medium and a program product. The method comprises the following steps: in response to a received data acquisition request for target data, determining the data volume of the requested target data, a corresponding data dimension and a requested time period; and determining at least one target acquisition mode of the target data from a plurality of selectable data modes based on at least one of the data volume, the corresponding data dimension and the time period, the plurality of selectable data modes at least corresponding to different acquisition durations and acquisition paths. And obtaining target data by routing the data obtaining request to the corresponding obtaining path based on the at least one target obtaining mode. Therefore, appropriate data acquisition paths can be selected for different target data, and the experience of acquiring the target data by the user is improved.

Description

Technical Field

[0001] Example embodiments of the present disclosure generally relate to the field of computer technology, and more particularly, to a method, apparatus, electronic device, computer-readable storage medium, and computer program product for data processing. Background Art

[0002] Currently, in the big data sector, data processing and analysis typically rely on three data processing modes: online, nearline, and offline, each with distinct characteristics and applicable scenarios. The online data processing mode can provide millisecond-level ad hoc queries and supports high-speed data read and write, making it suitable for queries and transactions that require timely responses. The nearline data processing mode offers a compromise between performance and cost. Using the same data source as the online mode, server-side task scheduling enables tidal cluster utilization and rolling access to larger amounts of data. The offline data processing mode, based on an offline data warehouse model, runs periodic data loading and processing tasks, making it suitable for batch data processing that does not require immediate responses. Summary of the Invention

[0003] In a first aspect of the present disclosure, a method for data processing is provided. The method includes, in response to receiving a data acquisition request for target data, determining a data volume, a corresponding data dimension, and a requested time period for the requested target data. Based on at least one of the data volume, the corresponding data dimension, and the time period, determining at least one target acquisition mode for the target data from a plurality of selectable data modes, the plurality of selectable data modes corresponding to at least different acquisition durations and acquisition paths. Furthermore, the target data is acquired by routing the data acquisition request to the corresponding acquisition path based on the at least one target acquisition mode.

[0004] In a second aspect of the present disclosure, a device for data processing is provided. The device includes: a request determination module configured to, in response to receiving a data acquisition request for target data, determine the data volume, corresponding data dimension, and requested time period of the requested target data; a path selection module configured to, based on at least one of the data volume, corresponding data dimension, and time period, determine at least one target acquisition mode for the target data from a plurality of optional data modes, the plurality of optional data modes corresponding to at least different acquisition durations and acquisition paths; and a data acquisition module configured to acquire the target data by routing the data acquisition request to a corresponding acquisition path based on the at least one target acquisition mode.

[0005] In a third aspect of the present disclosure, an electronic device is provided. The device includes at least one processing unit; and at least one memory coupled to the at least one processing unit and storing instructions for execution by the at least one processing unit. When executed by the at least one processing unit, the instructions cause the electronic device to perform the method of the first aspect.

[0006] In a fourth aspect of the present disclosure, a computer-readable storage medium is provided, wherein a computer program is stored on the medium, and when the computer program is executed by a processor, the method of the first aspect is implemented.

[0007] In a fifth aspect of the present disclosure, a computer program product is provided, which includes a computer program, wherein when the computer program is executed by a processor, the method according to the first aspect of the present disclosure is implemented.

[0008] It should be understood that the content described in this section is not intended to limit the key features or important features of the embodiments of the present disclosure, nor is it intended to limit the scope of the present disclosure. Other features of the present disclosure will become easily understood through the following description. BRIEF DESCRIPTION OF THE DRAWINGS

[0009] The above and other features, advantages and aspects of the various embodiments of the present disclosure will become more apparent hereinafter with reference to the following detailed description in conjunction with the accompanying drawings. In the accompanying drawings, the same or similar reference numerals represent the same or similar elements, wherein:

[0010] Figure 1 A schematic diagram illustrating an example environment in which embodiments of the present disclosure can be implemented;

[0011] Figure 2 A flowchart of a method for data processing according to some embodiments of the present disclosure is shown;

[0012] Figure 3 A schematic diagram illustrating an acquisition path conversion architecture of a data management platform according to some embodiments of the present disclosure is shown;

[0013] Figure 4A A schematic diagram of a data request link of a data management platform according to some embodiments of the present disclosure;

[0014] Figure 4B A flowchart illustrating an asynchronous download process of a data management platform according to some embodiments of the present disclosure is shown;

[0015] Figure 5 A schematic diagram illustrating a data management and synchronization process of a data management platform according to some embodiments of the present disclosure is shown;

[0016] Figure 6A schematic structural block diagram showing an apparatus for data processing according to certain embodiments of the present disclosure; and

[0017] Figure 7 A block diagram is shown of a computing device in which one or more embodiments of the present disclosure may be implemented. DETAILED DESCRIPTION

[0018] The following describes embodiments of the present disclosure in more detail with reference to the accompanying drawings. Although some embodiments of the present disclosure are shown in the accompanying drawings, it should be understood that the present disclosure can be implemented in various forms and should not be construed as limited to the embodiments described herein. Instead, these embodiments are provided to provide a more thorough and complete understanding of the present disclosure. It should be understood that the drawings and embodiments of the present disclosure are for illustrative purposes only and are not intended to limit the scope of protection of the present disclosure.

[0019] In the description of the embodiments of the present disclosure, the term "including" and similar terms should be understood as open inclusion, i.e., "including but not limited to." The term "based on" should be understood as "based at least in part on." The term "one embodiment" or "the embodiment" should be understood as "at least one embodiment." The term "some embodiments" should be understood as "at least some embodiments." Other explicit and implicit definitions may be included below.

[0020] It should be noted that the acquisition, storage and application of user personal information involved in the technical solution of this disclosure are in compliance with the provisions of relevant laws and regulations and do not violate public order and good morals.

[0021] It is understandable that before using the technical solutions disclosed in the various embodiments of this disclosure, the type, scope of use, usage scenarios, etc. of the personal information involved in this disclosure should be informed to the user and the user's authorization should be obtained in an appropriate manner in accordance with relevant laws and regulations.

[0022] For example, in response to receiving a user's active request, a prompt message is sent to the user to clearly remind the user that the operation requested to be performed will require obtaining and using the user's personal information, so that the user can independently choose whether to provide personal information to the electronic device, application, server or storage medium and other software or hardware that performs the operation of the technical solution of the present disclosure based on the prompt message.

[0023] As an optional but non-limiting implementation, in response to receiving a user's active request, a prompt message may be sent to the user, for example, in the form of a pop-up window, in which the prompt message may be presented in text form. Furthermore, the pop-up window may also include a selection control for the user to select "agree" or "disagree" to provide personal information to the electronic device.

[0024] It is understandable that the above notification and user authorization process are merely illustrative and do not limit the embodiments of the present disclosure. Other methods that meet relevant laws and regulations may also be applied to the embodiments of the present disclosure.

[0025] Figure 1 1 shows a schematic diagram of an example environment 100 in which embodiments of the present disclosure can be implemented. Figure 1 As shown, example environment 100 may include a client device 110 and a server device 120 .

[0026] In the example environment 100 , the client device 110 can allow users to submit data processing requests, such as registering, querying, modifying, and deleting indicator data. The server device 120 can perform data processing in response to the data processing request sent by the client device 110 and return the processing result to the client device 110 .

[0027] The client device 110 may include any computing system with computing capabilities, such as various computing devices / systems, terminal devices, server devices, etc. The terminal device may be any type of mobile terminal, fixed terminal, or portable terminal, including a mobile phone, a desktop computer, a laptop computer, a notebook computer, a netbook computer, a tablet computer, a media computer, a multimedia tablet, a handheld computer, a portable game terminal, a VR / AR device, a personal communication system (PCS) device, a personal navigation device, a personal digital assistant (PDA), an audio / video player, a digital camera / camcorder, a positioning device, a television receiver, a radio broadcast receiver, an e-book device, a gaming device, or any combination thereof, including accessories and peripherals of these devices or any combination thereof.

[0028] The server device 120 may be a standalone physical server, a server cluster or distributed system composed of multiple physical servers, or a cloud server that provides basic cloud computing services such as cloud services, cloud databases, cloud computing, cloud functions, cloud storage, network services, cloud communications, middleware services, domain name services, security services, content distribution networks, and big data and artificial intelligence platforms. For example, the server device may include a computing system / server, such as a mainframe, an edge computing node, a computing device in a cloud environment, and the like.

[0029] It should be understood that the structure and function of the various elements in the environment 100 are described for illustrative purposes only and do not imply any limitation on the scope of the present disclosure.

[0030] Currently, indicator data processing typically involves multi-level integration to create a final indicator data management platform. However, the definition and processing of indicator data at each level or source are inconsistent, resulting in a lack of standardized processes for data integration. This not only increases maintenance costs but also makes data restoration and verification more difficult, making data traceability and verification extremely challenging. Furthermore, online technologies face performance bottlenecks when processing large-scale and long-term data queries, limiting users' ability to access large amounts of data at these levels and over long periods of time. These issues impact the efficiency and accuracy of data processing.

[0031] In view of this, an embodiment of the present disclosure provides an improved solution for data processing. In this solution, in response to receiving a data acquisition request for target data, the data volume, corresponding data dimension and requested time period of the requested target data are determined. Based on at least one of the data volume, corresponding data dimension and time period, at least one target acquisition mode of the target data is determined from a plurality of optional data modes, and the plurality of optional data modes correspond to at least different acquisition durations and acquisition paths. And the target data is acquired by routing the data acquisition request to the corresponding acquisition path based on at least one target acquisition mode. In this way, appropriate data acquisition paths can be selected for different target data, thereby improving the user experience of acquiring target data. For example, for large-scale and long-period data, it can be converted into an offline task, which not only saves the cost of online storage resources, but also enables the acquisition of larger-scale and longer-period data, thereby comprehensively improving the user experience.

[0032] Some example embodiments of the present disclosure will be described below with continued reference to the accompanying drawings.

[0033] Figure 2 FIG2 shows a flow chart of a process 200 for data processing according to some embodiments of the present disclosure. The process 200 may be implemented at the server device 120. For ease of discussion, reference will be made to FIG20 for data processing. Figure 1 The process 200 is described with reference to the environment 100 of FIG.

[0034] At block 210 , in response to receiving a data acquisition request for target data, the server device 120 determines the data volume, corresponding data dimensions, and requested time period of the requested target data.

[0035] In the embodiment of the present disclosure, the data volume of the target data refers to the number of data included in the target data. Exemplarily, the target data is indicator data, and the data volume of the target data refers to how many indicator data the target data includes. Indicator data refers to various indicators, such as transaction amount, transaction volume and other indicators. The data dimension corresponding to the target data refers to the dimension requested in the data acquisition request, such as the product minimum stock unit SKU, buyer age, buyer gender, buyer education, buyer city and other descriptive or metric dimensions for indicator data (such as the transaction amount of a certain product). The requested time period refers to the time period requested to be acquired or queried in the data acquisition request. For example, for a certain indicator, a request is made to query data within one month or one year, where one month or one year is the requested time period.

[0036] It should be understood that the data volume of the target data can be determined by calculation based on the corresponding data dimension and the requested time period.

[0037] At box 220, the server device 120 determines at least one target acquisition mode of the target data from multiple optional data modes based on at least one of the data volume, the corresponding data dimension and the time period, and the multiple optional data modes correspond to at least different acquisition durations and acquisition paths.

[0038] In an embodiment of the present disclosure, selectable data modes include an online acquisition mode, a near-line acquisition mode, and an offline acquisition mode.

[0039] Figure 3 The schematic diagram of the data management platform acquisition path conversion architecture 300 according to some embodiments of the present disclosure is shown. The server device 120 can be based on Figure 3 The architecture shown is implemented.

[0040] like Figure 3 As shown, in the embodiment of the present disclosure, the data management platform provides metadata of management indicator data based on a unified logic, and intelligently determines the data acquisition mode based on the metadata of the unified logic.

[0041] For example, the online acquisition mode is completed through online calculation and processing, which has the characteristics of millisecond-level timeliness, shorter time periods, faster calculations, smaller data volumes, and the ability to perform ad-hoc queries. For example, taking indicator data as an example, the online acquisition mode supports the acquisition of less than 2,000 data items.

[0042] Nearline acquisition mode uses nearline computing and processing, with a timeliness of seconds to minutes, for example. It uses the same source as online data, and server-side devices can leverage cluster tidal rolling to acquire larger amounts of data than online acquisition mode. For example, using indicator data as an example, nearline acquisition mode supports acquisition of between 2,000 and 400,000 data items.

[0043] Offline acquisition mode is accomplished through offline calculation and processing, with a timeframe of minutes to hours, for example. This longer timeframe and more complex calculations are based on metadata from the unified logic of the data management platform. Simultaneously, it performs SQL language conversion or translation, converting system-state and user-state data into offline task scripts for data acquisition. For example, using indicator data as an example, offline acquisition mode supports the acquisition of between 400,000 and 100 million data items.

[0044] In some embodiments of the present disclosure, determining at least one target acquisition mode for target data based on data volume includes: determining a first data volume threshold and a second data volume threshold, wherein the second data volume threshold is greater than the first data volume threshold. In response to determining that the data volume does not exceed the first data volume threshold, determining an online acquisition mode as at least one target acquisition mode. Or in response to determining that the data volume exceeds the first data volume threshold and does not exceed the second data volume threshold, determining at least one target acquisition mode as a near-line acquisition mode; or in response to determining that the data volume exceeds the second data volume threshold, determining an offline acquisition mode as at least one target acquisition mode. Exemplarily, the first data volume threshold is 2,000 items, and the second data volume threshold is 400,000 items.

[0045] In some embodiments of the present disclosure, determining at least one target acquisition mode for target data based on data volume includes: determining a third data volume threshold, wherein the third data volume threshold is greater than the second data volume threshold and the first data volume threshold. In response to determining that the data volume is greater than the third data volume threshold, determining that the target data cannot be acquired; and presenting an indication of a target data acquisition failure. Exemplarily, the third data volume threshold is 100 million records. When the number of target data records exceeds 100 million, an indication of a target data acquisition failure may be returned, indicating that the data management platform does not support downloading 100 million records.

[0046] In some embodiments of the present disclosure, determining at least one target acquisition mode for target data based on a time period includes determining a first time period threshold. In response to determining that the time period does not exceed the first time period threshold, determining an online acquisition mode or a near-line acquisition mode as the at least one target acquisition mode. Alternatively, in response to determining that the time period exceeds the first time period threshold, determining an offline acquisition mode as the at least one target acquisition mode includes offline acquisition. Exemplarily, the first time period threshold is one year. For example, when a data acquisition request requests data within one year, the online acquisition mode or the near-line acquisition mode may be determined as the target acquisition mode. When the data acquisition request requests data for more than one year, the offline acquisition mode may be determined as the target acquisition mode.

[0047] In some embodiments of the present disclosure, determining at least one target acquisition mode for target data based on corresponding data dimensions includes: determining a first dimension number threshold. In response to determining that the number of dimensions of the corresponding data dimension does not exceed the first dimension number threshold, determining an online acquisition mode or a near-line acquisition mode as at least one target acquisition mode. Or in response to determining that the number of dimensions of the corresponding data dimension exceeds the first dimension number threshold, determining an offline acquisition mode as at least one target acquisition mode. Exemplarily, the first dimension number threshold is, for example, 9. When the number of dimensions of the aggregation corresponding to the data request exceeds 9, the offline acquisition mode is determined as the target acquisition mode. When the number of dimensions of the aggregation corresponding to the data request does not exceed 9, the online acquisition mode or the near-line acquisition mode is determined as the target acquisition mode.

[0048] It should be understood that although the above describes how to determine the target acquisition mode separately for the data volume, the corresponding data dimension and the time period, in the embodiments of the present disclosure, the determination of the target acquisition mode can also be determined comprehensively based on the data volume, the corresponding data dimension and the time period. As an example, for example, the target acquisition mode is first determined by the time period. For example, when the time period is greater than 1 year, the data acquisition request is directly converted to the offline acquisition mode, and the target data is acquired through the offline acquisition mode. After determining that the target acquisition mode can be acquired in an online or near-line acquisition mode through the time period, the target acquisition mode is determined by the dimension aggregated by the data request determined based on the corresponding data dimension. For example, when the aggregated dimension determined based on the corresponding data dimension is greater than 9, the data acquisition request is directly converted to the offline acquisition mode, and the target data is acquired through the offline acquisition mode. After determining that the target acquisition mode can be acquired in an online or near-line acquisition mode through the aggregated dimension, the target acquisition mode is determined by the data volume.

[0049] It should also be understood that even if the target acquisition mode is determined separately by the data volume, the corresponding data dimension and the time period, the determination mode is not limited to the above-described method, but can be various suitable methods.

[0050] Continue to refer Figure 2 At block 230 , the server device 120 obtains target data by routing the data acquisition request to a corresponding acquisition path based on at least one target acquisition mode.

[0051] In an embodiment of the present disclosure, when the target acquisition mode is an online acquisition mode, the data acquisition request can be routed to an online acquisition path based on the target acquisition mode, more specifically to different atomic services, and online data can be acquired through different atomic service online databases.

[0052] In the disclosed embodiments, when the target acquisition mode is the nearline acquisition mode, data acquisition requests can be routed to the nearline acquisition path based on the target acquisition mode. Specifically, a larger amount of data can be acquired by combining the online acquisition path with time-sharing and paging splitting. That is, a larger amount of data can be acquired by splitting the online path with time-sharing and paging splitting.

[0053] In an embodiment of the present disclosure, when the target acquisition mode is an offline acquisition mode, the target data may be acquired by routing a data acquisition request to an offline acquisition path based on the target acquisition mode.

[0054] In some embodiments of the present disclosure, acquiring target data through an offline acquisition path includes: in response to determining that the offline acquisition mode is at least one target acquisition mode, acquiring data dimensions supported by the offline acquisition path; determining, based on the data dimensions supported by the offline acquisition path, that the offline acquisition path supports corresponding data dimensions; in response to determining that the offline acquisition path supports the corresponding data dimensions, splitting a data acquisition request into multiple sub-requests; and acquiring, through the offline acquisition path, a file corresponding to each sub-request, where the file includes at least a portion of the target data.

[0055] In some embodiments of the present disclosure, obtaining the file corresponding to each sub-request through the offline acquisition path includes: estimating the size of the file corresponding to the sub-request; and obtaining the file corresponding to the sub-request in response to the estimated size of the file corresponding to the sub-request not exceeding a set file size threshold.

[0056] In some embodiments of the present disclosure, obtaining the file corresponding to each sub-request through the offline retrieval path further includes: determining that the file corresponding to the sub-request cannot be obtained in response to an estimated size of the file corresponding to the sub-request exceeding a set file size threshold; and presenting an indication that the file corresponding to the sub-request cannot be obtained.

[0057] Figure 4A Schematic diagram of a data request link 400A of a data management platform according to some embodiments of the present disclosure.

[0058] like Figure 4AAs shown, users submit data collection tasks through the GoldenEye Data Collection Center's periodic scheduling. Tasks are executed through the JMQ messaging component. The metrics service makes decisions and routing decisions based on data volume, dimensions, and time period. The online link routes to different atomic services based on metadata, retrieving online data from the online database. The nearline link combines time-sharing and paging based on the online path to obtain larger amounts of data. Hive resources connected to definition-driven production are automatically pushed offline through the back-end caliber to the offline Hive layer to ensure caliber consistency. Tasks that fail metric dimension verification or are denied permission to call are immediately reported as failed to the caller. According to the call chain architecture diagram, the definition-driven production offline link interacts through the JDQ messaging component. Given that there are no particularly high requirements for the query per second (QPS) of offline requests, data processing is serialized, and duplicate execution plans are avoided by checking whether the same task identifier exists in the database. Upon task completion, both nearline and offline tasks are uploaded to a cloud database (such as the Object-as-a-Service (OSS) database), and the metrics service uniformly notifies the caller.

[0059] Figure 4B A flowchart of an asynchronous download process of a data management platform according to some embodiments of the present disclosure is shown.

[0060] When the target data cannot be obtained online based on the data volume, dimension and time period, the target data can be obtained through asynchronous download. Figure 4B As shown, the target data is acquired through nearline or offline acquisition mode, determined by the time period, aggregation dimension, and data volume. When the target data is acquired through nearline acquisition mode, the data request is routed to the nearline path, and a rolling query is performed to acquire a larger amount of data than is available online. When the target data is acquired through offline acquisition mode, the data request is routed to the offline path. The offline path splits the request and routes it to different topic atomic services that support this metric. The request obtains the total number of rows of data for this query for this topic metric. For each topic, if the total data volume is too large, the request is routed to the offline path.

[0061] For example, the offline path routing splitting process is as follows: Data retrieval requests routed to the offline system are split into primary and secondary requests. Dimensions not supported by certain metrics are ignored to support merged queries across different topics. When routed offline, the request is split into secondary requests, ignoring unsupported filter dimensions while retaining the original primary request. Error messages during offline processing, as well as information about unsupported metric dimensions and ignored dimensions, are uniformly processed and returned to the user by the offline system. For example, a user-submitted data retrieval request is considered the primary request; secondary requests are split based on metrics. (Different metrics support different dimensions, and dimensions are ignored based on their supported dimensions. For example, if the requested dimensions are a, b, and c, and the query metrics are m1 and m2, with m1 supporting a and b and m2 supporting b and c, the query is split into two requests: m1(a, b) ignoring dimension c and m2(b, c) ignoring dimension a.) If different metrics support the same dimensions, they are merged into a single request. Requests split and merged according to this rule are subrequests. The main and sub-requests will be passed to the offline server, which will then generate and execute SQL statements based on the main and sub-requests, and return results and error messages.

[0062] In the disclosed embodiment, process 200 further includes: processing the indicator data in a set processing format in response to receiving the indicator data registration request; and registering the processed indicator data with the data management system. Specifically, in the disclosed embodiment, registering the indicator data in a unified processing format provides a unified processing basis for the indicator data service.

[0063] In the embodiment of the present disclosure, process 200 further includes: in response to receiving an indicator data update request, updating the status of the indicator data corresponding to the update request.

[0064] Figure 5 A schematic diagram of a data management and synchronization process 500 of a data management platform according to some embodiments of the present disclosure is shown.

[0065] like Figure 5 As shown in the figure, the data management platform includes a front-end, a back-end, a metadata module, and an indicator service module. Business parties (such as providers or consumers of indicator data) can submit data registration, query, modification, and deletion requests to the data management platform through the front-end, thereby registering, querying, modifying, and deleting indicator data in a database (such as a MySQL database). The metadata module can periodically refresh metadata such as indicators, dimensions, and services to ensure consistency and up-to-dateness. When the metadata module determines that there is a data update, it can broadcast it through a message queue (MQ) to notify related systems, such as changes to metadata related to the indicator service module, to achieve data synchronization.

[0066] In the disclosed embodiment, upon receiving an indicator data registration request, the platform processes the received indicator data according to a predetermined processing format, thereby achieving automated and standardized data integration, ensuring data consistency, and ensuring that indicator data from all sources follows the same indicator definition and caliber. For example, during the indicator definition process, based on the definition and acceleration of Hive (a data warehouse infrastructure), relevant metadata is accumulated to ensure the consistency and accuracy of the indicator caliber.

[0067] The data management and synchronization process implemented by this disclosure can achieve effective management of indicator data, ensure unified data caliber, and improve data processing efficiency.

[0068] Figure 6 FIG. 6 is a schematic structural block diagram of an apparatus 600 for data processing according to certain embodiments of the present disclosure.

[0069] like Figure 6 As shown, the apparatus 600 includes a request determination module 610 configured to determine the data volume, corresponding data dimension, and requested time period of the requested target data in response to receiving a data acquisition request for target data.

[0070] The device 600 also includes a path selection module 620, which is configured to determine at least one target acquisition mode of target data from multiple optional data modes based on at least one of the data volume, the corresponding data dimension and the time period, and the multiple optional data modes correspond to at least different acquisition durations and acquisition paths.

[0071] The apparatus 600 further includes a data acquisition module 630 configured to acquire target data by routing a data acquisition request to a corresponding acquisition path based on at least one target acquisition mode.

[0072] In some embodiments, the path selection module 620 is further configured to: determine a first data volume threshold and a second data volume threshold, wherein the second data volume threshold is greater than the first data volume threshold; in response to determining that the data volume does not exceed the first data volume threshold, determine the online acquisition mode as at least one target acquisition mode; or in response to determining that the data volume exceeds the first data volume threshold but does not exceed the second data volume threshold, determine the near-line acquisition mode as at least one target acquisition mode; or in response to determining that the data volume exceeds the second data volume threshold, determine the offline acquisition mode as at least one target acquisition mode.

[0073] In some embodiments of the present disclosure, the path selection module 620 is further configured to: determine a third data volume threshold, where the third data volume threshold is greater than the second data volume threshold and the first data volume threshold; in response to determining that the data volume is greater than the third data volume threshold, determine that the target data cannot be acquired; and present an indication of the target data acquisition failure.

[0074] In some embodiments of the present disclosure, the path selection module 620 is further configured to determine a first time period threshold. In response to determining that the time period does not exceed the first time period threshold, determine the online acquisition mode or the near-line acquisition mode as at least one target acquisition mode. Alternatively, in response to determining that the time period exceeds the first time period threshold, determine the offline acquisition mode as including offline acquisition in the at least one target acquisition mode.

[0075] In some embodiments of the present disclosure, the path selection module 620 is further configured to determine a first dimension number threshold. In response to determining that the number of dimensions of the corresponding data dimension does not exceed the first dimension number threshold, the online acquisition mode or the near-line acquisition mode is determined as at least one target acquisition mode. Alternatively, in response to determining that the number of dimensions of the corresponding data dimension exceeds the first dimension number threshold, the offline acquisition mode is determined as at least one target acquisition mode.

[0076] In some embodiments of the present disclosure, the data acquisition module 630 is further configured to: in response to determining that the offline acquisition mode is at least one target acquisition mode, obtain data dimensions supported by the offline acquisition path; determine, based on the data dimensions supported by the offline acquisition path, whether the offline acquisition path supports the corresponding data dimensions; in response to determining that the offline acquisition path supports the corresponding data dimensions, split the data acquisition request into multiple sub-requests; and obtain, through the offline acquisition path, a file corresponding to each sub-request, where the file includes at least a portion of the target data.

[0077] In some embodiments of the present disclosure, the data acquisition module 630 is further configured to: in response to determining that the offline acquisition mode is at least one target acquisition mode, obtain data dimensions supported by the offline acquisition path; determine, based on the data dimensions supported by the offline acquisition path, whether the offline acquisition path supports the corresponding data dimensions; in response to determining that the offline acquisition path supports the corresponding data dimensions, split the data acquisition request into multiple sub-requests; and obtain, through the offline acquisition path, a file corresponding to each sub-request, where the file includes at least a portion of the target data.

[0078] In some embodiments of the present disclosure, the data acquisition module 630 is further configured to estimate the size of the file corresponding to the sub-request, and in response to the estimated size of the file corresponding to the sub-request not exceeding a set file size threshold, acquire the file corresponding to the sub-request.

[0079] In some embodiments of the present disclosure, the data acquisition module 630 is further configured to: in response to the estimated size of the file corresponding to the sub-request exceeding a set file size threshold, determine that the file corresponding to the sub-request cannot be acquired, and present an indication that the file corresponding to the sub-request cannot be acquired.

[0080] In some embodiments of the present disclosure, the apparatus 600 further includes: a registration module configured to process the indicator data in a set processing format in response to receiving an indicator data registration request; and register the processed indicator data in a data management system.

[0081] In some embodiments of the present disclosure, the apparatus 600 further includes: an updating module configured to, in response to receiving an indicator data update request, update the status of the indicator data corresponding to the update request.

[0082] The units and / or modules included in the device 600 can be implemented in various ways, including software, hardware, firmware, or any combination thereof. In some embodiments, one or more units and / or modules can be implemented using software and / or firmware, such as machine executable instructions stored on a storage medium. In addition to or as an alternative to machine executable instructions, some or all of the units and / or modules in the device 600 can be implemented at least in part by one or more hardware logic components. By way of example and not limitation, exemplary types of hardware logic components that can be used include field programmable gate arrays (FPGAs), application specific integrated circuits (ASICs), application specific standard products (ASSPs), systems on chip (SOCs), complex programmable logic devices (CPLDs), and the like.

[0083] It should be understood that one or more steps in the above method can be performed by a suitable electronic device or combination of electronic devices. Such an electronic device or combination of electronic devices may include, for example, Figure 1 The electronic device 110 in.

[0084] Figure 7 1 shows a block diagram of an electronic device 700 in which one or more embodiments of the present disclosure may be implemented. Figure 7 The illustrated electronic device 700 is merely exemplary and should not be construed as limiting the functionality and scope of the embodiments described herein. Figure 7 The electronic device 700 shown can be used to implement Figure 1 electronic device 110 or Figure 6 device 600.

[0085] like Figure 7As shown, electronic device 700 is in the form of a general electronic device. Components of electronic device 700 may include, but are not limited to, one or more processors or processing units 710, memory 720, storage device 730, one or more communication units 740, one or more input devices 750, and one or more output devices 760. Processing unit 710 may be a real or virtual processor and is capable of performing various processes according to programs stored in memory 720. In a multi-processor system, multiple processing units execute computer-executable instructions in parallel to increase the parallel processing capabilities of electronic device 700.

[0086] The electronic device 700 typically includes a plurality of computer storage media. Such media can be any available media accessible to the electronic device 700, including but not limited to volatile and non-volatile media, removable and non-removable media. The memory 720 can be a volatile memory (e.g., registers, cache, random access memory (RAM)), a non-volatile memory (e.g., read-only memory (ROM), electrically erasable programmable read-only memory (EEPROM), flash memory) or some combination thereof. The storage device 730 can be a removable or non-removable medium and can include a machine-readable medium, such as a flash drive, a disk or any other medium, which can be used to store information and / or data and can be accessed within the electronic device 700.

[0087] The electronic device 700 may further include additional removable / non-removable, volatile / non-volatile storage media. Figure 7 As shown in FIG, a magnetic disk drive for reading from or writing to a removable, non-volatile magnetic disk (e.g., a "floppy disk") and an optical disk drive for reading from or writing to a removable, non-volatile optical disk may be provided. In these cases, each drive may be connected to a bus (not shown) by one or more data media interfaces. The memory 720 may include a computer program product 725 having one or more program modules configured to perform the various methods or actions of various embodiments of the present disclosure.

[0088] The communication unit 740 enables communication with other electronic devices via a communication medium. Additionally, the functions of the components of the electronic device 700 can be implemented as a single computing cluster or multiple computing machines that can communicate via a communication connection. Thus, the electronic device 700 can operate in a networked environment using a logical connection with one or more other servers, a network personal computer (PC), or another network node.

[0089] Input device 750 may be one or more input devices, such as a mouse, keyboard, or trackball. Output device 760 may be one or more output devices, such as a display, a speaker, or a printer. Electronic device 700 may also communicate with one or more external devices (not shown) via communication unit 740 as needed, such as storage devices, display devices, or the like, with one or more devices that allow a user to interact with electronic device 700, or with any device that allows electronic device 700 to communicate with one or more other electronic devices (e.g., a network card, a modem, etc.). Such communication may be performed via an input / output (I / O) interface (not shown).

[0090] According to an exemplary implementation of the present disclosure, a computer-readable storage medium is provided, on which computer-executable instructions are stored, wherein the computer-executable instructions are executed by a processor to implement the method described above. According to an exemplary implementation of the present disclosure, a computer program product is also provided, which is tangibly stored on a non-transitory computer-readable medium and includes computer-executable instructions, and the computer-executable instructions are executed by a processor to implement the method described above.

[0091] Various aspects of the present disclosure are described herein with reference to flowcharts and / or block diagrams of methods, apparatuses, devices, and computer program products implemented according to the present disclosure. It should be understood that each block of the flowcharts and / or block diagrams, and combinations of blocks in the flowcharts and / or block diagrams, can be implemented by computer-readable program instructions.

[0092] These computer-readable program instructions can be provided to a processing unit of a general-purpose computer, a special-purpose computer, or other programmable data processing device, thereby producing a machine, such that when these instructions are executed by the processing unit of the computer or other programmable data processing device, a device is generated that implements the functions / actions specified in one or more blocks in the flowchart and / or block diagram. These computer-readable program instructions can also be stored in a computer-readable storage medium, where these instructions cause the computer, programmable data processing device, and / or other device to operate in a specific manner. Thus, the computer-readable medium storing the instructions comprises an article of manufacture that includes instructions for implementing various aspects of the functions / actions specified in one or more blocks in the flowchart and / or block diagram.

[0093] Computer-readable program instructions can be loaded onto a computer, other programmable data processing apparatus, or other device so that a series of operational steps are performed on the computer, other programmable data processing apparatus, or other device to produce a computer-implemented process, thereby causing the instructions executed on the computer, other programmable data processing apparatus, or other device to implement the functions / actions specified in one or more boxes in the flowchart and / or block diagram.

[0094] The flow charts and block diagrams in the accompanying drawings show the possible architecture, functions and operations of the systems, methods and computer program products according to multiple implementations of the present disclosure. In this regard, each box in the flow chart or block diagram can represent a part for a module, program segment or instruction, and a part for a module, program segment or instruction comprises one or more executable instructions for realizing the logical function of the specification. In some alternative implementations, the functions marked in the box can also occur in a sequence different from that marked in the accompanying drawings. For example, two continuous boxes can actually be executed substantially in parallel, and they can sometimes be executed in the opposite order, depending on the functions involved. It should also be noted that each box in the block diagram and / or flow chart, and the combination of the boxes in the block diagram and / or flow chart can be realized by a special hardware-based system that performs the function or action of the specification, or can be realized by a combination of special hardware and computer instructions.

[0095] While various implementations of the present disclosure have been described above, the foregoing description is intended to be illustrative, not exhaustive, and not limited to the disclosed implementations. Many modifications and variations will be apparent to those skilled in the art without departing from the scope and spirit of the described implementations. The terminology used herein is selected to best explain the principles of the implementations, their practical applications, or improvements to existing technologies, or to enable others skilled in the art to understand the various implementations disclosed herein.

Claims

1. A method for data processing, comprising: In response to receiving a data acquisition request for target data, determining the requested data volume, corresponding data dimensions, and a requested time period of the target data; determining, based on at least one of the data volume, the corresponding data dimension, and the time period, at least one target acquisition mode for the target data from a plurality of selectable data modes, the plurality of selectable data modes corresponding to at least different acquisition durations and acquisition paths; as well as The target data is acquired by routing the data acquisition request to a corresponding acquisition path based on the at least one target acquisition mode.

2. The method according to claim 1, wherein determining at least one target acquisition mode of the target data based on the data volume comprises: determining a first data volume threshold and a second data volume threshold, wherein the second data volume threshold is greater than the first data volume threshold; In response to determining that the data amount does not exceed a first data amount threshold, determining an online acquisition mode as the at least one target acquisition mode; or In response to determining that the data volume exceeds a first data volume threshold and does not exceed a second data volume threshold, determining the at least one target acquisition mode in a near-line acquisition mode; or In response to determining that the data amount exceeds a second data amount threshold, an offline acquisition mode is determined as the at least one target acquisition mode.

3. The method according to claim 2, further comprising: determining a third data volume threshold, wherein the third data volume threshold is greater than the second data volume threshold and the first data volume threshold; In response to determining that the data amount is greater than a third data amount threshold, determining that the target data cannot be acquired; as well as An indication that the target data acquisition failed is presented.

4. The method of claim 1 , wherein determining at least one target acquisition mode of the target data based on the time period comprises: determining a first time period threshold; In response to determining that the time period does not exceed the first time period threshold, determining an online acquisition mode or a near-line acquisition mode as the at least one target acquisition mode; or In response to determining that the time period exceeds the first time period threshold, determining an offline acquisition mode as the at least one target acquisition mode includes offline acquisition.

5. The method according to claim 1 , wherein determining at least one target acquisition mode of the target data based on the corresponding data dimension comprises: Determine a first dimension number threshold; In response to determining that the number of dimensions of the corresponding data dimension does not exceed the first dimension number threshold, determining an online acquisition mode or a near-line acquisition mode as the at least one target acquisition mode; or In response to determining that the dimension number of the corresponding data dimension exceeds the first dimension number threshold, an offline acquisition mode is determined as the at least one target acquisition mode.

6. The method according to claim 1, further comprising: In response to determining the offline acquisition mode as the at least one target acquisition mode, acquiring data dimensions supported by the offline acquisition path; Based on the data dimension supported by the offline acquisition path, determining that the offline acquisition path supports the corresponding data dimension; In response to determining that the offline acquisition path supports the corresponding data dimension, splitting the data acquisition request into multiple sub-requests, and The file corresponding to each sub-request is obtained through the offline acquisition path, where the file includes at least a portion of the target data.

7. The method according to claim 6, wherein obtaining the file corresponding to each sub-request through the offline acquisition path comprises: Estimating the size of the file corresponding to the sub-request; as well as In response to the estimated size of the file corresponding to the sub-request not exceeding a set file size threshold, the file corresponding to the sub-request is obtained.

8. The method according to claim 7, further comprising: In response to the estimated size of the file corresponding to the sub-request exceeding the set file size threshold, determining that the file corresponding to the sub-request cannot be obtained; as well as An indication is presented that the file corresponding to the subrequest cannot be obtained.

9. The method according to any one of claims 1 to 8, further comprising: In response to receiving the indicator data registration request, processing the indicator data in a set processing format; as well as The processed indicator data is registered in a data management system.

10. The method according to any one of claims 1 to 8, further comprising: In response to receiving the indicator data update request, the status of the indicator data corresponding to the update request is updated.

11. A device comprising: a request determination module, configured to, in response to receiving a data acquisition request for target data, determine the requested data volume, corresponding data dimensions, and a requested time period of the target data; a path selection module configured to determine, based on at least one of the data volume, the corresponding data dimension, and the time period, at least one target acquisition mode for the target data from a plurality of selectable data modes, the plurality of selectable data modes corresponding to at least different acquisition durations and acquisition paths; as well as The data acquisition module is configured to acquire the target data by routing the data acquisition request to a corresponding acquisition path based on the at least one target acquisition mode.

12. An electronic device comprising: at least one processing unit; as well as At least one memory coupled to the at least one processing unit and storing instructions for execution by the at least one processing unit, the instructions causing the electronic device to perform the method according to any one of claims 1 to 10 when executed by the at least one processing unit.

13. A computer-readable storage medium having a computer program stored thereon, wherein the computer program can be executed by a processor to implement the method according to any one of claims 1 to 10.

14. A computer program product tangibly stored in a computer storage medium and comprising computer executable instructions which, when executed by a device, cause the device to perform the method according to any one of claims 1 to 10.

Images