Data processing method, apparatus, device, and medium

Abstract

The present disclosure provides a data processing method and device, equipment and medium, relates to the technical field of artificial intelligence, and can be applied to various scenes such as cloud technology, artificial intelligence, intelligent transportation and auxiliary driving. The method comprises: performing splitting processing on a query time range based on different time dimensions to obtain at least one time slice; for each time slice, obtaining full-volume business data corresponding to the time slice from a first storage area based on query information, wherein the full-volume business data comprises offline data that has been subjected to data partitioning processing and loaded into the first storage area; obtaining full-volume partition time corresponding to the full-volume business data, and obtaining incremental business data from a second storage area based on the query information; and generating a query result based on the full-volume business data and the incremental business data. Thus, the dependence on the query performance of downstream services is reduced, a large number of query requirements can be met, and the data query efficiency and overall system performance are improved.

Description

TECHNICAL FIELD

[0001] The present disclosure relates to the field of computers, and particularly relates to a data processing method and device, equipment and medium. BACKGROUND

[0002] In an automated advertising market watch system, there are many scenarios of concurrently calling remote services to query advertising report data. The traditional calling method is to query remote data by calling a network request, but when the performance of downstream data providing services reaches a bottleneck, it cannot meet the large query demand of the market watch system, greatly affecting the data query efficiency and overall performance of the current system. SUMMARY

[0003] The present disclosure provides a data processing method, device, equipment and medium to solve at least one technical problem in the prior art.

[0004] In one aspect, the present disclosure provides a data processing method, comprising:

[0005] obtaining a business query request, the business query request comprising a query time range and query information;

[0006] performing slicing processing on the query time range based on different time dimensions to obtain at least one time slice;

[0007] for each time slice, obtaining full-volume business data corresponding to the time slice from a first storage area based on the query information, the full-volume business data comprising offline data that has been processed by data partitioning and loaded into the first storage area;

[0008] obtaining a full-volume partition time corresponding to the full-volume business data, and obtaining incremental business data from a second storage area based on the query information, the full-volume partition time representing the generation time of full-volume data partitioning of full-volume business data, the incremental business data comprising offline data in which the query time in the query time range is later than the full-volume partition time; the second storage area and the first storage area are both located in a local device;

[0009] generating a query result based on the full-volume business data and the incremental business data.

[0010] In another aspect, a data processing device is also provided, comprising:

[0011] a first obtaining module configured to obtain a business query request, the business query request comprising a query time range and query information;

[0012] a slicing module configured to perform slicing processing on the query time range based on different time dimensions to obtain at least one time slice;

[0013] The second obtaining module is configured to, for each time slice, obtain full-volume service data corresponding to the time slice from the first storage area based on the query information, the full-volume service data including data that has been processed by data partitioning and loaded into the first storage area;

[0014] The third obtaining module is configured to obtain a full-volume partition time corresponding to the full-volume service data, and obtain incremental service data from the second storage area based on the query information, the full-volume partition time representing a generation time of a full-volume data partition of the full-volume service data, the incremental service data including data corresponding to a query time later than the full-volume partition time in the query time range; the second storage area and the first storage area are both located in a local device.

[0015] The first generating module is configured to generate a query result based on the full-volume service data and the incremental service data.

[0016] In another aspect, an electronic device is provided, which includes a processor and a memory, the memory storing at least one instruction or at least one program, the at least one instruction or the at least one program being loaded and executed by the processor to implement any of the above-described methods.

[0017] In another aspect, a computer-readable storage medium is provided, which stores at least one instruction or at least one program, the at least one instruction or the at least one program being loaded and executed by a processor to implement any of the above-described methods.

[0018] In another aspect, a computer program product or computer program is provided, which includes computer instructions stored in a computer-readable storage medium. A processor of a computer device reads the computer instructions from the computer-readable storage medium, and the processor executes the computer instructions to cause the computer device to perform any of the above-described methods.

[0019] The data processing method, device, equipment and medium provided by the present disclosure have the following technical effects:

[0020] The embodiment of the present disclosure obtains at least one time slice by performing splitting processing on the query time range based on different time dimensions; for each time slice, full-volume business data corresponding to the time slice is obtained from a first storage area based on the query information, the full-volume business data including offline data that has been processed by data partitioning and loaded into the first storage area; full-volume partition time corresponding to the full-volume business data is obtained, and incremental business data is obtained from a second storage area based on the query information, the full-volume partition time representing the generation time of full-volume data partition of the full-volume business data, and the incremental business data including offline data in the query time range whose query time is later than the full-volume partition time; and a query result is generated based on the full-volume business data and the incremental business data. Thus, by performing splitting processing on the query time range and obtaining offline full-volume business data and incremental business data from the first storage area and the second storage area respectively for each time slice, the dependence on the query performance of downstream services is reduced, a large number of query requirements can be met, and the data query efficiency and the overall system performance are improved. BRIEF DESCRIPTION OF DRAWINGS

[0021] In order to more clearly illustrate the technical solutions in the embodiments of the present disclosure or the prior art and the advantages thereof, a brief introduction will be given to the drawings needed in the embodiments or the prior art description. Obviously, the drawings in the following description are only some embodiments of the present disclosure, and for those skilled in the art, other drawings can also be obtained without creative labor based on these drawings.

[0022] Figure 1 is a schematic diagram of an application environment of a data processing method provided by the embodiment of the present disclosure;

[0023] Figure 2 is a flowchart of a data processing method provided by the embodiment of the present disclosure;

[0024] Figure 3 is a partial flowchart of a data processing method provided by the embodiment of the present disclosure;

[0025] Figure 4 is a partial flowchart of a data processing method provided by the embodiment of the present disclosure;

[0026] Figure 5 is a partial flowchart of a data processing method provided by the embodiment of the present disclosure;

[0027] Figure 6 is a flowchart of a data processing method provided by the embodiment of the present disclosure;

[0028] Figure 7 is a structural block diagram of a data processing apparatus provided by the embodiment of the present disclosure;

[0029] Figure 8 Fig. 1 is a hardware structure schematic diagram of an apparatus for implementing the method provided by the embodiments of the present disclosure. DETAILED DESCRIPTION

[0030] In order to enable persons skilled in the art to better understand the present disclosure scheme, the technical solutions in the embodiments of the present disclosure will be clearly and completely described below in conjunction with the drawings in the embodiments of the present disclosure. Obviously, the described embodiments are only a part of the embodiments of the present disclosure, not all the embodiments. Based on the embodiments in the present disclosure, all other embodiments obtained by persons skilled in the art without creative labor should be within the scope of protection of the present disclosure.

[0031] In order to make the purpose, technical solutions and advantages of the present disclosure clearer, the embodiments of the present disclosure will be further described in detail below in conjunction with the drawings.

[0032] In order to facilitate the understanding of the technical solutions and the technical effects generated by the above-mentioned embodiments of the present disclosure, the terms involved in the embodiments of the present disclosure are briefly introduced:

[0033] Cloud technology is a general term for network technology, information technology, integration technology, management platform technology, application technology and other technologies applied based on cloud computing business model, which can form a resource pool, and be used on demand, flexibly and conveniently. Cloud computing technology will become an important support. The background service of the technical network system needs a large amount of computing and storage resources, such as video websites, picture websites and more portal websites. With the high development and application of the Internet industry, every item in the future may have its own identification mark and needs to be transmitted to the background system for logical processing. Different levels of data will be processed separately, and various industry data needs strong system support, which can be realized through cloud computing.

[0034] Cloud security is a general term for security software, hardware, users, institutions, and security cloud platforms based on cloud computing business model applications. Cloud security integrates emerging technologies and concepts such as parallel processing, grid computing, and unknown virus behavior judgment. Through the abnormal monitoring of software behavior in the network by a large number of clients in a network, the latest information of Internet viruses and malicious programs is obtained and sent to the server for automatic analysis and processing, and the virus and Trojan solution is distributed to each client.

[0035] The main research directions of cloud security include: 1. Cloud computing security, mainly studying how to protect the security of the cloud itself and various applications on the cloud, including cloud computer system security, secure storage and isolation of user data, user access authentication, information transmission security, network attack protection, compliance audit, etc.; 2. Cloudification of security infrastructure, mainly studying how to build and integrate security infrastructure resources using cloud computing, optimize security protection mechanisms, including building a large-scale security event information collection and processing platform through cloud computing technology, realizing the collection and correlation analysis of massive information, and improving the network security event control ability and risk control ability; 3. Cloud security services, mainly studying various security services provided by cloud computing platforms for users, such as antivirus services, etc.

[0036] Cloud storage is a new concept extended and developed on the basis of the concept of cloud computing. A distributed cloud storage system (hereinafter referred to as a storage system) refers to a storage system that collects and cooperatively works together a large number of various types of storage devices (storage devices are also referred to as storage nodes) in a network through cluster application, grid technology, and distributed storage file system functions, and provides data storage and business access functions to the outside.

[0037] At present, the storage method of the storage system is: creating a logical volume, and allocating a physical storage space to each logical volume when creating the logical volume. The physical storage space may be a disk composed of a certain storage device or several storage devices. The client stores data on a certain logical volume, that is, stores data on the file system. The file system divides the data into many parts, each part being an object. The object contains not only data but also additional information such as data identification (IDentity). The file system writes each object to the physical storage space of the logical volume, and records the storage location information of each object, so that when the client requests to access the data, the file system can access the data according to the storage location information of each object.

[0038] The process of allocating a physical storage space to a logical volume by the storage system is as follows: according to the capacity estimation of the objects stored in the logical volume (the estimation often has a large margin relative to the actual capacity of the objects to be stored) and the group of redundant arrays of independent disks (RAID), the physical storage space is pre-divided into sections. A logical volume can be understood as a section, so that the logical volume is allocated a physical storage space.

[0039] In short, a database can be regarded as an electronic file cabinet, i.e., a place for storing electronic files, and a user can add, query, update, delete, etc. data in the files. The so-called "database" is a collection of data stored together in a certain way, shared by multiple users, with as little redundancy as possible, and independent of application programs.

[0040] A database management system (DBMS) is a computer software system designed to manage databases, generally with basic functions such as storage, interception, security, backup, etc. Database management systems can be classified according to the database model they support, such as relational, XML (Extensible Markup Language), or according to the type of computer they support, such as server clusters, mobile phones, or according to the query language they use, such as SQL (Structured Query Language), XQuery, or according to the performance focus, such as maximum size, highest speed, or other classification methods. Regardless of the classification method used, some DBMSs can cross categories, such as supporting multiple query languages at the same time. The scheme provided by the embodiments of the present disclosure relates to cloud technology and other technologies, which are specifically described as follows.

[0041] The data processing method provided by the present disclosure can be applied to an application environment as shown in Figure 1 As shown in Figure 1 The hardware environment can at least include a terminal 10, a server 20, and a server 30.

[0042] For example, in the context of an advertisement watchlist, the terminal 10 can be a device of an advertisement watchlist personnel, the server 20 can be a server corresponding to an advertisement watchlist system, and the server 30 can be a downstream server providing service data.

[0043] The terminal 10 described above can be a smart phone, a tablet computer, a notebook computer, a desktop computer, a smart speaker, a smart watch, a vehicle-mounted terminal, a smart television, etc., but is not limited thereto.

[0044] The server 20 and the server 30 described above can be a stand-alone physical server, a server cluster or a distributed system composed of multiple physical servers, or a cloud server providing cloud computing services. The terminal and the server can be directly or indirectly connected through wired or wireless communication, which is not limited in the present disclosure. It should be noted that the server 20 and the server 30 described above can be implemented as a cloud server in the cloud.

[0045] In some embodiments, servers 20 and 30 can also be implemented as nodes in a blockchain system. Blockchain is a novel application model of computer technologies such as distributed data storage, peer-to-peer transmission, consensus mechanisms, and cryptographic algorithms. Essentially, a blockchain is a decentralized database, a chain of data blocks linked using cryptographic methods. Each data block contains information about a batch of network transactions, used to verify the validity of the information (anti-counterfeiting) and generate the next block. A blockchain can include a blockchain underlying platform, a platform product service layer, and an application service layer.

[0046] It should be noted that in practical applications, the above data processing methods can also be implemented on a server, or jointly by a terminal and a server.

[0047] Of course, the methods provided in this disclosure are not limited to... Figure 1 The hardware environment shown can also be used in other possible hardware environments, and this disclosure does not limit the scope of the embodiments. Figure 1 The functions that each device in the hardware environment shown can perform will be described in subsequent method embodiments, and will not be elaborated on here.

[0048] Figure 2 This is a flowchart illustrating a data processing method provided in an embodiment of this disclosure. This disclosure provides the operational steps of the method described in the embodiments or flowchart, but based on conventional or non-inventive labor, more or fewer operational steps may be included. The order of steps listed in the embodiments is merely one possible execution order among many steps and does not represent a unique execution order. The executing entity of this data processing method can be the data processing device provided in the embodiments of this disclosure, or a server integrating the data processing device, wherein the data processing device can be implemented in hardware or software. Taking the executing entity as described above... Figure 1 Let's take the server in the example of this as an illustration. Figure 2 As shown, the method may include:

[0049] S201: Obtain a business query request, which includes a query time range and query information.

[0050] The business query request can be triggered by a business query control to retrieve corresponding business data. This business data refers to a large amount of data that needs to be obtained in a short period, and may include, but is not limited to, at least one type of data such as report data, order data, and billing data. Taking an advertising monitoring scenario as an example, the business data may include report data. For instance, this report data may include at least one type of report data such as ad impressions, clicks, and conversions.

[0051] The query time range refers to a time interval formed by a request query start time and a request query end time of the business data to be obtained. For example, the query time range can be a time interval of "2021-07-01~2021-09-03", or a time interval of "2021-07-01~to date". For another example, the query time range can be the last n months, the last m weeks, or the last t days, and the like, where n, m, and t are positive integers.

[0052] The query information is used to indicate the business data corresponding to the target content to be obtained. For example, in the advertising market scenario, the target content can include an advertiser a, an exposure report, a click report, a conversion report, an order, a bill, and the like, and the query information can include a data identifier corresponding to the target content, such as a data id, which is used to uniquely determine the business data and can be formed by at least one string such as a number, a letter, and the like. For example, for the case where the business data is report data, the data identifier can be a report name; for the case where the business data is order data, the data identifier can be an order number; for the case where the business data is a bill data file, the data identifier can be a file name, and the like.

[0053] S203: performing a splitting processing based on different time dimensions on the query time range to obtain at least one time slice.

[0054] The time dimension can include a month dimension and a day dimension. For example, for data with a query time range of "2021-07-01~2021-09-03", after the splitting processing based on different time dimensions, a time slice of 202107 month, a time slice of 202108 month, a time slice of 20210901 day, a time slice of 20210902 day, and a time slice of 20210903 day can be obtained.

[0055] It should be noted that the time dimension included in the time slice can be more or less, for example, the time dimension can also include a year dimension, a minute dimension, or a second dimension, and the like; for another example, for a query time range of the last 3 days, the query time range can be split into 3 time slices containing only a day dimension.

[0056] S205: For each time slice, full amount business data corresponding to the time slice is obtained from the first storage area based on the query information, and the full amount business data includes offline data that has been processed by data partitioning and loaded into the first storage area.

[0057] The first storage area is a storage location for storing full-volume business data, which can be a storage location corresponding to a local device, such as a local memory or the like. The full-volume business data can be stored in the first storage area according to time-based slicing of data time to achieve partitioned storage. The full-volume business data can be stored by a Map structure, and in this case, the first storage area can be a full-volume data Map. The Map structure is a dictionary data structure, that is, a hash structure for storing non-repeating key values.

[0058] The full-volume business data corresponding to each time slice is different, and the storage state of the full-volume business data is different. For each time slice, the full-volume business data corresponding to each time slice is obtained from the first storage area according to the data id corresponding to the query information. That is, for data conversion in the whole month range to query the month data, other dates are queried using day data. The full-volume business data includes offline data that has been processed by data partitioning and loaded into the first storage area, that is, the obtained full-volume business data is offline data stored in the local device, and is not online data called through a remote data interface. The data partitioning refers to dividing business data or its constituent elements into different independent parts. The data partitioning here can refer to data partitioning that reaches a preset number, data partitioning that reaches a preset time, and the like. Through data partitioning, fast access to business data can be achieved, and the overall system can be improved.

[0059] In an optional embodiment, the obtaining of the full-volume business data corresponding to each time slice from the first storage area based on the query information can include:

[0060] In a case where it is determined that the full-volume business data corresponding to the time slice and the query information has been loaded offline to the first storage area, the full-volume business data corresponding to the time slice is obtained from the first storage area;

[0061] In a case where it is determined that the full-volume business data corresponding to the time slice and the query information has not been loaded offline to the first storage area, the target full-volume data corresponding to the time slice is loaded to the first storage area, and the target full-volume data is obtained from the first storage area synchronously, and the target full-volume data is taken as the full-volume business data; the target full-volume data is offline data corresponding to a maximum boundary time close to the query time range in full-volume partition time.

[0062] Wherein, the full partition time can be related to the current query time. For example, if a full business data of the previous day is generated at a certain time of the day (e.g. 7 am). When querying at 8 am on October 10, the full partition time of the full business data in the first storage area is October 9, that is, the full data in the first storage area is the latest full partition time. When querying at 6 am on October 10, since the latest business data in the first storage area has not been generated, that is, the full data in the first storage area is not the latest full partition time, at this time, the full partition time of the full business data in the first storage area can be the previous day, that is, October 8.

[0063] For the time interval of the time query range "2021-07-01~2021-09-03", the maximum boundary time is "2021-09-03". The target full data is the data corresponding to the full partition time close to the maximum boundary time in the query time range, that is, the target full data is the data corresponding to the latest full partition time.

[0064] In actual application, from the full data Map (the first storage area) of the local device, it is queried whether the full business data corresponding to the data time and the query information exists, that is, it is determined whether the full business data corresponding to the time slice and the query information has been offline loaded to the first storage area, which can be divided into the following cases:

[0065] 1) If the full data Map exists the full business data, that is, in the case of determining that the full business data corresponding to the time slice and the query information has been offline loaded to the first storage area, and the full partition time of the full business data is the latest full partition time, the stored full business data is obtained from the full data Map and used.

[0066] 2) If the full data Map exists the full business data, that is, in the case of determining that the full business data corresponding to the time slice and the query information has been offline loaded to the first storage area, but the full partition time of the full business data is not the latest full partition time, the stored full business data is obtained from the full data Map, and the latest full business data is constituted and used according to the real-time obtained incremental business data, and the latest full partition full business data is loaded from the target storage location (such as a database DB) to the full data Map (the first storage area) asynchronously.

[0067] Continuing the above example, at 6:00 am on October 10, the latest business data in the first storage area has not yet been generated, that is, the full data in the first storage area is not the latest full partition time, and the full partition time in the first storage area is October 9. After obtaining the full business data in the first storage area (full partition time is October 9), the latest full business data is obtained by combining the obtained incremental business data on October 9.

[0068] 3) If the full data Map does not exist for the full business data, that is, it is determined that the full business data corresponding to the time slice and the query information is not offline loaded to the first storage area, the target full data corresponding to the time slice can be obtained from the target storage location (for example, a database DB), the target full data is data corresponding to a time close to the maximum boundary time in the query time range, that is, the latest full partition data is obtained, and the target full data is loaded to the first storage area, and the target full data obtained from the first storage area is synchronized, the loaded target full data is used as the full business data.

[0069] S207: Obtain the full partition time corresponding to the full business data, and obtain incremental business data based on the query information from the second storage area, the incremental business data including offline data corresponding to a query time later than the full partition time in the query time range.

[0070] Wherein, the second storage area and the first storage area are located in the local device. The full partition time represents the generation time of the full data partition of the full business data.

[0071] Optionally, after obtaining the full business data, the incremental business data corresponding to the query information is obtained from the second storage area according to the full partition time corresponding to the full business data, that is, the offline data corresponding to a query time later than the full partition time in the query time range is obtained.

[0072] For example, if the query time range is data from October 9 to October 10, the full business data with a full partition time of October 9 is obtained, and then the incremental business data with a query time later than the full partition time is obtained, that is, the incremental business data on October 10 is obtained, and if the incremental business data is obtained, the incremental business data is obtained.

[0073] S209: Based on the full business data and the incremental business data, a query result is generated.

[0074] Optionally, the obtained full business data and incremental business data are sorted and calculated according to the business query request, and a query result is returned.

[0075] The embodiments of the present disclosure perform splitting processing on the query time range based on different time dimensions to obtain at least one time slice; for each time slice, full-volume business data corresponding to the time slice is obtained from a first storage area based on the query information, the full-volume business data including offline data that has been processed by data partitioning and loaded into the first storage area; a full-volume partition time corresponding to the full-volume business data is obtained, and incremental business data is obtained from a second storage area based on the query information, the full-volume partition time representing a generation time of a full-volume data partition of the full-volume business data, and the incremental business data including offline data corresponding to a query time point in the query time range that is later than the full-volume partition time; and a query result is generated based on the full-volume business data and the incremental business data. Thus, by performing splitting processing on the query time range and obtaining offline full-volume business data and incremental business data from the first storage area and the second storage area respectively for each time slice, the dependence on the query performance of downstream services is reduced, a large number of query requirements can be met, and the data query efficiency and the overall system performance are improved.

[0076] In addition, by storing the loaded full-volume business data and incremental business data in the first storage area and the second storage area respectively, when the incremental business data is loaded, the incremental business data only needs to be accumulated to the corresponding incremental data in the second storage area, without the need to accumulate the incremental data to the full-volume business data, so that a large amount of full-volume business data is avoided from being loaded into the local device at one time, the memory demand of the local device is reduced, and the data processing efficiency is improved.

[0077] In an optional implementation manner, as shown in Figure 3 The method further includes:

[0078] S301: In a case where it is detected that the preset offline data acquisition condition is not met, an online service query interface is invoked, and target business data is obtained based on the query information.

[0079] The preset offline data acquisition condition includes at least one of the following: the offline query function is enabled, the offline business data is normally loaded, and a consistency detection result corresponding to the offline business data meets a preset condition.

[0080] Optionally, the S301 step can include at least the following cases:

[0081] a. In a case where a business query request is obtained, it can be detected whether the offline query function of the system is enabled. If the detection result is that the offline query function is enabled, it is determined that the preset offline data acquisition condition is met, and the subsequent steps S203 and the following steps are continued to be executed. If the detection result is that the offline query function is not enabled, it is determined that the preset offline data acquisition condition is not met, and the online service query interface is invoked, that is, the target business data is directly obtained by accessing the downstream interface and based on the query information through a remote network request.

[0082] b、In the case of obtaining a service query request or in the process of executing the offline data acquisition, whether the offline service data is delayed can be detected by the online checking program. If the detection result is that there is no delay, that is, the data is normally loaded, the service data in the first storage area and the second storage area can be used to determine that the preset offline data acquisition condition is met, and step S203 and subsequent steps are continued. If the detection result is that the data is delayed, that is, the data is not normally loaded, it is determined that the preset offline data acquisition condition is not met, the acquisition of offline data is stopped, and the online service query interface is called, that is, the target service data is obtained by directly accessing the downstream interface based on the query information through the remote network request.

[0083] c、The same service query request can be subjected to data consistency checking, that is, the query result is obtained by calling the remote service query interface and by acquiring offline service data, and the consistency of the results is checked. If the checking result is that the query results of the two are consistent, it is determined that the preset offline data acquisition condition is met, otherwise, it is determined that the preset offline data acquisition condition is not met, and the online service query interface is called, that is, the target service data is obtained by directly accessing the downstream interface based on the query information through the remote network request. By performing data consistency checking, the accuracy of the loaded service data in the system is ensured while the data acquisition efficiency is ensured, and the reliability of the system is improved.

[0084] Optionally, the data checking process can include two stages: 1) After the incremental service data loaded by the timing task is completed, the loaded part of the service data is sampled and checked to check whether the newly loaded service data is consistent. 2) When the service query request is obtained, the request is sampled and checked at a certain proportion.

[0085] Optionally, after the checking is completed, the checking result can be reported to the log system for data analysis. At the same time, the checking result can also be reported to the monitoring platform to alarm abnormal fluctuations, to master the system state in real time and to ensure stability.

[0086] S303: Generating a query result based on the target service data.

[0087] Optionally, after the target service data is obtained, the target service data can be arranged and calculated to return a query result.

[0088] The above embodiment realizes the data query function combining offline and online by calling the online service query interface when it is detected that the preset offline data acquisition condition is not met, generating a query result based on the target service data obtained based on the query information, and improves the stability and reliability of data query.

[0089] In an optional implementation, as shown in Figure 4 The method further includes:

[0090] S401: Obtain offline service data;

[0091] S403: Perform data slicing processing on the offline service data based on different time dimensions to obtain at least one historical data slice;

[0092] S405: Perform data aggregation on each historical data slice respectively, and save the aggregated offline service data to a target storage location.

[0093] Optionally, an offline task thread can be invoked to obtain offline service data provided by a downstream service. Then, the offline service data is subjected to data slicing processing based on different time dimensions, which can include day dimension and minute dimension. For example, for full volume service data, the full volume service data can be in day dimension, and a full volume service data containing data of yesterday can be generated every day. Each full volume service data can include multiple service data, and each service data has its corresponding data identifier (e.g., data ID) for uniquely determining the service data, which can be composed of at least one string such as number, letter, etc. Then, the data within the query range is aggregated into day data and month data according to the data identifier (e.g., data ID), where the day data can be the most recent 31 days, and the month data can be the most recent 12 months. For incremental data, each incremental data can include multiple service data, and each service data has its corresponding data identifier (e.g., data ID) for uniquely determining the service data, which can be composed of at least one string such as number, letter, etc. The incremental data can be in minute dimension, and the service data within the most recent t minutes of the day can be aggregated according to the data identifier (e.g., data ID) to generate the latest incremental service data. Then, the aggregated offline service data is saved in a target storage location. By way of example only, the target storage location can include, but is not limited to, a database, a cloud, etc.

[0094] The above embodiments, by performing data slicing processing on offline service data based on different time dimensions to obtain at least one historical data slice, and performing data aggregation on each historical data slice respectively, and saving the aggregated offline service data to a target storage location, achieve the generation of offline service data into day-granularity historical full volume data and minute-granularity daily incremental data through data slicing and aggregation processing, thereby reducing the data storage volume, and the aggregated data is convenient for direct and fast loading, improving the data processing efficiency.

[0095] In an optional implementation, as shown in Figure 4 The method further includes:

[0096] S407: Obtain a historical service query request;

[0097] S409: Obtain a target data identifier corresponding to the historical service query request, and save the target data identifier;

[0098] S411: In the case of meeting the data loading condition, load the target offline service data corresponding to the target data identifier from the target storage location to the first storage area, and record the first partition time corresponding to the target offline service data.

[0099] The first partition time represents the generation time of the full data partition loaded into the first storage area.

[0100] Optionally, the historical service query request can be obtained by analyzing the historical request log. Then, the target data identifier corresponding to the historical service query request is queried from the database, and the target data identifier is saved. Then, in the case of system startup, it can be determined that the data loading condition is met, and then the target offline service data corresponding to the target data identifier is queried from the target storage location, and the target offline service data obtained by querying is loaded into the first storage area, and the first partition time corresponding to the target offline service data is recorded, so that subsequent query tasks can be assisted according to the first partition time. Since the target offline service data obtained by querying has already performed data aggregation processing, the target offline service data can be directly written into the first storage area (i.e. full data Map) according to the Map data format.

[0101] The above embodiment obtains the target data identifier corresponding to the historical service query request, and loads the target offline service data corresponding to the target data identifier into the first storage area. Only the data with recent service query request is loaded, avoiding full service data loading when the system starts. Since the full service data of each day is very large (at least hundreds of millions), by loading only the data with recent service query request, the problem of long loading time and large memory occupation is avoided.

[0102] In an optional implementation, the first storage area includes a plurality of first storage sub-areas, and the loading of the target offline service data corresponding to the target data identifier from the target storage location to the first storage area can include:

[0103] S4111: Obtain data attribute information of the target offline service data corresponding to the target data identifier from the target storage location, the data attribute information including at least one of data type, data partition time and data time;

[0104] S4113: Load the target offline business data into the corresponding first storage sub-area in the first storage area based on the data attribute information.

[0105] The first storage sub-area can be used to store business data of a Map data structure. Taking an advertisement exchange scenario as an example, the data types can include data levels such as an advertiser level, a promotion plan level, an advertisement level, and a creative level. The data partition time represents the generation time of the data partition used to store the business data. The data time represents the generation time of the data.

[0106] Optionally, the data attribute information corresponding to the target offline business data is obtained from the target storage location, and the data attribute information includes at least one of a data type, a data partition time, and a data time. Then, the data is further partitioned or classified according to the data attribute information corresponding to the target offline business data, and a plurality of first storage sub-areas are obtained, each of which can be used to store data with different data attribute information.

[0107] The above embodiment loads the target offline business data into the corresponding plurality of first storage sub-areas in the first storage area based on the data attribute information, avoids storing a large amount of data in a single full data Map, and affects the write and read performance, and at the same time, reduces the lock competition during writing, and improves the overall concurrent performance of the first storage area.

[0108] In an optional embodiment, when storing data, in order to reduce the storage space, the data is serialized for storage using, for example, a Protobuf format. For example, in a report data, about 200 index fields are included, but generally only about 10 fields have values, and the others are 0. If the data is directly stored, a large amount of storage space will be occupied by these null value data. Protobuf does not store null value fields during serialization, and is compressed, which greatly reduces the storage space and is beneficial to improving the overall performance of the system.

[0109] In an optional embodiment, as shown in Figure 5 the method further includes:

[0110] S501: Obtain each candidate incremental data whose incremental partition time is later than the first partition time; the incremental partition time represents the generation time of the incremental data partition of the incremental data;

[0111] S503: Load the candidate incremental data into the second storage area according to the time sequence of the incremental partition time corresponding to the candidate incremental data, and record the second partition time corresponding to the candidate incremental data.

[0112] The second partition time represents the generation time of the incremental data partition loaded into the second storage area.

[0113] Optionally, after the recordation of the full amount of business data is completed, and the first partition time of the currently loaded full amount of business data is recorded in the memory, a timing loading task of the incremental business data partition can be started. In the timing loading task, each candidate incremental data whose incremental partition time is after the first partition time corresponding to the loaded full amount of business data can be queried from the database, and loaded into the second storage area in the order of the incremental partition time corresponding to the candidate incremental data. Specifically, in the loading process, for each incremental partition, the corresponding incremental business data is batch queried, and is concurrently loaded into the second storage area. When the incremental data loading is completed, the latest incremental partition time that has been loaded can be saved, that is, the second partition time corresponding to the candidate incremental data is saved to the local memory, and the next time the task is started, the data partition that has been loaded this time is executed.

[0114] The above embodiment realizes the ordered loading of the incremental business data by loading the candidate incremental data into the second storage area in the time order of the incremental partition time corresponding to the candidate incremental data, and recording the second partition time corresponding to the candidate incremental data, and further improves the data processing efficiency.

[0115] In an optional embodiment, the second storage area includes a plurality of second storage sub-areas, and the loading of the candidate incremental data into the second storage area includes:

[0116] S5031: converting the candidate incremental data into incremental converted data of different time dimensions;

[0117] S5033: when it is detected that there is a second storage sub-area of the dimension time corresponding to the incremental converted data, storing the incremental converted data into the second storage sub-area of the corresponding dimension time;

[0118] S5035: when it is detected that there is no second storage sub-area of the dimension time corresponding to the incremental converted data, storing the incremental converted data into the second storage area.

[0119] Optionally, for each candidate incremental data, the data time can be respectively converted into day dimension and month dimension, and it is queried from the incremental data Map of the second storage area whether the incremental business data of the corresponding data date exists, if it exists, the data is added and updated, that is, the incremental converted data is stored into the second storage sub-area of the corresponding dimension time, otherwise, it is directly set to the incremental data Map.

[0120] The above embodiment converts the candidate incremental data into incremental conversion data of different time dimensions, stores the incremental conversion data into the second storage sub-area corresponding to the dimensional time after detecting that the dimensional time corresponding to the incremental conversion data exists, realizes further aggregation of data, simplifies the data storage amount, and improves the data processing efficiency.

[0121] For ease of understanding, the following takes the advertisement watchlist scene as an example for illustration, as shown in FIG. 1, the data processing method can include the following steps: Figure 6

[0122] 1) Through a Spark (a big data computing engine) offline task, pre-aggregate the report offline data provided by a downstream to generate historical full data of day granularity and daily incremental data of minute granularity, and save them to a database DB;

[0123] 2) Through a Spark offline task, parse the historical request log to obtain the data ID of the historical request, and save it to the database DB;

[0124] 3) When the system starts, batch load the ID data of the recent request from the database DB to the memory in a Map data format;

[0125] 4) After the system starts, through a timing task, load the incremental data recently generated in the database DB to the memory in a Map data format every minute;

[0126] 5) When a query request is received, determine whether the offline data needs to be used through a configuration switch;

[0127] 6) When the switch is closed, directly call an online interface to obtain the report data;

[0128] 7) When the switch is opened, use the offline report data, first split the request date range into time slices, and query each slice respectively;

[0129] 8) Query the full data from the memory full data Map, and if the full data does not exist, load it from the database DB;

[0130] 9) Query the incremental data from the memory incremental data Map, and accumulate it to the full data;

[0131] 10) Arrange and return the final report data.

[0132] The technical details not described in the above embodiment are described in the method provided by any of the above embodiments of the present disclosure, and will not be described here.

[0133] ​The following is an embodiment of the device of the present disclosure, which can be used to execute the method embodiment of the present disclosure. For details not disclosed in the device embodiment of the present disclosure, please refer to the method embodiment of the present disclosure.

[0134] Please refer to Figure 7 which shows a structural block diagram of a data processing device provided by an embodiment of the present disclosure. The device has the functions in the above method examples, which can be realized by hardware or corresponding software executed by hardware. The data processing device can include:

[0135] The first obtaining module 710 is configured to obtain a service query request, wherein the service query request comprises a query time range and query information.

[0136] The segmentation module 720 is configured to perform segmentation processing on the query time range based on different time dimensions to obtain at least one time slice.

[0137] The second obtaining module 730 is configured to, for each time slice, obtain full-volume service data corresponding to the time slice from the first storage area based on the query information, wherein the full-volume service data comprises data that has been subjected to data partitioning processing and loaded into the first storage area.

[0138] The third obtaining module 740 is configured to obtain a full-volume partition time corresponding to the full-volume service data, and obtain incremental service data from the second storage area based on the query information, wherein the full-volume partition time represents the generation time of full-volume data partitioning of the full-volume service data, the incremental service data comprises data corresponding to a query time later than the full-volume partition time in the query time range; and the second storage area and the first storage area are both located in a local device.

[0139] The first generating module 750 is configured to generate a query result based on the full-volume service data and the incremental service data.

[0140] In an optional embodiment, the second obtaining module comprises:

[0141] The first obtaining sub-module is configured to, in a case where it is determined that the full-volume service data corresponding to the time slice and the query information has been loaded offline to the first storage area, obtain the full-volume service data corresponding to the time slice from the first storage area.

[0142] The second obtaining sub-module is configured to, in a case where it is determined that the time slice and full-volume service data corresponding to the query information are not loaded offline to the first storage area, load target full-volume data corresponding to the time slice into the first storage area, and synchronously obtain the target full-volume data from the first storage area, and take the target full-volume data as the full-volume service data; the target full-volume data is offline data corresponding to a full-volume partition time close to a maximum boundary time in the query time range.

[0143] In an optional implementation, the apparatus further includes:

[0144] The calling module is configured to, in a case where it is detected that the preset offline data obtaining condition is not met, call an online service query interface, and obtain target service data based on the query information.

[0145] The second generating module is configured to generate a query result based on the target service data.

[0146] The preset offline data obtaining condition includes at least one of the following: an offline query function is enabled, offline service data is normally loaded, and a consistency detection result corresponding to the offline service data meets a preset condition.

[0147] In an optional implementation, the apparatus further includes:

[0148] The fourth obtaining module is configured to obtain offline service data.

[0149] The fifth obtaining module is configured to perform data slicing processing on the offline service data based on different time dimensions, and obtain at least one historical data slice.

[0150] The aggregation module is configured to perform data aggregation on each historical data slice respectively, and save the aggregated offline service data to a target storage location.

[0151] In an optional implementation, the apparatus further includes:

[0152] The sixth obtaining module is configured to obtain a historical service query request.

[0153] The seventh obtaining module is configured to obtain a target data identifier corresponding to the historical service query request, and save the target data identifier.

[0154] The first loading module is configured to, in a case where a data loading condition is met, load target offline service data corresponding to the target data identifier from the target storage location into the first storage area, and record a first partition time corresponding to the target offline service data, the first partition time representing a generation time of full-volume data partition loaded into the first storage area.

[0155] In an optional implementation, the first storage area includes a plurality of first storage sub-areas, and the first loading module includes:

[0156] The attribute obtaining module is configured to obtain, from the target storage location, data attribute information of the target offline service data corresponding to the target data identifier, the data attribute information including at least one of a data type, a data partition time, and a data time.

[0157] The first loading sub-module is configured to load the target offline service data into corresponding first storage sub-areas of the first storage area based on the data attribute information.

[0158] In an optional implementation, the apparatus further includes:

[0159] The eighth obtaining module is configured to obtain each candidate incremental data whose incremental partition time is later than the first partition time, the incremental partition time representing a generation time of an incremental data partition of the incremental data.

[0160] The second loading module is configured to load the candidate incremental data into the second storage area according to a time sequence of the incremental partition time corresponding to the candidate incremental data, and record a second partition time corresponding to the candidate incremental data, the second partition time representing a generation time of an incremental data partition loaded into the second storage area.

[0161] In an optional implementation, the second storage area includes a plurality of second storage sub-areas, and the second loading module includes:

[0162] The data conversion sub-module is configured to convert the candidate incremental data into incremental converted data of different time dimensions.

[0163] The first storage sub-module is configured to store the incremental converted data into a second storage sub-area corresponding to a dimension time of the incremental converted data, when it is detected that the second storage sub-area corresponding to the dimension time of the incremental converted data exists.

[0164] The second storage sub-module is configured to store the incremental converted data into the second storage area, when it is detected that the second storage sub-area corresponding to a dimension time of the incremental converted data does not exist.

[0165] The apparatuses provided in the above embodiments can execute the corresponding methods in the embodiments of the present disclosure, and have the corresponding function modules and advantages of executing the methods. Technical details not described in the above embodiments can be referred to the methods provided in any embodiment of the present disclosure.

[0166] The computer device can include a processor and a memory having stored therein at least one instruction, at least one program, a code set or an instruction set, which are loaded and executed by the processor to implement the method according to any one of the above method embodiments.

[0167] The computer device can include a processor and a memory having stored therein at least one instruction, at least one program, a code set or an instruction set, which are loaded and executed by the processor to implement the method according to any one of the above method embodiments.

[0168] The computer device can include a processor and a memory having stored therein at least one instruction, at least one program, a code set or an instruction set, which are loaded and executed by the processor to implement the method according to any one of the above method embodiments.

[0169] Further, Figure 8 A hardware structure schematic diagram of a device for implementing the method provided in the embodiments of the present disclosure is shown, which can be a computer terminal, a mobile terminal or other device, and the device can also participate in constituting or containing the apparatus provided in the embodiments of the present disclosure. As shown in Figure 8 The computer terminal 11 can include one or more (in the figure, 112a, 112b, …, 112n are shown) processors 112 (the processor 112 can include but is not limited to a processing device such as a microprocessor MCU or a programmable logic device FPGA), a memory 114 for storing data, and a transmission device 116 for communication function. In addition, it can also include a display, an input / output interface (I / O interface), a universal serial bus (USB) port (which can be included as one of the ports of the I / O interface), a network interface, a power supply and / or a camera. Those skilled in the art can understand that Figure 8 The structure shown is only schematic, which does not limit the structure of the above-mentioned electronic device. For example, the computer terminal 11 can include more or less components than those shown in Figure 8 or have a different configuration from that shown in Figure 8 .

[0170] It should be noted that the one or more processors 112 and / or other data processing circuitry described above can be generally referred to herein as "data processing circuitry". The data processing circuitry can be embodied in whole or in part as software, hardware, firmware, or any combination thereof. In addition, the data processing circuitry can be a single standalone processing module, or incorporated in whole or in part within any of the other elements of the computer terminal 11 (or mobile device). As referred to in the embodiments of the present disclosure, the data processing circuitry functions as a processor to control, for example, the selection of the variable resistance terminal path connected to the interface.

[0171] The memory 114 can be used to store software programs of application software and modules, such as program instructions / data storage means corresponding to the method described in the embodiments of the present disclosure. The processor 112 can execute various functional applications and data processing by running the software programs and modules stored in the memory 114, i.e. implement the neural network processing method described above. The memory 114 can include a high-speed random access memory, and can further include a non-volatile memory, such as one or more magnetic storage devices, flash memories, or other non-volatile solid-state memories. In some examples, the memory 114 can further include a memory disposed remotely with respect to the processor 112, which can be connected to the computer terminal 11 through a network. Examples of the network can include, but are not limited to, the Internet, an intranet, a local area network, a mobile communication network, and combinations thereof.

[0172] The transmission device 116 is configured to receive or send data via a network. Specific examples of the network can include a wireless network provided by a communication provider of the computer terminal 11. In one example, the transmission device 116 includes a network adapter (NIC), which can be connected to other network devices through a base station so as to communicate with the Internet. In one example, the transmission device 116 can be a radio frequency (RF) module, which is configured to communicate with the Internet in a wireless manner.

[0173] The display can be, for example, a touch screen type liquid crystal display (LCD), which can enable a user to interact with the user interface of the computer terminal 11 (or mobile device).

[0174] It should be noted that the above-mentioned sequence of the embodiments of the present disclosure is only for description, and does not represent the advantages and disadvantages of the embodiments. And the above describes the specific embodiments of the present disclosure. Other embodiments are within the scope of the appended claims. In some cases, the actions or steps recited in the claims can be performed in a different order than the order in which they are recited and still achieve desirable results. In addition, the processes depicted in the figures do not necessarily require the particular order shown, or sequential order, to achieve the desired results. In certain implementations, multitasking and parallel processing can be advantageous or possible.

[0175] Each of the embodiments in the present disclosure is described in a progressive manner, and the same or similar parts between the embodiments can be referred to each other. Each embodiment focuses on the difference from other embodiments. In particular, for the device and server embodiments, since they are basically similar to the method embodiments, the description is relatively simple, and the relevant parts can be referred to the part of the method embodiments.

[0176] A person of ordinary skill in the art can understand that all or part of the steps of the above-mentioned embodiments can be completed by hardware, or by program instructing relevant hardware to complete, and the program can be stored in a computer readable storage medium. The storage medium mentioned above can be a read-only memory, a magnetic disk or an optical disk.

[0177] The above only describes the preferred embodiments of the present disclosure, and is not intended to limit the present disclosure. Any modification, equivalent replacement, improvement, etc. made within the spirit and principle of the present disclosure shall be included in the protection scope of the present disclosure.

Claims

1. A data processing method, characterized in that, include: Obtain a business query request, which includes a query time range and query information; The query time range is segmented based on different time dimensions to obtain at least one time slice; the dimension category of the time dimension is related to the query time range, and the same time dimension corresponds to at least one time slice. For each time slice, based on the query information, the full business data corresponding to the time slice is obtained from the first storage area. The full business data includes offline data that has been processed by data partitioning and has been loaded from the target storage location into the first storage area. The offline data has undergone data segmentation and data aggregation processing based on multiple time dimensions at the target storage location. The first storage area includes multiple first storage sub-areas, and each first storage sub-area corresponds to offline data with different data attribute information. Based on the business data generation time corresponding to the query information and the current query time corresponding to the business query request, the full partition time corresponding to the full business data is obtained. Based on the full partition time and the query time range, the incremental query time corresponding to the incremental business data is determined. Based on the query information and the incremental query time, the incremental business data is obtained from the second storage area. The full partition time represents the generation time of the full data partition of the full business data. The incremental business data includes offline data whose query time in the query time range is later than the full partition time. Both the second storage area and the first storage area are located in the memory of the local device. Based on the full volume of business data and the incremental business data, query results are generated.

2. The method according to claim 1, characterized in that, The step of retrieving the full business data corresponding to each time slice from the first storage area based on the query information includes: If it is determined that the full business data corresponding to the time slice and the query information has been loaded offline into the first storage area, the full business data corresponding to the time slice is obtained from the first storage area. If it is determined that the full business data corresponding to the time slice and the query information has not been loaded offline into the first storage area, the target full data corresponding to the obtained time slice is loaded into the first storage area, and the target full data is obtained from the first storage area simultaneously, and the target full data is used as the full business data; the target full data is the offline data corresponding to the full partition time that is close to the maximum boundary time in the query time range.

3. The method according to claim 1, characterized in that, The method further includes: If the preset offline data acquisition conditions are not met, the online service query interface is invoked to obtain the target business data based on the query information. Based on the target business data, generate query results; The preset offline data acquisition conditions include at least one of the following: offline query function is enabled, offline business data is loaded normally, and the consistency detection result corresponding to the offline business data meets the preset conditions.

4. The method according to any one of claims 1-3, characterized in that, The method further includes: Obtain offline business data; The offline business data is segmented based on different time dimensions to obtain at least one historical data slice; Data aggregation is performed on each historical data slice, and the aggregated offline business data is saved to the target storage location.

5. The method according to claim 4, characterized in that, The method further includes: Retrieve historical business query requests; Obtain the target data identifier corresponding to the historical business query request, and save the target data identifier; If the data loading conditions are met, the target offline business data corresponding to the target data identifier is loaded from the target storage location into the first storage area, and the first partition time corresponding to the target offline business data is recorded. The first partition time represents the generation time of the full data partition loaded into the first storage area.

6. The method according to claim 5, characterized in that, The first storage area includes several first storage sub-areas, and loading the target offline service data corresponding to the target data identifier from the target storage location into the first storage area includes: From the target storage location, obtain the data attribute information of the target offline business data corresponding to the target data identifier, wherein the data attribute information includes at least one of data type, data partition time, and data time; Based on the data attribute information, the target offline business data are loaded into the corresponding first storage sub-area in the first storage area.

7. The method according to claim 5, characterized in that, The method further includes: Obtain candidate incremental data whose incremental partitioning time is after the first partitioning time; the incremental partitioning time represents the generation time of the incremental data partition. According to the time sequence of the incremental partition time corresponding to the candidate incremental data, the candidate incremental data is loaded into the second storage area, and the second partition time corresponding to the candidate incremental data is recorded. The second partition time represents the generation time of the incremental data partition loaded into the second storage area.

8. The method according to claim 7, characterized in that, The second storage area includes several second storage sub-areas, and loading the candidate incremental data into the second storage area includes: The candidate incremental data is transformed into incremental transformed data in different time dimensions; Upon detecting the existence of a second storage sub-area corresponding to the dimension time of the incremental transformation data, the incremental transformation data is stored in the second storage sub-area corresponding to the dimension time. If the second storage sub-area corresponding to the dimension time of the incremental transformation data is not detected, the incremental transformation data is stored in the second storage area.

9. A data processing apparatus, characterized in that, The device includes: The first acquisition module is used to acquire a business query request, which includes a query time range and query information. The segmentation module is used to segment the query time range based on different time dimensions to obtain at least one time slice; the dimension category of the time dimension is related to the query time range, and the same time dimension corresponds to at least one time slice. The second acquisition module is used to acquire the full business data corresponding to each time slice from the first storage area based on the query information. The full business data includes data that has been processed by data partitioning and has been loaded from the target storage location into the first storage area. The offline data is processed by data segmentation and data aggregation based on multiple time dimensions at the target storage location. The first storage area includes multiple first storage sub-areas, and each first storage sub-area corresponds to offline data with different data attribute information. The third acquisition module is used to acquire the full partition time corresponding to the full business data based on the business data generation time corresponding to the query information and the current query time corresponding to the business query request, and to determine the incremental query time corresponding to the incremental business data based on the full partition time and the query time range, and to acquire the incremental business data from the second storage area based on the query information and the incremental query time. The full partition time represents the generation time of the full data partition of the full business data, and the incremental business data includes data whose query time in the query time range is later than the full partition time. Both the second storage area and the first storage area are located in the memory of the local device. The first generation module is used to generate query results based on the full volume of business data and the incremental business data.

10. An electronic device, characterized in that, The electronic device includes a processor and a memory, the memory storing at least one instruction or at least one program, the at least one instruction or the at least one program being loaded and executed by the processor to implement the data processing method as described in any one of claims 1-8.

11. A computer-readable storage medium, characterized in that, The storage medium stores at least one instruction or at least one program segment, which is loaded and executed by a processor to implement the data processing method as described in any one of claims 1-8.

12. A computer program product, characterized in that, The computer program product includes at least one instruction or at least one program segment, which is loaded and executed by a processor to implement the data processing method as described in any one of claims 1-8.

Images