Data processing method, device and electronic equipment

Abstract

The embodiments of this specification provide a data processing method, device and electronic device. The method includes receiving a data stream to be processed input by an upstream system, and performing the following processing on the data in the data stream according to the order of the data in the data stream: performing two-stage aggregation on the data of the first number of rows in the data stream, and after performing the first stage aggregation in the two-stage aggregation, determining whether the aggregation degree of the first stage aggregation is less than a threshold; wherein the two-stage aggregation refers to performing the first stage aggregation and the second stage aggregation; if so, determining the data of the second number of rows after the data of the first number of rows as the data to be processed, and performing the second stage aggregation of the two-stage aggregation on the data to be processed; if not, determining the data of the second number of rows after the data of the first number of rows as the data to be processed, and performing the two-stage aggregation on the data to be processed.

Description

Technical Field

[0001] The embodiments of this specification relate to the field of data processing technology, and in particular, to a data processing method, device, and electronic device. Background Art

[0002] Data aggregation involves grouping and clustering data based on its characteristics. Hash aggregation methods typically include single-stage and two-stage aggregations. Single-stage aggregation is recommended when the overall degree of data aggregation is high, while two-stage aggregation is recommended when the overall degree of data aggregation is low.

[0003] In practical applications, because the degree of aggregation between data is usually not high, technicians often use two-stage aggregation. As the name suggests, the two-stage aggregation method is divided into two stages, where the first stage is called partial aggregation (Partial Aggregation), and the second stage after the first stage is called final aggregation (Final Aggregation). During implementation, for the data to be processed, it is first necessary to partially aggregate the data in the first stage, and then continue to perform final aggregation on the partially aggregated data in the second stage.

[0004] Although two-stage aggregation can achieve data aggregation with an overall low degree of aggregation, it also has problems such as slow aggregation speed and large amount of calculation due to the need to perform aggregation twice. Summary of the Invention

[0005] The embodiments of this specification provide a data processing method, device, and electronic device.

[0006] According to a first aspect of an embodiment of this specification, a data processing method is provided, the method comprising:

[0007] Receive the data stream to be processed input by the upstream system, and perform the following processing on the data in the data stream according to the order of the data in the data stream:

[0008] performing two-stage aggregation on a first number of rows of data in the data stream, and after performing a first-stage aggregation in the two-stage aggregation, determining whether an aggregation degree of the first-stage aggregation is less than a threshold; wherein the two-stage aggregation refers to performing the first-stage aggregation and the second-stage aggregation;

[0009] If yes, the data of the second number of rows after the data of the first number of rows are determined as the data to be processed, and the second stage aggregation of the two-stage aggregation is performed on the data to be processed; if no, the data of the second number of rows after the data of the first number of rows are determined as the data to be processed, and the two-stage aggregation is performed on the data to be processed.

[0010] Optionally, determining whether the degree of polymerization in the first stage is less than a threshold includes:

[0011] Obtaining the number of data groups after aggregation in the first stage;

[0012] determining a ratio of the first number of rows to the number of groups as an aggregation degree of the first stage aggregation;

[0013] The degree of polymerization of the first-stage polymerization is compared with a threshold value to determine whether the degree of polymerization of the first-stage polymerization is less than the threshold value.

[0014] Optionally, after the second stage aggregation of the two-stage aggregation of the data to be processed, the method further includes:

[0015] Sampling and estimating the tail data in the data to be processed, and estimating whether the aggregation degree of the first stage aggregation is less than a threshold when the tail data is subjected to the first stage aggregation;

[0016] If yes, determining the second number of rows of data following the data to be processed as new data to be processed, and repeatedly performing the second stage aggregation of the two-stage aggregation on the data to be processed;

[0017] If not, a second number of rows of data following the data to be processed are determined as new data to be processed, and the two-stage aggregation of the second number of rows of data is repeated.

[0018] Optionally, after performing two-stage aggregation on the data to be processed, the method further includes:

[0019] Determining whether an aggregation degree of the first-stage aggregation of the data to be processed is less than a threshold when the first-stage aggregation is performed;

[0020] If yes, determining the second number of rows of data following the data to be processed as new data to be processed, and repeatedly performing the second stage aggregation of the two-stage aggregation on the data to be processed;

[0021] If not, a second number of rows of data following the data to be processed are determined as new data to be processed, and the two-stage aggregation of the second number of rows of data is repeated.

[0022] Optionally, the sampling and estimating of the tail data in the data to be processed includes:

[0023] The tail data in the data to be processed are sampled and estimated by a lightweight sampling and estimation algorithm; wherein the computational complexity of the lightweight sampling and estimation algorithm is less than the computational complexity of performing the first stage aggregation on the tail data.

[0024] Optionally, the sampling estimation algorithm includes a HyperLogLog algorithm.

[0025] Optionally, the upstream system includes a massively parallel processing database.

[0026] Optionally, the two-stage aggregation includes Two-stage Agg, the first-stage aggregation includes PartialAgg, and the second-stage aggregation includes Final Agg.

[0027] According to a second aspect of the embodiments of this specification, a data processing device is provided, the device comprising:

[0028] The receiving unit receives the data stream to be processed input by the upstream system and performs the following processing on the data in the data stream according to the order of the data in the data stream:

[0029] a determining unit configured to perform two-stage aggregation on a first number of rows of data in the data stream, and after performing a first-stage aggregation in the two-stage aggregation, determine whether an aggregation degree of the first-stage aggregation is less than a threshold; wherein the two-stage aggregation refers to performing the first-stage aggregation and the second-stage aggregation;

[0030] If yes, the first aggregation unit determines the data of the second number of rows following the data of the first number of rows as data to be processed, and performs a second stage of two-stage aggregation on the data to be processed;

[0031] If not, the second aggregation unit determines the data of the second number of rows following the data of the first number of rows as data to be processed, and performs two-stage aggregation on the data to be processed.

[0032] According to a third aspect of the embodiments of this specification, there is provided an electronic device, including:

[0033] processor;

[0034] a memory for storing processor-executable instructions;

[0035] Wherein, the processor is configured as any of the above-mentioned data processing methods.

[0036] According to a fourth aspect of the embodiments of this specification, a computer-readable storage medium is provided. When instructions in the computer-readable storage medium are executed by a processor of an electronic device, the electronic device is enabled to perform any of the above-mentioned data processing methods.

[0037] The embodiments of this specification provide a data aggregation solution that calculates the actual degree of aggregation of data in a data stream to match the grouping aggregation strategy for subsequent data. For example, for data with a high degree of local aggregation, the first stage of aggregation can be skipped and the second stage of aggregation can be performed directly, while for data with a low degree of local aggregation, the full two-stage aggregation is still performed. In this way, different two-stage aggregation processes are performed on different parts of the data stream through different grouping aggregation strategies, thereby reducing unnecessary aggregation processes and improving the performance of grouping aggregation. BRIEF DESCRIPTION OF THE DRAWINGS

[0038] Figure 1 This is a schematic diagram of a two-stage polymerization provided in one embodiment of this specification;

[0039] Figure 2 is a flow chart of a data processing method provided in one embodiment of this specification;

[0040] Figure 3 is a schematic diagram of adaptive data aggregation provided by an embodiment of this specification;

[0041] Figure 4 This is a hardware structure diagram of a data processing device provided in one embodiment of this specification;

[0042] Figure 5 It is a module of the data processing device provided in an embodiment of this specification;

[0043] Figure 6 This is a schematic block diagram of an electronic device provided in one embodiment of this specification. DETAILED DESCRIPTION

[0044] Exemplary embodiments will be described in detail herein, with examples illustrated in the accompanying drawings. In the following description, when referring to the drawings, identical numerals in different figures represent identical or similar elements unless otherwise indicated. The embodiments described in the following exemplary embodiments are not intended to represent all embodiments consistent with this specification. Rather, they are merely examples of apparatus and methods consistent with certain aspects of this specification, as detailed in the appended claims.

[0045] The terms used in this specification are for the purpose of describing specific embodiments only and are not intended to limit this specification. As used in this specification and the appended claims, the singular forms "a," "an," "the," and "the" are intended to include the plural forms as well, unless the context clearly indicates otherwise. It should also be understood that the term "and / or" as used herein refers to and encompasses any and all possible combinations of one or more of the associated listed items.

[0046] It should be understood that although the terms first, second, third, etc. may be used in this specification to describe various information, such information should not be limited to these terms. These terms are merely used to distinguish information of the same type from one another. For example, first information may also be referred to as second information, and similarly, second information may also be referred to as first information without departing from the scope of this specification. Depending on the context, the term "if" as used herein may be interpreted as "when," "when," or "in response to determining."

[0047] Please refer to the following Figure 1 The diagram below shows a two-stage aggregation process. The two-stage aggregation process consists of two sequential stages: the first stage is called the first stage aggregation, and the second stage is called the second stage aggregation. In implementation, each compute node performs a first-stage aggregation (PartialAgg, also known as partial aggregation) on the data in the first stage. The number of compute nodes is at least two. In the second stage, the results of the first-stage aggregation from each compute node are combined, which is called the second-stage aggregation (FinalAgg, also known as final aggregation).

[0048] However, two-stage aggregation has problems such as slow aggregation speed and large computational complexity because it requires two polymerizations.

[0049] To this end, this specification aims to provide a data processing solution that can adaptively adjust the data aggregation of a two-stage aggregation process, and perform two-stage aggregation of different processes on different parts of the data stream through different grouping aggregation strategies. For example, for data with a high degree of local aggregation, the first stage of aggregation can be skipped and the second stage of aggregation can be directly performed. This aggregation strategy is called the first strategy (bypass mode); while for data with a low degree of local aggregation, the full two-stage aggregation is still performed (i.e., the first stage of aggregation is performed first, followed by the second stage of aggregation). This aggregation strategy is called the second strategy (hash mode).

[0050] Among the data with an overall low degree of aggregation, there is also data with a locally high degree of aggregation. For these locally high-degree data, it is not necessary to perform the complete two-stage aggregation process. Through the aforementioned first strategy, the first stage of aggregation can be skipped and the second stage of aggregation can be performed directly. In this way, by reducing unnecessary aggregation processes, the aggregation speed is increased and the amount of aggregation calculations is reduced.

[0051] The following combination Figure 2 The flowchart of the data processing method provided in this specification is used to introduce relevant embodiments, and the method includes:

[0052] Step 210: Receive the data stream to be processed input by the upstream system, and perform the following processing on the data in the data stream according to the order of the data in the data stream:

[0053] Step 220: performing two-stage aggregation on the first number of rows of data in the data stream, and after performing the first stage of aggregation in the two-stage aggregation, determining whether the aggregation degree of the first stage of aggregation is less than a threshold; wherein the two-stage aggregation refers to performing the first stage of aggregation and the second stage of aggregation;

[0054] Step 230: If yes, determine the second number of rows of data following the first number of rows of data as data to be processed, and perform the second stage of the two-stage aggregation on the data to be processed;

[0055] Step 231: If not, determine the second number of rows of data following the first number of rows of data as data to be processed, and perform two-stage aggregation on the data to be processed.

[0056] The embodiments provided in this specification can be applied to any software, hardware, or a combination of software and hardware that has a need for group aggregation, including but not limited to a server that groups and aggregates data.

[0057] The group aggregation method may include two-stage aggregation; wherein, the first stage corresponding to the two-stage aggregation performs first-stage aggregation, and the second stage corresponding to the two-stage aggregation performs second-stage aggregation.

[0058] As mentioned above, the first stage polymerization in the two-stage polymerization can also be called partial polymerization (Partial Agg), and the second stage polymerization can be called final polymerization (Final Agg); in the following embodiments, partial polymerization and final polymerization will be used as examples for explanation.

[0059] In this specification, the upstream system may include a Massively Parallel Processing (MPP) database; and the server may include a computing engine of the MPP database.

[0060] Taking an MPP database scenario as an example, an MPP database can send data from the database as a data stream to the computing engine, which then groups and aggregates the data stream. A data stream is an ordered sequence of data with a clear temporal relationship, typically arranged in chronological order.

[0061] Of course, the server can also be the server of an application APP, and the upstream system can be a third-party system associated with the application APP. For example, for a navigation map application APP, the upstream system can be a third-party system that provides navigation data, such as a satellite service system. The satellite service system can provide a data stream of navigation data to the server corresponding to the map application APP, and the server can group and aggregate the data stream.

[0062] The following combination Figure 3 The adaptive data aggregation diagram is shown in FIG. Figure 3 As shown, after receiving the input data stream from the upstream system, the server can process the data in the data stream according to the sequence of the data in the data stream.

[0063] First, the server can perform a complete two-stage aggregation on the first number of rows of data in the data stream, and after performing partial aggregation in the two-stage aggregation, determine whether the aggregation degree of the partial aggregation is less than a threshold; wherein the complete two-stage aggregation means performing partial aggregation first and then performing final aggregation.

[0064] If yes, the data of the second number of rows after the data of the first number of rows are determined as the data to be processed, and when performing two-stage aggregation on the data to be processed, skip part of the aggregation and directly perform the final aggregation; if no, the data of the second number of rows after the data of the first number of rows are determined as the data to be processed, and perform a complete two-stage aggregation on the data to be processed.

[0065] like Figure 3 As shown, a complete two-stage aggregation can be performed on the M rows of data in the data stream, and when a partial aggregation in the two-stage aggregation is performed on the M rows of data, the aggregation degree of the partial aggregation is calculated to determine whether the aggregation degree of the partial aggregation is less than a threshold;

[0066] Secondly, if the degree of aggregation of the partial aggregation is less than a threshold, the N rows of data following the M rows of data can be determined as the data to be processed. When performing two-stage aggregation on these N rows of data using the first strategy described above, the partial aggregation can be skipped and the final aggregation can be performed directly.

[0067] On the contrary, if the degree of aggregation of the partial aggregation is not less than the threshold, the N rows of data after the M rows of data can be determined as data to be processed, and the aforementioned second strategy can be used to perform a complete two-stage aggregation on these N rows of data to be processed.

[0068] In this specification, the degree of aggregation represents the degree of partial aggregation of data. A higher degree of aggregation between the original data corresponds to a lower degree of aggregation; conversely, a lower degree of aggregation between the original data corresponds to a higher degree of aggregation. It is understood that the degree of aggregation is inversely proportional to the degree of aggregation between the data.

[0069] In an exemplary embodiment, in step 220, determining whether the degree of polymerization of the partial polymerization is less than a threshold value may include:

[0070] Obtaining the number of groups of the partially aggregated data groups;

[0071] determining a ratio of the first number of rows to the number of groups as a degree of aggregation of the partial aggregation;

[0072] The degree of polymerization of the partial polymerization is compared with a threshold value to determine whether the degree of polymerization of the partial polymerization is less than the threshold value.

[0073] For example, assume that a complete two-stage aggregation is performed on 30 rows (the first number of rows) of data; wherein, after the first stage, i.e., partial aggregation, of these 30 rows of data, the data of these 30 rows are grouped into 3 groups; then, the degree of aggregation of the partial aggregation = the ratio of the first number of rows to the number of groups = 30 / 3 = 10.

[0074] In the above embodiment, the degree of aggregation is calculated by the actual number of groups and the actual number of data rows after partial aggregation. The calculated degree of aggregation is more reliable and can better reflect the actual degree of aggregation of the data, thereby providing reliable data support for determining the aggregation strategy.

[0075] In this specification, after step 230, the method may further include:

[0076] Sampling and estimating tail data in the data to be processed, and estimating whether an aggregation degree of the partial aggregation is less than a threshold when the tail data is partially aggregated;

[0077] If yes, determining the second number of rows of data after the data to be processed as new data to be processed, and repeatedly performing the two-stage aggregation on the data to be processed, skipping some aggregations and directly performing the final aggregation;

[0078] If not, a second number of rows of data following the data to be processed are determined as new data to be processed, and the complete two-stage aggregation of the second number of rows of data is repeatedly performed.

[0079] In this specification, a lightweight sampling estimation method can be used to perform sampling estimation on the tail data in the data to be processed; wherein the computational complexity of the lightweight sampling estimation method is less than the computational complexity of partially aggregating the tail data.

[0080] To maximize the accuracy of sampling estimation, you can use algorithms such as the HyperLogLog algorithm and the Bloom filter algorithm. The HyperLogLog algorithm has higher accuracy than other sampling estimation algorithms, so using the HyperLogLog algorithm for sampling estimation will result in higher data aggregation performance.

[0081] The following continues as Figure 3 As shown, after only the final aggregation is performed on N rows of data, the tail data in the N rows of data are sampled and estimated to estimate whether the aggregation degree of the partial aggregation is less than a threshold when the tail data is partially aggregated.

[0082] If the aggregation degree of the tail data in the N rows of data is less than the threshold, the N rows of data after the N rows of data are determined as new data to be processed. When the first strategy is continued to be used to perform two-stage aggregation on the new N rows of data to be processed, the partial aggregation is skipped and the final aggregation is performed directly;

[0083] If the aggregation degree of the partial aggregation of the tail data in these N rows of data is not less than the threshold, the N rows of data after these N rows of data are determined as new data to be processed, and the current first strategy is switched to the second strategy, and a complete two-stage aggregation is performed on the data to be processed, that is, the new N rows of data.

[0084] Similarly, after step 231, the method may further include:

[0085] Determining whether a degree of aggregation of the partial aggregation of the data to be processed is less than a threshold when the partial aggregation is performed;

[0086] If yes, determining the second number of rows of data after the data to be processed as new data to be processed, and repeatedly performing the two-stage aggregation on the data to be processed, skipping some aggregations and directly performing the final aggregation;

[0087] If not, a second number of rows of data following the data to be processed are determined as new data to be processed, and the complete two-stage aggregation of the second number of rows of data is repeatedly performed.

[0088] Note that unlike the previously mentioned sampling estimation, the second strategy performs a complete two-stage aggregation of the processed data. Therefore, when performing partial aggregation, the actual degree of aggregation can be calculated. Furthermore, the degree of aggregation calculated here is for the N rows of data, while the previously mentioned sampling estimation calculates the degree of aggregation for the tail of the N rows.

[0089] The following continues as Figure 3 As shown, when performing a complete two-stage aggregation on N rows of data, the aggregation degree of the N rows of data when performing partial aggregation is calculated in real time, and it is determined whether the aggregation degree is less than a threshold;

[0090] If the degree of partial aggregation of these N rows of data is less than the threshold, the N rows of data after these N rows of data are determined as new data to be processed. The current second strategy is switched to the first strategy. When performing two-stage aggregation on the new N rows of data to be processed, the partial aggregation is skipped and the final aggregation is performed directly.

[0091] If the degree of partial aggregation of these N rows of data is not less than the threshold, the N rows of data after these N rows of data are determined as new data to be processed, and the second strategy is continued to perform a complete two-stage aggregation on the data to be processed, i.e., the new N rows of data.

[0092] By applying the above embodiment, with N rows of data as a period, the aggregation strategy is periodically switched based on the real-time aggregation degree to achieve adaptive adjustment of the two-stage aggregation, which can further improve the efficiency of data aggregation.

[0093] Corresponding to the aforementioned data processing method embodiment, this specification also provides an embodiment of a data processing device. The device embodiment can be implemented by software, or by hardware or a combination of software and hardware. Taking software implementation as an example, as a device in a logical sense, it is formed by the processor of the device in which it is located reading the corresponding computer program in the non-volatile memory into the memory and running it. From the hardware level, such as Figure 4 The figure is a hardware structure diagram of the device where the data processing device of this manual is located, except Figure 4 In addition to the processor, network interface, memory and non-volatile memory shown, the device in the embodiment where the apparatus is located may also include other hardware according to the actual function of data aggregation, which will not be described in detail.

[0094] See Figure 5 , is a block diagram of a data processing device provided in an embodiment of this specification, the device corresponds to Figure 1 In the embodiment shown, the device comprises:

[0095] The receiving unit 510 receives a data stream to be processed from an upstream system and performs the following processing on the data in the data stream according to the order of the data in the data stream:

[0096] A determining unit 520 performs two-stage aggregation on a first number of rows of data in the data stream, and after performing a first-stage aggregation in the two-stage aggregation, determines whether an aggregation degree of the first-stage aggregation is less than a threshold; wherein the two-stage aggregation refers to performing the first-stage aggregation and the second-stage aggregation;

[0097] If yes, the first aggregation unit 530 determines the second number of rows of data following the first number of rows of data as data to be processed, and performs a second stage of two-stage aggregation on the data to be processed;

[0098] If not, the second aggregation unit 531 determines the second number of rows of data following the first number of rows of data as data to be processed, and performs two-stage aggregation on the data to be processed.

[0099] In an exemplary embodiment, the determining unit 520 determines whether the aggregation degree of the first-stage aggregation is less than a threshold, including:

[0100] Obtain the number of groups of the data grouped after the first stage of aggregation; determine the ratio of the first number of rows to the number of groups as the aggregation degree of the first stage of aggregation; compare the aggregation degree of the first stage of aggregation with a threshold to determine whether the aggregation degree of the first stage of aggregation is less than the threshold.

[0101] In an exemplary embodiment, the first aggregation unit 530 further includes:

[0102] The tail data in the data to be processed is sampled and estimated to estimate whether the aggregation degree of the first stage aggregation is less than a threshold value when the tail data is subjected to the first stage aggregation; if so, the data of the second number of rows after the data to be processed is determined as new data to be processed, and the new data to be processed is input into the first aggregation unit 530; if not, the data of the second number of rows after the data to be processed is determined as new data to be processed, and the new data to be processed is input into the second aggregation unit 531.

[0103] In an exemplary embodiment, the second aggregation unit 531 further includes:

[0104] Determine whether the aggregation degree of the first stage aggregation is less than a threshold value when the data to be processed is being aggregated in the first stage; if so, determine the data of the second number of rows after the data to be processed as new data to be processed, and input the new data to be processed into the first aggregation unit 530; if not, determine the data of the second number of rows after the data to be processed as new data to be processed, and input the new data to be processed into the second aggregation unit 531.

[0105] In an exemplary embodiment, the first aggregation unit 530 performs sampling and estimation on the tail data in the data to be processed, including:

[0106] The tail data in the data to be processed are sampled and estimated by a lightweight sampling and estimation algorithm; wherein the computational complexity of the lightweight sampling and estimation algorithm is less than the computational complexity of performing the first stage aggregation on the tail data.

[0107] In an exemplary embodiment, the sampling prediction algorithm includes a HyperLogLog algorithm.

[0108] In an exemplary embodiment, the upstream system includes a massively parallel processing database; the server includes a computing engine of the massively parallel processing database.

[0109] In an exemplary embodiment, the group aggregation includes Hash Agg, the two-stage aggregation includes Two-stage Agg, the first-stage aggregation includes Partial Agg, and the second-stage aggregation includes Final Agg.

[0110] The systems, devices, modules, or units described in the above embodiments may be implemented by computer chips or entities, or by products having certain functions. A typical implementation device is a computer, which may be in the form of a personal computer, laptop computer, cellular phone, camera phone, smartphone, personal digital assistant, media player, navigation device, email transceiver, game console, tablet computer, wearable device, or any combination of these devices.

[0111] The implementation process of the functions and effects of each unit in the above-mentioned device is specifically described in the implementation process of the corresponding steps in the above-mentioned method, and will not be repeated here.

[0112] For the device embodiments, since they basically correspond to the method embodiments, the relevant parts can be referred to the partial description of the method embodiments. The device embodiments described above are merely illustrative, wherein the units described as separate components may or may not be physically separated, and the components displayed as units may or may not be physical units, that is, they may be located in one place, or they may be distributed on multiple network units. Some or all of the modules may be selected according to actual needs to achieve the purpose of the scheme of this specification. A person of ordinary skill in the art can understand and implement it without paying any creative work.

[0113] Figure 6 1 is a schematic block diagram of an electronic device according to an embodiment of the present disclosure. Figure 6 , electronic device 400 may include one or more of the following components: a processing component 402, a memory 404, a power component 406, a multimedia component 408, an audio component 410, an input / output (I / O) interface 412, a sensor component 414, and a communication component 418. The above electronic devices may adopt a similar hardware architecture.

[0114] Processing component 402 generally controls the overall operation of electronic device 400, such as operations associated with display, phone calls, data communications, camera operation, and recording. Processing component 402 may include one or more processors 420 to execute instructions to perform all or part of the steps of the above-described method for generating a three-dimensional virtual avatar. Furthermore, processing component 402 may include one or more modules to facilitate interaction between processing component 402 and other components. For example, processing component 402 may include a multimedia module to facilitate interaction between multimedia component 408 and processing component 402.

[0115] The memory 404 is configured to store various types of data to support operations on the electronic device 400. Examples of such data include instructions for any application or method operating on the electronic device 400, contact data, phone book data, messages, pictures, videos, etc. The memory 404 can be implemented by any type of volatile or non-volatile storage device, or a combination thereof, such as static random access memory (SRAM), electrically erasable programmable read-only memory (EEPROM), erasable programmable read-only memory (EPROM), programmable read-only memory (PROM), read-only memory (ROM), magnetic memory, flash memory, magnetic disk, or optical disk.

[0116] The power supply assembly 406 provides power to the various components of the electronic device 400. The power supply assembly 406 may include a power management system, one or more power supplies, and other components associated with generating, managing, and distributing power to the electronic device 400.

[0117] The multimedia component 408 includes a screen that provides an output interface between the electronic device 400 and the user. In some embodiments, the screen may include a liquid crystal display (LCD) and a touch panel (TP). If the screen includes a touch panel, the screen may be implemented as a touch screen to receive input signals from the user. The touch panel includes one or more touch sensors to sense touches, slides, and gestures on the touch panel. The touch sensor can not only sense the boundaries of the touch or slide action, but also detect the duration and pressure associated with the touch or slide operation. In some embodiments, the multimedia component 408 includes a front camera and / or a rear camera. When the electronic device 400 is in an operating mode, such as a shooting mode or a video mode, the front camera and / or the rear camera can receive external multimedia data. Each front camera and rear camera can be a fixed optical lens system or an optical lens system with a focal length and optical zoom capability.

[0118] The audio component 410 is configured to output and / or input audio signals. For example, the audio component 410 includes a microphone (MIC), which is configured to receive external audio signals when the electronic device 400 is in an operating mode, such as a call mode, a recording mode, and a voice recognition mode. The received audio signal can be further stored in the memory 404 or transmitted via the communication component 418. In some embodiments, the audio component 410 also includes a speaker for outputting audio signals.

[0119] I / O interface 412 provides an interface between processing component 402 and peripheral interface modules, such as a keyboard, click wheel, options, etc. These options may include but are not limited to: home options, volume options, startup options, and lock options.

[0120] The sensor assembly 414 includes one or more sensors for providing various aspects of status assessment for the electronic device 400. For example, the sensor assembly 414 can detect the open / closed state of the electronic device 400, the relative positioning of components, such as the display and keypad of the electronic device 400. The sensor assembly 414 can also detect changes in the position of the electronic device 400 or a component of the electronic device 400, the presence or absence of user contact with the electronic device 400, the orientation or acceleration / deceleration of the electronic device 400, and temperature changes of the electronic device 400. The sensor assembly 414 may include a proximity sensor configured to detect the presence of nearby objects without any physical contact. The sensor assembly 414 may also include a light sensor, such as a CMOS or CCD image sensor, for use in imaging applications. In some embodiments, the sensor assembly 414 may also include an accelerometer, a gyroscope sensor, a magnetic sensor, a pressure sensor, or a temperature sensor.

[0121] The communication component 418 is configured to facilitate wired or wireless communication between the electronic device 400 and other devices. The electronic device 400 can access a wireless network based on a communication standard, such as WiFi, an operator network (such as 2G, 3G, 4G or 4G), or a combination thereof. In an exemplary embodiment, the communication component 418 receives a broadcast signal or broadcast-related information from an external broadcast management system via a broadcast channel. In an exemplary embodiment, the communication component 418 also includes a near field communication (NFC) module to facilitate short-range communication. For example, the NFC module can be implemented based on radio frequency identification (RFID) technology, infrared data association (IrDA) technology, ultra-wideband (UWB) technology, Bluetooth (BT) technology and other technologies.

[0122] In one embodiment of the present disclosure, the electronic device 400 can be implemented by one or more application-specific integrated circuits (ASICs), digital signal processors (DSPs), digital signal processing devices (DSPDs), programmable logic devices (PLDs), field programmable gate arrays (FPGAs), controllers, microcontrollers, microprocessors or other electronic components to execute the method for locating abnormal problems described in any of the above embodiments.

[0123] In one embodiment of the present disclosure, a computer-readable storage medium including instructions is further provided, such as a memory 404 including instructions. The instructions can be executed by the processor 420 of the electronic device 400 to implement the abnormality locating method described in any of the above embodiments. For example, the computer-readable storage medium can be a ROM, a random access memory (RAM), a CD-ROM, a magnetic tape, a floppy disk, an optical data storage device, or the like.

[0124] The various embodiments in this specification are described in a progressive manner. Similar parts between the various embodiments can be referred to in conjunction with each other. Each embodiment focuses on the differences between the other embodiments. In particular, the electronic device embodiment is generally similar to the method embodiment, so the description is relatively simple. For relevant parts, refer to the partial description of the method embodiment.

[0125] Other embodiments of the present invention will readily occur to those skilled in the art after considering the specification and practicing the invention disclosed herein. This specification is intended to cover any variations, uses, or adaptations of the present invention that follow the general principles of this specification and include common knowledge or customary techniques in the art not disclosed herein. The description and examples are to be considered as exemplary only, with the true scope and spirit of the present invention being indicated by the following claims.

[0126] It should be understood that the present description is not limited to the exact structure that has been described above and shown in the drawings, and that various modifications and changes may be made without departing from the scope thereof. The scope of the present description is limited only by the appended claims.

Claims

1. A data processing method, comprising: Receive the data stream to be processed input by the upstream system, and perform the following processing on the data in the data stream according to the order of the data in the data stream: performing two-stage aggregation on a first number of rows of data in the data stream, and after performing a first-stage aggregation in the two-stage aggregation, determining whether an aggregation degree of the first-stage aggregation is less than a threshold; wherein the two-stage aggregation refers to performing the first-stage aggregation and the second-stage aggregation; If yes, determine the data of the second number of rows following the data of the first number of rows as data to be processed, and perform the second stage aggregation of the two-stage aggregation on the data to be processed; if no, determine the data of the second number of rows following the data of the first number of rows as data to be processed, and perform the two-stage aggregation on the data to be processed; The data aggregation refers to grouping and clustering data according to data characteristics; The determining whether the degree of polymerization of the first-stage polymerization is less than a threshold comprises: Obtaining the number of data groups after aggregation in the first stage; determining a ratio of the first number of rows to the number of groups as an aggregation degree of the first stage aggregation; comparing the degree of polymerization of the first stage polymerization with a threshold value to determine whether the degree of polymerization of the first stage polymerization is less than the threshold value; After performing two-stage aggregation on the data to be processed, the method further includes: Determining whether an aggregation degree of the first-stage aggregation of the data to be processed is less than a threshold when the first-stage aggregation is performed; If yes, determining the second number of rows of data following the data to be processed as new data to be processed, and repeatedly performing the second stage aggregation of the two-stage aggregation on the data to be processed; If not, a second number of rows of data following the data to be processed are determined as new data to be processed, and the two-stage aggregation of the second number of rows of data is repeated.

2. The method according to claim 1, further comprising, after the second stage aggregation of the two-stage aggregation of the data to be processed: Sampling and estimating the tail data in the data to be processed, and estimating whether the aggregation degree of the first stage aggregation is less than a threshold when the tail data is subjected to the first stage aggregation; If yes, determining the second number of rows of data following the data to be processed as new data to be processed, and repeatedly performing the second stage aggregation of the two-stage aggregation on the data to be processed; If not, a second number of rows of data following the data to be processed are determined as new data to be processed, and the two-stage aggregation of the second number of rows of data is repeated.

3. The method according to claim 2, wherein sampling and estimating the tail data in the data to be processed comprises: The tail data in the data to be processed are sampled and estimated by a lightweight sampling and estimation algorithm; wherein the computational complexity of the lightweight sampling and estimation algorithm is less than the computational complexity of performing the first stage aggregation on the tail data. The method according to claim 1 , wherein the upstream system comprises a massively parallel processing database. The method according to claim 1 , wherein the two-stage polymerization comprises a two-stage Agg, the first-stage polymerization comprises a partial Agg, and the second-stage polymerization comprises a final Agg.

6. A data processing device, comprising: The receiving unit receives the data stream to be processed input by the upstream system and performs the following processing on the data in the data stream according to the order of the data in the data stream: a determining unit configured to perform two-stage aggregation on a first number of rows of data in the data stream, and after performing a first-stage aggregation in the two-stage aggregation, determine whether an aggregation degree of the first-stage aggregation is less than a threshold; wherein the two-stage aggregation refers to performing the first-stage aggregation and the second-stage aggregation; If yes, the first aggregation unit determines the data of the second number of rows following the data of the first number of rows as data to be processed, and performs a second stage of two-stage aggregation on the data to be processed; If not, the second aggregation unit determines the data of the second number of rows following the data of the first number of rows as data to be processed, and performs two-stage aggregation on the data to be processed; The data aggregation refers to grouping and clustering data according to data characteristics; The determining whether the degree of polymerization of the first-stage polymerization is less than a threshold comprises: Obtaining the number of data groups after aggregation in the first stage; determining a ratio of the first number of rows to the number of groups as an aggregation degree of the first stage aggregation; comparing the degree of polymerization of the first stage polymerization with a threshold value to determine whether the degree of polymerization of the first stage polymerization is less than the threshold value; After performing two-stage aggregation on the data to be processed, the method further includes: Determining whether an aggregation degree of the first-stage aggregation of the data to be processed is less than a threshold when the first-stage aggregation is performed; If yes, determining the second number of rows of data following the data to be processed as new data to be processed, and repeatedly performing the second stage aggregation of the two-stage aggregation on the data to be processed; If not, a second number of rows of data following the data to be processed are determined as new data to be processed, and the two-stage aggregation of the second number of rows of data is repeated.

7. An electronic device comprising: processor; a memory for storing processor-executable instructions; The processor is configured to execute the method according to any one of claims 1 to 5. 8 . A computer-readable storage medium, wherein when instructions in the computer-readable storage medium are executed by a processor of an electronic device, the electronic device is enabled to perform the method according to claim 1 .

Images