Data processing method, data processing system, electronic device and readable storage medium
By introducing memory modules and computing nodes into the data processing system, the problem of slow computation speed in big data scenarios of existing word frequency analysis methods is solved, achieving efficient word frequency statistics and improved computation speed.
Patent Information
- Authority / Receiving Office
- CN · China
- Patent Type
- Patents(China)
- Current Assignee / Owner
- BEIJING ZITIAO NETWORK TECH CO LTD
- Filing Date
- 2021-10-20
- Publication Date
- 2026-06-02
AI Technical Summary
Existing word frequency analysis methods cannot guarantee normal completion within a cycle time when faced with exponentially growing text data, resulting in slow data reading speed and reduced calculation speed.
By introducing a memory module and multiple computing nodes into the data processing system, the query statement is divided and distributed to the computing nodes for word segmentation and word frequency statistics using the scheduling node. The statistical results are stored in the memory module and then merged and stored in the target storage, avoiding the need to write the word frequency statistics results back to the disk.
It improves data reading efficiency and computing speed, ensuring that offline word frequency analysis tasks in big data scenarios can be completed normally.
Smart Images
Figure CN115994170B_ABST
Abstract
Description
Technical Field
[0001] This application belongs to the field of data processing technology, specifically relating to a data processing method, a data processing system, an electronic device, and a readable storage medium. Background Technology
[0002] In text analysis, word frequency analysis is used to count the frequency of words, thereby assessing the importance of a word to a document. However, word frequency analysis methods suffer from slow data reading speeds and reduced computation speed due to reading and writing intermediate results to disk. When text data grows exponentially, it cannot be guaranteed that word frequency statistics will be completed normally within the specified time period. Summary of the Invention
[0003] The purpose of this application is to provide a data processing method, a data processing system, an electronic device, and a readable storage medium that can solve the problem that word frequency analysis schemes in related technologies cannot enable the computational task to be completed normally.
[0004] In a first aspect, embodiments of this application provide a data processing method for a data processing system, the data processing system including a memory module, a scheduling node, and multiple computing nodes, the data processing method including:
[0005] The scheduling node obtains the query statement of the task to be executed, and divides the query statement into multiple query statement fragments;
[0006] The scheduling node distributes multiple query statements to multiple compute nodes in a shard format;
[0007] The computing node reads the query data in segments according to the query statement, performs word segmentation on the query data to obtain multiple word segmentation results, and counts the first word frequency information of the first keyword in the word segmentation results, and stores the first word frequency information in the memory module;
[0008] After the first word frequency information collected by multiple computing nodes is stored in the memory module, the computing nodes merge the first word frequency information in the memory module to generate the second word frequency information.
[0009] The computing node stores the second word frequency information into the target memory.
[0010] Secondly, embodiments of this application provide a data processing system, including a memory module, a scheduling node, and multiple computing nodes;
[0011] The scheduling node is used to: obtain the query statement of the task to be executed, divide the query statement into multiple query statement fragments, and distribute the multiple query statement fragments to multiple computing nodes.
[0012] The computing nodes are used to: read query data in segments according to the query statement, perform word segmentation on the query data to obtain multiple word segmentation results, count the first word frequency information of the first keyword in the word segmentation results, store the first word frequency information in the memory module, and after the first word frequency information counted by multiple computing nodes is stored in the memory module, merge the first word frequency information in the memory module to generate the second word frequency information, and store the second word frequency information in the target memory.
[0013] Thirdly, embodiments of this application provide an electronic device, which includes a processor, a memory, and a program or instructions stored in the memory and executable on the processor, wherein the program or instructions, when executed by the processor, implement the steps of the method as described in the first aspect.
[0014] Fourthly, embodiments of this application provide a readable storage medium on which a program or instructions are stored, and when the program or instructions are executed by a processor, they implement the steps of the method as described in the first aspect.
[0015] Fifthly, embodiments of this application provide a chip, which includes a processor and a communication interface. The communication interface and the processor are coupled, and the processor is used to run programs or instructions to implement the method as described in the first aspect.
[0016] In this embodiment, the data processing system includes a memory module, a scheduling node, multiple computing nodes, and a client. The client receives a query statement input by the user for a task to be executed. The scheduling node obtains the query statement and divides it into multiple query statement fragments. The scheduling node then sends each query statement fragment to a corresponding computing node, which performs data reading and word frequency calculation. Specifically, on the computing node, query data is read according to the query statement fragments. Then, according to the specific word segmentation algorithm input by the user on the client, the read query data is segmented to obtain multiple word segmentation results. The word segmentation results include multiple keywords. The first word frequency information of the first keyword in the word segmentation results is statistically analyzed and stored in the memory module. It should be noted that there can be one or more computing nodes to obtain the first keyword after word segmentation. After word frequency statistics are performed on all one or more computing nodes, the first word frequency information of the first keyword is stored in the memory module. Further, the first word frequency information stored in the memory module is merged and summarized to obtain the second word frequency information of the first keyword, and the second word frequency information is stored in the target storage (i.e., MySQL). By employing the above methods, on the one hand, query data is read in fragmented form and distributed to different computing nodes, thereby improving data reading efficiency. On the other hand, the word frequency statistics results are stored in memory, allowing computing nodes to remotely read data based on keywords, avoiding the time overhead of writing word frequency statistics results back to disk and thus improving computing speed. Attached Figure Description
[0017] Figure 1 This is a schematic flowchart of the data processing method according to an embodiment of this application;
[0018] Figure 2 This is a schematic block diagram of a data processing system according to an embodiment of this application;
[0019] Figure 3 This is a system architecture diagram of the data processing system according to an embodiment of this application;
[0020] Figure 4 This is one of the logical diagrams of word frequency calculation in the embodiments of this application;
[0021] Figure 5 This is the second logical diagram of word frequency calculation in an embodiment of this application;
[0022] Figure 6 This is one of the schematic block diagrams of the electronic device according to an embodiment of this application;
[0023] Figure 7 This is a second schematic block diagram of an electronic device according to an embodiment of this application. Detailed Implementation
[0024] The technical solutions of the embodiments of this application will be clearly described below with reference to the accompanying drawings. Obviously, the described embodiments are only some, not all, of the embodiments of this application. All other embodiments obtained by those skilled in the art based on the embodiments of this application are within the scope of protection of this application.
[0025] The terms "first," "second," etc., used in the specification and claims of this application are used to distinguish similar objects and not to describe a specific order or sequence. It should be understood that such use of data can be interchanged where appropriate so that embodiments of this application can be implemented in orders other than those illustrated or described herein. Furthermore, in the specification and claims, "and / or" indicates at least one of the connected objects, and the character " / " generally indicates that the preceding and following objects are in an "or" relationship.
[0026] The data processing method, data processing system, electronic device, and readable storage medium provided in this application will be described in detail below with reference to the accompanying drawings and through specific embodiments and application scenarios.
[0027] This application provides a data processing method for a data processing system, which includes a memory module, a scheduling node, and multiple computing nodes, such as... Figure 1 As shown, the data processing method includes:
[0028] Step 102: The scheduling node obtains the query statement of the task to be executed and divides the query statement into multiple query statement fragments.
[0029] Step 104: The scheduling node distributes multiple query statements to multiple compute nodes in shards;
[0030] Step 106: The computing node reads the query data in segments according to the query statement, performs word segmentation on the query data to obtain multiple word segmentation results, and counts the first word frequency information of the first keyword in the word segmentation results, and stores the first word frequency information in the memory module.
[0031] Step 108: After the first word frequency information collected by multiple computing nodes is stored in the memory module, the computing nodes merge the first word frequency information in the memory module to generate the second word frequency information.
[0032] Step 110: The computing node stores the second word frequency information into the target memory.
[0033] In this embodiment, the data processing system includes a memory module, a scheduling node, multiple computing nodes, and a client. The client receives a query statement input by the user for a task to be executed. The scheduling node obtains the query statement and divides it into multiple query statement fragments. The scheduling node then sends each query statement fragment to a corresponding computing node, which performs data reading and word frequency calculation. Specifically, on the computing node, query data is read according to the query statement fragments. Then, the read query data is segmented according to the specific word segmentation algorithm input by the user on the client, resulting in multiple word segmentation results. The word segmentation results include multiple keywords, such as Apple, Boy, Cat, and Dog. The first word frequency information of the first keyword in the word segmentation results is then calculated. For example, if the first keyword is Dog, its first word frequency information is 6, and this first word frequency information is stored in the memory module.
[0034] It should be noted that after the word segmentation operation, there can be one or more first keyword calculation nodes. After word frequency statistics are performed on all one or more calculation nodes, the first word frequency information of the first keyword is stored in the memory module.
[0035] Furthermore, the first word frequency information stored in the memory module is merged and summarized to obtain the second word frequency information of the first keyword, and the second word frequency information is stored in the target storage (i.e., MySQL). For example, for the first keyword "dog", its word frequency is 6 on the first computing node and 3 on the second computing node. Then the merged and summarized word frequency (i.e., the second word frequency information) of the first keyword "dog" is 9.
[0036] By employing the above methods, on the one hand, query data is read in fragmented form and distributed to different computing nodes, thereby improving data reading efficiency. On the other hand, the word frequency statistics results are stored in memory, allowing computing nodes to remotely read data based on keywords, avoiding the time overhead of writing word frequency statistics results back to disk and thus improving computing speed.
[0037] This application embodiment can solve the problem of offline word frequency analysis in big data scenarios by improving reading efficiency and computing speed, and ensure the normal completion of offline tasks with billions of text data.
[0038] Furthermore, in one embodiment of this application, the step of dividing the query statement to obtain multiple query statement fragments includes: obtaining the number of nodes of multiple computing nodes; dividing the query statement according to the number of nodes and the query time corresponding to the query statement to obtain multiple query statement fragments.
[0039] In this embodiment, the query statement is sharded according to the current number of computing nodes and the query time period. This allows the query data to be read in the form of query statement shards and read into different computing nodes, thereby improving the data reading efficiency.
[0040] Furthermore, in one embodiment of this application, the step of dividing a query statement into multiple query statement fragments based on the number of nodes and the query time period corresponding to the query statement includes: dividing the query time period corresponding to the query statement into multiple query statement fragments on an average basis according to the number of nodes; wherein the number of query statement fragments corresponding to each computing node is equal.
[0041] In this embodiment, the query statement is divided into segments according to the query time period based on the current number of computing nodes. The segmentation rule is to divide the query time period into equal parts, that is, to divide the total query time range into n equal parts according to the rule of left open and right closed, where n is a multiple of the number of nodes m, and the specific multiple can be set by the user.
[0042] By dividing the query statement into segments according to the above method of evenly dividing the query time period, and allocating an equal number of query statement segments to each computing node, the amount of query data allocated to each computing node is relatively balanced, thereby improving the speed of word frequency calculation.
[0043] Furthermore, in one embodiment of this application, the step of the computing node reading query data according to the query statement fragments includes: the computing node reading query data from the ClickHouse memory according to the query statement fragments.
[0044] In this embodiment, the columnar storage ClickHouse, characterized by its fast speed and multiple indexes, is introduced as the underlying data storage source. Since the volume of query data is large, MySQL or Elasticsearch cannot meet the requirements for fast queries. Therefore, storing the query data in ClickHouse storage ensures fast query speeds.
[0045] Furthermore, in one embodiment of this application, the step of storing the first word frequency information into the memory module includes: storing the first word frequency information into the memory module in the form of a WordSet.
[0046] In this embodiment, after the first word frequency information is calculated, the first word frequency information is stored in the memory module in the form of WordSet, and can be remotely retrieved to the specified computing node. Thus, the intermediate results of the calculation can be efficiently transferred to the next computing node, improving the calculation speed.
[0047] It should be noted that WordSet is a logical concept, namely a HashMap, used to store word frequency information.
[0048] Furthermore, in one embodiment of this application, the data processing method further includes: the computing node statistically analyzes the third word frequency information and the fourth word frequency information of the second keyword in the (i-1)th query time period in the word segmentation result, and forms the fifth word frequency information according to a preset format, where i is a positive integer greater than or equal to 2; the computing node stores the fifth word frequency information in the memory module; after the fifth word frequency information statistically analyzed by multiple computing nodes is stored in the memory module, the computing node merges the fifth word frequency information in the memory module to generate the sixth word frequency information; the computing node calculates the word frequency growth rate of the second keyword based on the sixth word frequency information.
[0049] In this embodiment, the word frequency statistics of the second keyword are performed in each query time period. Specifically, within a query time period, the query statement is segmented, the corresponding query data is read, the query data is segmented to obtain the segmentation results, and the word frequency information of the second keyword in the segmentation results is calculated. Then, the third and fourth word frequency information of the second keyword in two adjacent query time periods (i.e., the (i-1)th query time period and the ith query time period) are recorded according to the preset format "second keyword:(p:q)", that is, recorded as the fifth word frequency information of the second keyword, where p is the third word frequency information of the second keyword in the (i-1)th query time period, and q is the fourth word frequency information of the second keyword in the ith query time period. For example, Apple:(1:5) means that the word frequency of the second keyword Apple is 1 in the previous query time period and 5 in the current query time period. The fifth word frequency information of the second keyword is then stored in the memory module.
[0050] Furthermore, after the fifth word frequency information collected from multiple computing nodes is stored in the memory module, the fifth word frequency information in the memory module is merged to generate the sixth word frequency information. The sixth word frequency information includes the total word frequency information of the second keyword in the (i-1)th query time period and the total word frequency information of the second keyword in the ith query time period. For example, as... Figure 5As shown, the fifth word frequency information of the second keyword "Cat" corresponding to the first calculation node is (4:7), the fifth word frequency information of the second keyword "Cat" corresponding to the second calculation node is (3:6), and the fifth word frequency information of the second keyword "Cat" corresponding to the third calculation node is (2:8). Therefore, the sixth word frequency information of the merged second keyword "Cat" is (9:21). Based on the total word frequency information of the (i-1)th query time period and the total word frequency information of the second keyword in the i-th query time period, the word frequency growth rate of the second keyword is calculated, which is 133%.
[0051] It should be noted that keywords with the same hash value or first letter can be grouped together, and each computing node can process one or more groups of keywords. For example, ... Figure 5 As shown, keywords with the same first letter are grouped together. The Executor of the first computing node can process keyword groups that start with the letter A and keyword groups that start with the letter D.
[0052] In this embodiment, the word frequency of a certain keyword in the current time interval and the previous time interval is counted. In this way, the number of times the keyword appears before and after is counted in a single Executor of the calculation node. Then, the word frequency growth rate can be calculated based on the difference between the two time intervals. It is not necessary to obtain all the word frequency data in the two time intervals before calculating the word frequency, thereby improving the calculation speed of the word frequency growth rate and enabling the rapid identification of soaring words (i.e., keywords with a large word frequency growth rate).
[0053] It should be noted that the data processing method provided in this application embodiment can be executed by a data processing system, or by a control module within that data processing system for executing the data processing method. This application embodiment uses the execution of the data processing method by a data processing system as an example to illustrate the data processing system provided in this application embodiment.
[0054] This application provides a data processing system, such as... Figure 2 As shown, the data processing system 200 includes a memory module 202, a scheduling node 204, and multiple computing nodes 206;
[0055] The scheduling node 204 is used to: obtain the query statement of the task to be executed, divide the query statement into multiple query statement fragments, and distribute the multiple query statement fragments to multiple computing nodes.
[0056] The computing node 206 is used to: read query data in segments according to the query statement, perform word segmentation on the query data to obtain multiple word segmentation results, count the first word frequency information of the first keyword in the word segmentation results, and store the first word frequency information in the memory module; and after the first word frequency information counted by multiple computing nodes is stored in the memory module, merge the first word frequency information in the memory module to generate second word frequency information, and store the second word frequency information in the target memory.
[0057] In this embodiment, the data processing system includes a memory module, a scheduling node, multiple computing nodes, and a client. The client receives a query statement input by the user for a task to be executed. The scheduling node obtains the query statement and divides it into multiple query statement fragments. The scheduling node then sends each query statement fragment to a corresponding computing node, which performs data reading and word frequency calculation. Specifically, on the computing node, query data is read according to the query statement fragments. Then, according to the specific word segmentation algorithm input by the user on the client, the read query data is segmented to obtain multiple word segmentation results. The word segmentation results include multiple keywords. The first word frequency information of the first keyword in the word segmentation results is statistically analyzed and stored in the memory module. It should be noted that there can be one or more computing nodes to obtain the first keyword after word segmentation. After word frequency statistics are performed on all one or more computing nodes, the first word frequency information of the first keyword is stored in the memory module. Further, the first word frequency information stored in the memory module is merged and summarized to obtain the second word frequency information of the first keyword, and the second word frequency information is stored in the target storage (i.e., MySQL). By employing the above methods, on the one hand, query data is read in fragmented form and distributed to different computing nodes, thereby improving data reading efficiency. On the other hand, the word frequency statistics results are stored in memory, allowing computing nodes to remotely read data based on keywords, avoiding the time overhead of writing word frequency statistics results back to disk and thus improving computing speed.
[0058] Furthermore, in one embodiment of this application, the scheduling node 204 is specifically used to obtain the number of nodes of multiple computing nodes, and to divide the query statement into multiple query statement fragments based on the number of nodes and the query time period corresponding to the query statement.
[0059] Furthermore, in one embodiment of this application, the scheduling node 204 is specifically used to divide the query time period corresponding to the query statement into multiple query statement fragments according to the number of nodes; wherein, the number of query statement fragments corresponding to each computing node is equal.
[0060] Furthermore, in one embodiment of this application, computing node 206 is specifically used to read query data from the ClickHouse memory according to the query statement sharding.
[0061] Furthermore, in one embodiment of this application, computing node 206 is specifically used to store the first word frequency information in the form of a WordSet into a memory module.
[0062] Furthermore, in one embodiment of this application, the computing node 206 is also used to: count the third word frequency information and the fourth word frequency information of the second keyword in the (i-1)th query time period in the word segmentation result, and form the fifth word frequency information according to the third word frequency information and the fourth word frequency information in a preset format, where i is a positive integer greater than or equal to 2; store the fifth word frequency information in the memory module; merge the fifth word frequency information in the memory module to generate the sixth word frequency information; and calculate the word frequency growth rate of the second keyword based on the sixth word frequency information.
[0063] In one specific embodiment of this application, the architecture of the data processing system is as follows: Figure 3 As shown, it includes several parts: client program, scheduling layer, computing layer, and data layer. The functions of each part are as follows:
[0064] (1) The client program is the initiating node for the computation task. The main program of the computation task starts here, and the specific task content to be executed is defined here.
[0065] (2) The scheduling layer is responsible for the management and scheduling of computing tasks. After the client program starts the program, it submits the task content to the scheduling layer, which then segments the task and distributes it to various nodes in the computing layer. The worker nodes in the scheduling layer adopt a master-slave mode, including one master node and one slave node. ZooKeeper is used for master-slave node management to prevent single point of failure (SPOF).
[0066] (3) The computation layer is responsible for the specific execution of computation tasks. After receiving the tasks distributed by the scheduling layer, the computation layer completes data reading and computation in the nodes. Here, a Worker refers to a single physical computation node, and the specific computation tasks are executed in the Executor. For word frequency calculation tasks, data needs to be exchanged between multiple Executors to complete the counting of individual words, so Workers can interact with each other.
[0067] (4) Data layer, responsible for data storage, including data to be read (i.e., query data) and word frequency data after calculation. Among them, the data to be read is large in volume, and MySQL or ES cannot meet the needs of fast querying, so the data to be read is stored in ClickHouse storage. ClickHouse storage is a columnar database with features such as high speed and multiple indexes. The word frequency data after statistics are completed is stored in MySQL.
[0068] The process of word frequency calculation includes:
[0069] (1) The client program submits a computation task to the scheduling layer. The computation task includes: data query statement, word segmentation algorithm, word segmentation rules, and the range of values for the final result.
[0070] (2) After receiving the computation task submitted by the client program, the scheduling layer divides the data query statement of the computation task into time slices according to the current number of Executors m. The slicing rule is to divide the time equally, that is, to divide the total query time range into n equal parts according to the rule of left-open and right-closed (n is a multiple of m, and the multiple can be set. The purpose of this setting is to make the amount of data received by the Worker nodes relatively balanced). Then, the slicing query statement is distributed to the Worker nodes of the computation layer (each query statement slice is distributed to only one Worker node), and the task distribution status is recorded in the scheduling layer.
[0071] (3) Figure 4 As shown, after receiving a task, the Worker node starts the Executor and reads data from ClickHouse. Then, based on the word segmentation algorithm submitted by the client program, it performs word segmentation on the read data, counts the words on its own node, and stores the count results in memory as WordSets. After the counting is complete, it reports the completion of the Map task to the scheduling layer.
[0072] (4) After all nodes complete the Map task, the scheduling layer issues Reduce tasks to the Executors, assigning WordSets with the same key to a specific Executor for merge counting operations. For example, all WordSet1s are assigned to one Executor for merge counting, all WordSet2s to one Executor for merge counting, and all WordSet3s to one Executor for merge counting. In the Reduce task, each Executor retrieves WordSet key values independently, and the allocation is based on the HashValue method, i.e., according to the hash value. After the merge counting is completed, the calculation results are summarized and written to MySQL, and the scheduling layer is notified that the task is complete. The scheduling layer then sends the task completion information back to the client program. At this point, the entire calculation task is finished.
[0073] like Figure 5 As shown, the specific word frequency calculation process for determining trending words is as follows:
[0074] (1) Perform data statistics for each word segmentation on the Executor, and calculate the word frequency of the current time period and the word frequency of the previous time period. This application embodiment proposes a WordSet object format of "KeyWord:(PreWordNum,CurrentWordNum)". Taking Apple:(1:5) data as an example, it indicates that the word Apple appeared once in the previous time period and five times in the current time period. After the Map task is completed, the scheduling layer Master node is notified that the Map task is complete.
[0075] (2) When the scheduling layer Master node receives the Map task completion message, it informs the downstream Executors that they can start the Reduce task and divides the range of WordSets to be pulled by the Executors (divided by hash method). For example, Executor1 pulls WordSets that start with the first letter A and the first letter D, Executor2 pulls WordSets that start with the first letter B and the first letter E, and so on, to ensure that WordSets corresponding to the same key value are not distributed to multiple Executors for Reduce operation, and that all Map results are shuffled to the next computing node.
[0076] (3) The Executor counts the frequency of a word in the current time period and the previous time period in the Reduce task. In this way, the frequency of a word before and after the previous time period is counted by a single Executor. The Executor can calculate the word frequency growth rate based on the difference between the word frequencies before and after the previous time period, and then determine the soaring words based on the word frequency growth rate.
[0077] This application optimizes word frequency analysis performance from two aspects: data access and computation methods. First, it introduces the columnar storage engine ClickHouse as the underlying data storage source and uses time-sharing data partitioning to read data into different nodes, improving data reading efficiency. Then, the system introduces distributed worker nodes and continues the MapReduce design approach. That is, data computation is divided into Map tasks and Reduce tasks, which are distributed to multiple worker nodes. Task distribution and control are handled by the scheduling layer Master node. Furthermore, after a Map task is completed, the Map result is no longer returned to disk for merging but is stored in memory as a WordSet and remotely pulled to a specified worker node in the Reduce task. This allows intermediate results to be efficiently transferred to the next computation step, improving computation speed and ensuring system scalability. In addition, for the calculation of trending words, a data shuffle scheme based on WordSet is proposed, enabling distributed calculation of word frequency growth rate.
[0078] The data processing system 200 in this embodiment can be a device, or a component, integrated circuit, or chip in a terminal. The device can be a mobile electronic device or a non-mobile electronic device. For example, mobile electronic devices can be mobile phones, tablets, laptops, PDAs, in-vehicle electronic devices, wearable devices, ultra-mobile personal computers (UMPCs), netbooks, or personal digital assistants (PDAs), etc., while non-mobile electronic devices can be servers, network attached storage (NAS), personal computers (PCs), televisions (TVs), ATMs, or self-service machines, etc. This embodiment does not impose specific limitations.
[0079] The data processing system 200 in this embodiment can be a device with an operating system. This operating system can be Android, iOS, or other possible operating systems; this embodiment does not specifically limit its use.
[0080] The data processing system 200 provided in this application embodiment can achieve... Figure 1 The various processes implemented in the data processing method embodiments are not described in detail here to avoid repetition.
[0081] Optional, such as Figure 6 As shown, this application embodiment also provides an electronic device 600, including a processor 602, a memory 604, and a program or instructions stored in the memory 604 and executable on the processor 602. When the program or instructions are executed by the processor 602, they implement the various processes of the above-described data processing method embodiments and achieve the same technical effects. To avoid repetition, they will not be described again here.
[0082] It should be noted that the electronic devices in the embodiments of this application include the mobile electronic devices and non-mobile electronic devices described above.
[0083] Figure 7 A schematic diagram of the hardware structure of an electronic device to implement an embodiment of this application.
[0084] The electronic device 700 includes, but is not limited to, components such as: radio frequency unit 702, network module 704, audio output unit 706, input unit 708, sensor 710, display unit 712, user input unit 714, interface unit 716, memory 718, and processor 720.
[0085] Those skilled in the art will understand that the electronic device 700 may also include a power supply (such as a battery) for supplying power to various components. The power supply may be logically connected to the processor 720 through a power management system, thereby enabling functions such as managing charging, discharging, and power consumption through the power management system. Figure 7 The electronic device structure shown does not constitute a limitation on the electronic device. The electronic device may include more or fewer components than shown, or combine certain components, or have different component arrangements, which will not be elaborated here.
[0086] The data processing system of the processor 720 includes a memory module, a scheduling node, and multiple computing nodes. The scheduling node is used to: obtain the query statement of the task to be executed, divide the query statement into multiple query statement fragments, and distribute the multiple query statement fragments to multiple computing nodes. The computing nodes are used to: read the query data according to the query statement fragments, perform word segmentation on the query data to obtain multiple word segmentation results, count the first word frequency information of the first keyword in the word segmentation results, and store the first word frequency information in the memory module; after the first word frequency information counted by multiple computing nodes is stored in the memory module, merge the first word frequency information in the memory module to generate second word frequency information, and store the second word frequency information in the target memory.
[0087] In this embodiment, the data processing system includes a memory module, a scheduling node, multiple computing nodes, and a client. The client receives a query statement input by the user for a task to be executed. The scheduling node obtains the query statement and divides it into multiple query statement fragments. The scheduling node then sends each query statement fragment to a corresponding computing node, which performs data reading and word frequency calculation. Specifically, on the computing node, query data is read according to the query statement fragments. Then, according to the specific word segmentation algorithm input by the user on the client, the read query data is segmented to obtain multiple word segmentation results. The word segmentation results include multiple keywords. The first word frequency information of the first keyword in the word segmentation results is statistically analyzed and stored in the memory module. It should be noted that there can be one or more computing nodes to obtain the first keyword after word segmentation. After word frequency statistics are performed on all one or more computing nodes, the first word frequency information of the first keyword is stored in the memory module. Further, the first word frequency information stored in the memory module is merged and summarized to obtain the second word frequency information of the first keyword, and the second word frequency information is stored in the target storage (i.e., MySQL). By employing the above methods, on the one hand, query data is read in fragmented form and distributed to different computing nodes, thereby improving data reading efficiency. On the other hand, the word frequency statistics results are stored in memory, allowing computing nodes to remotely read data based on keywords, avoiding the time overhead of writing word frequency statistics results back to disk and thus improving computing speed.
[0088] Furthermore, in one embodiment of this application, the scheduling node is specifically used to obtain the number of nodes of multiple computing nodes, and to divide the query statement into multiple query statement fragments based on the number of nodes and the query time period corresponding to the query statement.
[0089] Furthermore, in one embodiment of this application, the scheduling node is specifically used to divide the query time period corresponding to the query statement into multiple query statement fragments according to the number of nodes; wherein, the number of query statement fragments corresponding to each computing node is equal.
[0090] Furthermore, in one embodiment of this application, the computing node is specifically used to read query data from the ClickHouse memory according to the query statement shards.
[0091] Furthermore, in one embodiment of this application, the computing node is specifically used to store the first word frequency information in the form of a WordSet into the memory module.
[0092] Furthermore, in one embodiment of this application, the computing node is also used to: count the third word frequency information and the fourth word frequency information of the second keyword in the (i-1)th query time period in the word segmentation result, and form the fifth word frequency information according to the third word frequency information and the fourth word frequency information in a preset format, where i is a positive integer greater than or equal to 2; store the fifth word frequency information in the memory module; merge the fifth word frequency information in the memory module to generate the sixth word frequency information; and calculate the word frequency growth rate of the second keyword based on the sixth word frequency information.
[0093] It should be understood that, in the embodiments of this application, the radio frequency unit 702 can be used to send and receive information or signals during a call, specifically, to receive downlink data from the base station or to send uplink data to the base station. The radio frequency unit 702 includes, but is not limited to, an antenna, at least one amplifier, a transceiver, a coupler, a low-noise amplifier, a duplexer, etc.
[0094] Network module 704 provides users with wireless broadband internet access, such as helping users send and receive emails, browse web pages, and access streaming media.
[0095] The audio output unit 706 can convert audio data received by the radio frequency unit 702 or the network module 704 or stored in the memory 718 into audio signals and output them as sound. Furthermore, the audio output unit 706 can also provide audio output related to specific functions performed by the electronic device 700 (e.g., call signal reception sound, message reception sound, etc.). The audio output unit 706 includes a speaker, a buzzer, and a receiver, etc.
[0096] Input unit 708 is used to receive audio or video signals. Input unit 708 may include a graphics processing unit (GPU) 7082 and a microphone 7084. GPU 7082 processes image data of still images or videos acquired by an image capture device (such as a camera) in video capture mode or image capture mode. The processed image frames may be displayed on display unit 712, stored in memory 718 (or other storage medium), or transmitted via radio frequency unit 702 or network module 704. Microphone 7084 can receive sound and process the sound into audio data. The processed audio data can be converted into a format that can be transmitted to a mobile communication base station via radio frequency unit 702 in telephone call mode.
[0097] The electronic device 700 also includes at least one sensor 710, such as a fingerprint sensor, pressure sensor, iris sensor, molecular sensor, gyroscope, barometer, hygrometer, thermometer, infrared sensor, light sensor, motion sensor, and other sensors.
[0098] The display unit 712 is used to display information input by the user or information provided to the user. The display unit 712 may include a display panel 7122, which may be configured in the form of a liquid crystal display, an organic light-emitting diode, or the like.
[0099] User input unit 714 can be used to receive input numeric or character information, and generate key signal inputs related to user settings and function control of electronic devices. Specifically, user input unit 714 includes touch panel 7142 and other input devices 7144. Touch panel 7142, also known as touch screen, can collect touch operations by the user on or near it. Touch panel 7142 can include two parts: a touch detection device and a touch controller. The touch detection device detects the user's touch position and the signal generated by the touch operation, and transmits the signal to the touch controller; the touch controller receives touch information from the touch detection device, converts it into touch point coordinates, and sends it to processor 720, and receives and executes commands from processor 720. Other input devices 7144 may include, but are not limited to, physical keyboards, function keys (such as volume control buttons, power buttons, etc.), trackballs, mice, and joysticks, which will not be described in detail here.
[0100] Furthermore, the touch panel 7142 can cover the display panel 7122. When the touch panel 7142 detects a touch operation on or near it, it transmits the information to the processor 720 to determine the type of touch event. Subsequently, the processor 720 provides corresponding visual output on the display panel 7122 according to the type of touch event. The touch panel 7142 and the display panel 7122 can be two separate components or integrated into one component.
[0101] Interface unit 716 serves as an interface for connecting external devices to electronic device 700. For example, external devices may include a wired or wireless headphone port, an external power supply (or battery charger) port, a wired or wireless data port, a memory card port, a port for connecting a device with an identification module, an audio input / output (I / O) port, a video I / O port, a headphone port, and so on. Interface unit 716 can be used to receive input from external devices (e.g., data, power, etc.) and transmit the received input to one or more components within electronic device 700, or it can be used to transmit data between electronic device 700 and external devices.
[0102] The memory 718 can be used to store software programs and various data. The memory 718 may primarily include a program storage area and a data storage area. The program storage area may store the operating system, application programs required for at least one function (such as sound playback, image playback, etc.), etc.; the data storage area may store data created based on the use of the mobile terminal (such as audio data, phonebook, etc.). Furthermore, the memory 718 may include high-speed random access memory, and may also include non-volatile memory, such as at least one disk storage device, flash memory device, or other volatile solid-state storage device.
[0103] The processor 720 performs various functions and processes data of the electronic device 700 by running or executing software programs and / or modules stored in the memory 718 and calling data stored in the memory 718, thereby providing overall monitoring of the electronic device 700. The processor 720 may include one or more processing units; preferably, the processor 720 may integrate an application processor and a modem processor, wherein the application processor mainly handles the operating system, user interface, and applications, and the modem processor mainly handles wireless communication.
[0104] This application also provides a readable storage medium storing a program or instructions. When the program or instructions are executed by a processor, they implement the various processes of the above-described data processing method embodiments and achieve the same technical effects. To avoid repetition, they will not be described again here.
[0105] The processor is the processor in the electronic device described in the above embodiments. The readable storage medium includes computer-readable storage media, such as computer read-only memory (ROM), random access memory (RAM), magnetic disk, or optical disk.
[0106] This application also provides a chip, which includes a processor and a communication interface. The communication interface and the processor are coupled. The processor is used to run programs or instructions to implement the various processes of the above data processing method embodiments and can achieve the same technical effect. To avoid repetition, it will not be described again here.
[0107] It should be understood that the chip mentioned in the embodiments of this application may also be referred to as a system-on-a-chip, system chip, chip system, or system-on-a-chip, etc.
[0108] It should be noted that, in this document, the terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such a process, method, article, or apparatus. Without further limitations, an element defined by the phrase "comprising one..." does not exclude the presence of other identical elements in the process, method, article, or apparatus that includes that element. Furthermore, it should be noted that the scope of the methods and apparatuses in the embodiments of this application is not limited to performing functions in the order shown or discussed, but may also include performing functions substantially simultaneously or in the reverse order, depending on the functions involved. For example, the described methods may be performed in a different order than described, and various steps may be added, omitted, or combined. Additionally, features described with reference to certain examples may be combined in other examples.
[0109] Through the above description of the embodiments, those skilled in the art can clearly understand that the methods of the above embodiments can be implemented by means of software plus necessary general-purpose hardware platforms. Of course, they can also be implemented by hardware, but in many cases the former is a better implementation method. Based on this understanding, the technical solution of this application, in essence, or the part that contributes to the prior art, can be embodied in the form of a computer software product. This computer software product is stored in a storage medium (such as ROM / RAM, magnetic disk, optical disk) and includes several instructions to cause a terminal (which may be a mobile phone, computer, server, or network device, etc.) to execute the methods described in the various embodiments of this application.
[0110] The embodiments of this application have been described above with reference to the accompanying drawings. However, this application is not limited to the specific embodiments described above. The specific embodiments described above are merely illustrative and not restrictive. Those skilled in the art can make many other forms under the guidance of this application without departing from the spirit and scope of the claims, and all of these forms are within the protection scope of this application.
Claims
1. A data processing method, characterized in that, A data processing system, the data processing system including a memory module, a scheduling node and multiple computing nodes, the data processing method including: The scheduling node obtains the query statement of the task to be executed and divides the query statement into multiple query statement fragments. The scheduling node distributes multiple query statements to multiple computing nodes in shards; The computing node reads query data in segments according to the query statement, performs word segmentation on the query data to obtain multiple word segmentation results, counts the first word frequency information of the first keyword in the word segmentation results, and stores the first word frequency information in the memory module. After the first word frequency information collected by multiple computing nodes is stored in the memory module, the computing node merges the first word frequency information in the memory module to generate second word frequency information. The computing node stores the second word frequency information into the target memory, and The data processing method further includes: The computing node counts the third word frequency information and the fourth word frequency information of the second keyword in the word segmentation result during the (i-1)th query time period, and then forms the fifth word frequency information by combining the third word frequency information and the fourth word frequency information according to a preset format, where i is a positive integer greater than or equal to 2. The computing node stores the fifth word frequency information into the memory module; After the fifth word frequency information collected by multiple computing nodes is stored in the memory module, the computing node merges the fifth word frequency information in the memory module to generate the sixth word frequency information. The computing node calculates the word frequency growth rate of the second keyword based on the sixth word frequency information.
2. The data processing method according to claim 1, characterized in that, The step of dividing the query statement into multiple query statement fragments includes: Obtain the number of nodes for the multiple computing nodes; Based on the number of nodes and the query time period corresponding to the query statement, the query statement is divided into multiple query statement fragments.
3. The data processing method according to claim 2, characterized in that, The step of dividing the query statement into multiple query statement fragments based on the number of nodes and the query time period corresponding to the query statement includes: Based on the number of nodes, the query time period corresponding to the query statement is divided into multiple query statement fragments on an average basis. In this case, the number of query statement shards corresponding to each computing node is equal.
4. The data processing method according to claim 1, characterized in that, The step of the computing node reading query data in shards according to the query statement includes: The computing node shards the query data according to the query statement and reads the query data from the ClickHouse memory.
5. The data processing method according to claim 1, characterized in that, The step of storing the first word frequency information into the memory module includes: The first word frequency information is stored in the memory module in the form of a WordSet.
6. A data processing system, characterized in that, It includes a memory module, a scheduling node, and multiple computing nodes; The scheduling node is used to: obtain the query statement of the task to be executed, divide the query statement into multiple query statement fragments, and distribute the multiple query statement fragments to multiple computing nodes. The computing node is used to: read query data in segments according to the query statement; perform word segmentation on the query data to obtain multiple word segmentation results; count the first word frequency information of the first keyword in the word segmentation results and store the first word frequency information in the memory module; and after the first word frequency information counted by multiple computing nodes is stored in the memory module, merge the first word frequency information in the memory module to generate second word frequency information and store the second word frequency information in the target memory. The computing node is also used for: The third word frequency information and the fourth word frequency information of the second keyword in the word segmentation result in the (i-1)th query time period are statistically analyzed, and the third word frequency information and the fourth word frequency information are combined into the fifth word frequency information according to a preset format, where i is a positive integer greater than or equal to 2; The fifth word frequency information is stored in the memory module; The fifth word frequency information in the memory module is merged to generate the sixth word frequency information; Based on the sixth word frequency information, calculate the word frequency growth rate of the second keyword.
7. The data processing system according to claim 6, characterized in that, The scheduling node is specifically used to obtain the number of nodes of the multiple computing nodes, and to divide the query statement into multiple query statement fragments based on the number of nodes and the query time period corresponding to the query statement.
8. The data processing system according to claim 7, characterized in that, The scheduling node is specifically used to divide the query time period corresponding to the query statement into multiple query statement fragments according to the number of nodes. In this case, the number of query statement shards corresponding to each computing node is equal.
9. The data processing system according to claim 6, characterized in that, The computing node is specifically used to read the query data from the ClickHouse memory according to the sharding of the query statement.
10. The data processing system according to claim 6, characterized in that, The computing node is specifically used to store the first word frequency information in the form of a WordSet into the memory module.
11. An electronic device, characterized in that, It includes a processor, a memory, and a program or instructions stored in the memory and executable on the processor, wherein the program or instructions, when executed by the processor, implement the steps of the data processing method as described in any one of claims 1 to 5.
12. A readable storage medium having a program or instructions stored thereon, characterized in that, When the program or instructions are executed by the processor, they implement the steps of the data processing method as described in any one of claims 1 to 5.