Marketing data processing method and device, computer equipment and storage medium
Through distributed computing technology and RDD technology processing marketing data, the problem of limited resources of a single computer is solved, efficient data processing and real-time computing are realized, and high concurrency and real-time requirements are met.
Patent Information
- Application Number
- CN202410958261.8
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2024-07-17
- Publication Date
- 2024-12-10
AI Technical Summary
The existing marketing data processing technology relies on limited resources of a single computer, which leads to slow data processing speed and difficulty in processing large amounts of data, and cannot guarantee the real-time and high concurrent computing requirements of marketing data.
Using distributed computing technology, a distributed data set is generated through RDD technology, and divided it into multiple partitioned data subsets according to preset strategies, distributed to available computing nodes through the network for calculation processing, using Kafka message queues and broadcast variables Optimize data distribution.
It significantly improves data processing speed, meets the high concurrency and real-time marketing rule calculation needs, and ensures the effectiveness of marketing data.
Smart Images

Figure CN119106049A_ABST
Abstract
Description
Technical Field
[0001] The present invention relates to the technical field of marketing data processing, and in particular to a marketing data processing method, device, computer equipment and storage medium. Background Art
[0002] The existing marketing data is processed by traditional batch processing programs and traditional real-time processing programs, and the marketing data is calculated and stored in the database. Traditional batch processing programs and traditional real-time processing programs use the resources of a single computer for processing, but the computing resources of a single computer are limited, resulting in slow data processing speed, difficulty in processing large amounts of data, and failure to ensure the effectiveness of marketing data. For marketing systems, traditional technologies have been unable to meet the growing amount of data and the increasing number of marketing rule calculations. Summary of the invention
[0003] Based on this, it is necessary to provide a marketing data processing method, device, computer equipment and storage medium to address the above technical problems.
[0004] A marketing data processing method, comprising:
[0005] Detect whether the application pushes marketing data to the message queue;
[0006] When it is detected that the application pushes the marketing data to the message queue, acquiring the marketing data;
[0007] Based on distributed computing technology, the marketing data is used to generate a distributed data set through RDD technology;
[0008] Dividing the distributed data set into a plurality of partition data subsets according to a preset strategy;
[0009] Each of the partitioned data subsets is distributed to available computing nodes via a network for computing and processing.
[0010] In one embodiment, the message queue is a Kafka message queue.
[0011] In one embodiment, the step of distributing each of the partitioned data subsets to available computing nodes for computing and processing via a network includes:
[0012] Obtaining the storage location of each partition data subset;
[0013] Based on the storage location of each partition data subset, each partition data subset is distributed to an available computing node closest to the partition data subset through a network for computing and processing.
[0014] In one of the embodiments, the preset strategy includes clustering rules.
[0015] In one embodiment, the step of distributing each of the partitioned data subsets to available computing nodes for computing and processing via a network includes:
[0016] Each partition data subset is distributed to available computing nodes for computing and processing by using broadcast variables over the network.
[0017] In one embodiment, the step of distributing each of the partitioned data subsets to available computing nodes for computing and processing via a network includes:
[0018] Detecting whether the data amount of each partition data subset is less than a preset data amount;
[0019] When the data amount of the partitioned data subset is smaller than the preset data amount, the partitioned data subset smaller than the preset data amount is distributed to available computing nodes for computing and processing via the network using broadcast variables.
[0020] A marketing data processing device, comprising:
[0021] Data push detection module, used to detect whether the application pushes marketing data to the message queue;
[0022] A marketing data acquisition module, configured to acquire the marketing data when detecting that the application pushes the marketing data to the message queue;
[0023] A distributed data set generation module, used to generate a distributed data set from the marketing data using RDD technology based on distributed computing technology;
[0024] A data subset partitioning module, used for partitioning the distributed data set into a plurality of partition data subsets according to a preset strategy;
[0025] The data subset distribution module is used to distribute each of the partitioned data subsets to available computing nodes through the network for computing and processing.
[0026] In one embodiment, the data subset distribution module includes:
[0027] A storage location acquisition unit, used to acquire the storage location of each partition data subset;
[0028] The distribution unit is used to distribute each partition data subset to an available computing node closest to the partition data subset through a network based on the storage location of each partition data subset for computing and processing.
[0029] A computer device comprises a memory and a processor, wherein the memory stores a computer program, and wherein the processor implements the following steps when executing the computer program:
[0030] Detect whether the application pushes marketing data to the message queue;
[0031] When it is detected that the application pushes the marketing data to the message queue, acquiring the marketing data;
[0032] Based on distributed computing technology, the marketing data is used to generate a distributed data set through RDD technology;
[0033] Dividing the distributed data set into a plurality of partition data subsets according to a preset strategy;
[0034] Each of the partitioned data subsets is distributed to available computing nodes via a network for computing and processing.
[0035] A computer-readable storage medium stores a computer program, which, when executed by a processor, implements the following steps:
[0036] Detect whether the application pushes marketing data to the message queue;
[0037] When it is detected that the application pushes the marketing data to the message queue, acquiring the marketing data;
[0038] Based on distributed computing technology, the marketing data is used to generate a distributed data set through RDD technology;
[0039] Dividing the distributed data set into a plurality of partition data subsets according to a preset strategy;
[0040] Each of the partitioned data subsets is distributed to available computing nodes via a network for computing and processing.
[0041] The above-mentioned marketing data processing method, device, computer equipment and storage medium realize distributed computing through RDD technology based on high-performance distributed computing capabilities, greatly improve the data processing speed, meet the high concurrency and real-time marketing rule computing needs, and effectively ensure the effectiveness of marketing data. BRIEF DESCRIPTION OF THE DRAWINGS
[0042] Figure 1 A flowchart of a marketing data processing method in one embodiment;
[0043] Figure 2 is a structural block diagram of a marketing data processing device in one embodiment;
[0044] Figure 3FIG. 4 is a diagram showing the internal structure of a computer device in one embodiment. DETAILED DESCRIPTION
[0045] In order to make the purpose, technical solution and advantages of the present application more clearly understood, the present application is further described in detail below in conjunction with the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are only used to explain the present application and are not used to limit the present application.
[0046] Embodiment 1
[0047] In this embodiment, Figure 1 As shown, a marketing data processing method is provided, which includes:
[0048] Step 110 , detecting whether the application pushes marketing data to the message queue.
[0049] In this embodiment, the event of an application pushing marketing data to a message queue is monitored in real time. The application may also be called an application system, which is used to generate marketing data or process marketing data. During the processing, the marketing data needs to be processed concurrently through a message queue.
[0050] In one embodiment, the message queue is a Kafka message queue. The Kafka message queue is a distributed message queue based on a publish / subscribe mode, and is mainly used in the field of real-time processing of big data.
[0051] Step 120: When it is detected that the application program pushes the marketing data to the message queue, the marketing data is obtained.
[0052] In this embodiment, when an event that an application pushes marketing data to a message queue is detected, the marketing data is acquired.
[0053] Step 130: Generate a distributed data set from the marketing data using RDD technology based on distributed computing technology.
[0054] In this embodiment, the marketing data is transformed into a distributed dataset using RDD (Resilient Distributed Datasets) based on distributed computing technology. The distributed dataset provides a highly restricted shared memory model.
[0055] Step 140: divide the distributed data set into a plurality of partition data subsets according to a preset strategy.
[0056] In this embodiment, the distributed data set is divided into a plurality of partitioned data subsets capable of distributed computing according to a preset strategy, and the plurality of partitioned data subsets constitute the distributed data set. Each partitioned data subset can be individually computed and processed, so that each partitioned data subset can be distributed to different computer nodes for computing and processing.
[0057] Step 150: Distribute each of the partitioned data subsets to available computing nodes through the network for computing and processing.
[0058] In this embodiment, the computing node is a computer node, which can be implemented by a single computer such as a single server, or by a server cluster composed of multiple servers. Each partition data subset is distributed to each computing node in the network one by one through the network, and each computing node performs computing processing on the partition data subset. The computing of the partition data subset by the distributed computing nodes can effectively improve the computing processing speed and meet the high concurrency requirements.
[0059] In the above embodiment, traditional batch processing and real-time processing are no longer used. Instead, distributed computing is realized through RDD technology based on high-performance distributed computing capabilities, which greatly improves the data processing speed and meets the high concurrency and real-time marketing rule computing requirements, effectively ensuring the effectiveness of marketing data.
[0060] In one embodiment, the step of distributing each of the partitioned data subsets to available computing nodes for computing and processing via a network includes:
[0061] Obtaining the storage location of each partition data subset;
[0062] Based on the storage location of each partition data subset, each partition data subset is distributed to an available computing node closest to the partition data subset through a network for computing and processing.
[0063] In this embodiment, when the distributed data set is divided, a corresponding storage location is selected for each partitioned data subset. It should be understood that each computing node has a storage node. Since each computing node is distributed in different physical locations, the corresponding storage nodes are also distributed in different locations. Therefore, in this embodiment, according to the storage location of each partitioned data subset, the storage node corresponding to the location of the storage node closest to each storage location is selected, and the computing node corresponding to each storage node is determined. Each partitioned data subset is distributed to each determined computing node through the network. In this way, the partitioned data subset can be processed by the computing node close to the storage location, maintaining the locality of the data and improving storage efficiency.
[0064] In one embodiment, the preset strategy includes a clustering rule.
[0065] In this embodiment, clustering rules are a data analysis method, and its principle is to classify similar things into one category. In the process of data analysis, the use of clustering rules can help discover patterns and laws in the data, so as to better understand the data. The principle of clustering rules is to divide the data set into several categories so that the data similarity within the same category is high and the similarity between different categories is low. The process of clustering rules includes: first determining the number of clusters, and then assigning the data to different categories. The number of clusters can be determined by trial and error, or by professional knowledge and experience. In the process of clustering, it is necessary to select a suitable similarity measurement method. Commonly used similarity measurement methods include Euclidean distance, Manhattan distance, cosine similarity, etc. Different similarity measurement methods are suitable for different data types and application scenarios. The results of clustering can be used for marketing data visualization and data mining.
[0066] In this embodiment, the distributed data set is divided into multiple partitioned data subsets according to clustering rules, and the data of the distributed data set is divided according to the categories to obtain multiple partitioned data subsets. In one embodiment, the clustering rules include marketing rule clustering, and in one embodiment, the clustering rules include province clustering. By dividing the distributed data set according to the above clustering rules, each partitioned data subset can be classified more accurately, which is conducive to the visualization and data mining of subsequent marketing data.
[0067] In one embodiment, the step of distributing each of the partitioned data subsets to available computing nodes for computing and processing via a network includes:
[0068] Each partition data subset is distributed to available computing nodes for computing and processing by using broadcast variables over the network.
[0069] In this embodiment, a broadcast variable is used to send each partition data subset to all computing nodes.
[0070] In one embodiment, the step of distributing each of the partitioned data subsets to available computing nodes for computing and processing via a network includes:
[0071] Detecting whether the data amount of each partition data subset is less than a preset data amount;
[0072] When the data amount of the partitioned data subset is smaller than the preset data amount, the partitioned data subset smaller than the preset data amount is distributed to available computing nodes for computing and processing via the network using broadcast variables.
[0073] In this embodiment, the data volume of the partitioned data subset is less than the preset data volume, indicating that the data volume of the partitioned data subset is small. In this embodiment, for a data set with a small data volume, a broadcast variable is used to send the data to all computing nodes for processing, so that the distribution efficiency and processing efficiency can be improved. After processing, the data is cached in the memory of the node for next use. When the data volume of the partitioned data subset is greater than or equal to the preset data volume, the partitioned data subset is distributed to the computing node closest to its storage location for processing.
[0074] Embodiment 2
[0075] The existing marketing rule calculation method basically uses traditional batch processing programs and traditional real-time processing programs to calculate marketing data and store it in the database. Traditional batch processing programs and traditional real-time processing programs are difficult to implement distributed computing, and the resources of a single computing node are limited, resulting in slow data processing speed, difficulty in processing large amounts of data, and the inability to guarantee the effectiveness of marketing data. For marketing systems, traditional technologies have been unable to meet the growing amount of data and the increasing number of marketing rule calculations.
[0076] In this embodiment, in order to meet the diversified, rapid and accurate marketing needs, the marketing system must always grasp the current marketing data. Different organizational levels, such as headquarters, regional companies, and provincial companies, have different requirements for the timeliness of marketing data. When marketing activities are at their peak, the marketing system needs to quickly and in real time handle a large amount of transaction data to meet the query needs of the business system.
[0077] The traditional solution is to use a single-node batch computing program or a single-node real-time computing program to calculate the marketing data and then store it in a data warehouse. The calculation method of the traditional solution will cause high latency in the storage of marketing data and will be unable to calculate large quantities of marketing data, causing the marketing rules to become invalid and affecting the marketing accuracy of the marketing system.
[0078] How to provide high-concurrency and high-time marketing data computing capabilities in the process of large-scale marketing data access is a difficult point. The purpose of the present invention is to realize a marketing rule processing method that supports high efficiency and large data volume. Based on distributed computing technology, a large amount of data is generated through RDD technology to generate a distributed data set, and the RDD is divided into a series of partitions (Partition) according to marketing strategies (clustering by marketing rules, clustering by province, etc.). Each partition contains a part of the data in the RDD, and these partitions are distributed to available computing nodes through the network for computing and processing. For smaller data sets, broadcast variables can be used to send data to all computing nodes and cache them in the memory of the nodes for next use. This can avoid repeated data transmission and improve efficiency. For larger data sets, the data will be divided into multiple partitions, and each partition will be sent to a different computing node for processing. Usually, the partition data is sent to the computing node closest to its storage location for processing to maintain data locality. Redundant backup of marketing rule detail data is performed to ensure the accuracy of marketing rule detail data. Efficient processing and analysis of marketing rule detail data is achieved to support real-time marketing decisions.
[0079] In order to realize a distributed and real-time marketing data processing method, the embodiment of the present application distributes a large amount of data to computing nodes through RDD technology based on distributed computing technology for parallel computing, improves computing efficiency through cache and memory, and performs redundant backup of marketing detail data, thereby ensuring the accuracy of marketing rule detail data. Real-time distributed processing of marketing data is realized. Efficient processing and analysis of marketing data supports real-time marketing decisions.
[0080] It should be understood that although Figure 1 The steps in the flowchart are shown in sequence as indicated by the arrows, but these steps are not necessarily executed in the order indicated by the arrows. Unless otherwise specified in this document, there is no strict order restriction for the execution of these steps, and these steps can be executed in other orders. Moreover, Figure 1 At least part of the steps may include multiple sub-steps or multiple stages. These sub-steps or stages are not necessarily executed at the same time, but can be executed at different times. The execution order of these sub-steps or stages is not necessarily sequential, but can be executed in turn or alternately with other steps or at least part of the sub-steps or stages of other steps.
[0081] Embodiment 3
[0082] In this embodiment, Figure 2 As shown, a marketing data processing device is provided, comprising:
[0083] A data push detection module 210 is used to detect whether the application pushes marketing data to the message queue;
[0084] The marketing data acquisition module 220 is used to acquire the marketing data when it is detected that the application pushes the marketing data to the message queue;
[0085] A distributed data set generation module 230 is used to generate a distributed data set from the marketing data by using RDD technology based on distributed computing technology;
[0086] A data subset partitioning module 240, configured to partition the distributed data set into a plurality of partition data subsets according to a preset strategy;
[0087] The data subset distribution module 250 is used to distribute each of the partitioned data subsets to available computing nodes through the network for computing and processing.
[0088] In one embodiment, the message queue is a Kafka message queue.
[0089] In one embodiment, the data subset distribution module includes:
[0090] A storage location acquisition unit, used to acquire the storage location of each partition data subset;
[0091] The distribution unit is used to distribute each partition data subset to an available computing node closest to the partition data subset through a network based on the storage location of each partition data subset for computing and processing.
[0092] In one embodiment, the preset strategy includes a clustering rule.
[0093] In one embodiment, the data subset distribution module is further used to distribute each partition data subset to available computing nodes for computing and processing using broadcast variables over a network.
[0094] In one embodiment, the data subset distribution module includes:
[0095] A data volume detection unit, used to detect whether the data volume of each partition data subset is less than a preset data volume;
[0096] The broadcast unit is used to distribute the partitioned data subset smaller than the preset data amount to available computing nodes for computing and processing by using broadcast variables through the network when the data amount of the partitioned data subset is smaller than the preset data amount.
[0097] The specific definition of the marketing data processing device can be found in the definition of the marketing data processing method above, which will not be repeated here. Each unit in the above marketing data processing device can be implemented in whole or in part by software, hardware, or a combination thereof. Each of the above units can be embedded in or independent of the processor in the computer device in the form of hardware, or can be stored in the memory of the computer device in the form of software, so that the processor can call and execute the operations corresponding to each of the above units.
[0098] Embodiment 4
[0099] In this embodiment, a computer device is provided. Its internal structure diagram can be shown as follows: Figure 3 As shown. The computer device includes a processor, a memory, a network interface, a display screen and an input device connected through a system bus. Among them, the processor of the computer device is used to provide computing and control capabilities. The memory of the computer device includes a non-volatile storage medium and an internal memory. The non-volatile storage medium stores an operating system and a computer program, and the non-volatile storage medium is deployed with a database, which is used to store marketing data. The internal memory provides an environment for the operation of the operating system and computer program in the non-volatile storage medium. The network interface of the computer device is used to communicate with other computer devices deployed with application software. When the computer program is executed by the processor, a marketing data processing method is implemented. The display screen of the computer device can be a liquid crystal display screen or an electronic ink display screen, and the input device of the computer device can be a touch layer covered on the display screen, or a key, trackball or touchpad set on the computer device housing, or an external keyboard, touchpad or mouse, etc.
[0100] Those skilled in the art will understand that Figure 3 The structure shown in the figure is only a block diagram of a part of the structure related to the solution of the present application, and does not constitute a limitation on the computer device to which the solution of the present application is applied. The specific computer device may include more or fewer components than those shown in the figure, or combine certain components, or have a different arrangement of components.
[0101] In one embodiment, a computer device is provided, including a memory and a processor, wherein the memory stores a computer program, and when the processor executes the computer program, the following steps are implemented:
[0102] Detect whether the application pushes marketing data to the message queue;
[0103] When it is detected that the application pushes the marketing data to the message queue, acquiring the marketing data;
[0104] Based on distributed computing technology, the marketing data is used to generate a distributed data set through RDD technology;
[0105] Dividing the distributed data set into a plurality of partition data subsets according to a preset strategy;
[0106] Each of the partitioned data subsets is distributed to available computing nodes via a network for computing and processing.
[0107] In one embodiment, the message queue is a Kafka message queue.
[0108] In one embodiment, when the processor executes the computer program, the processor further implements the following steps:
[0109] Obtaining the storage location of each partition data subset;
[0110] Based on the storage location of each partition data subset, each partition data subset is distributed to an available computing node closest to the partition data subset through a network for computing and processing.
[0111] In one embodiment, the preset strategy includes a clustering rule.
[0112] In one embodiment, when the processor executes the computer program, the processor further implements the following steps:
[0113] Each partition data subset is distributed to available computing nodes for computing and processing by using broadcast variables over the network.
[0114] In one embodiment, when the processor executes the computer program, the processor further implements the following steps:
[0115] Detecting whether the data amount of each partition data subset is less than a preset data amount;
[0116] When the data amount of the partitioned data subset is smaller than the preset data amount, the partitioned data subset smaller than the preset data amount is distributed to available computing nodes for computing and processing via the network using broadcast variables.
[0117] Embodiment 5
[0118] In this embodiment, a computer-readable storage medium is provided, on which a computer program is stored. When the computer program is executed by a processor, the following steps are implemented:
[0119] Detect whether the application pushes marketing data to the message queue;
[0120] When it is detected that the application pushes the marketing data to the message queue, acquiring the marketing data;
[0121] Based on distributed computing technology, the marketing data is used to generate a distributed data set through RDD technology;
[0122] Dividing the distributed data set into a plurality of partition data subsets according to a preset strategy;
[0123] Each of the partitioned data subsets is distributed to available computing nodes via a network for computing and processing.
[0124] In one embodiment, the message queue is a Kafka message queue.
[0125] In one embodiment, when the computer program is executed by a processor, the following steps are also implemented:
[0126] Obtaining the storage location of each partition data subset;
[0127] Based on the storage location of each partition data subset, each partition data subset is distributed to an available computing node closest to the partition data subset through a network for computing and processing.
[0128] In one embodiment, the preset strategy includes a clustering rule.
[0129] In one embodiment, when the computer program is executed by a processor, the following steps are also implemented:
[0130] Each partition data subset is distributed to available computing nodes for computing and processing by using broadcast variables over the network.
[0131] In one embodiment, when the computer program is executed by a processor, the following steps are also implemented:
[0132] Detecting whether the data amount of each partition data subset is less than a preset data amount;
[0133] When the data amount of the partitioned data subset is smaller than the preset data amount, the partitioned data subset smaller than the preset data amount is distributed to available computing nodes for computing and processing via the network using broadcast variables.
[0134] Those skilled in the art can understand that all or part of the processes in the above-mentioned embodiment methods can be completed by instructing the relevant hardware through a computer program, and the computer program can be stored in a non-volatile computer-readable storage medium. When the computer program is executed, it can include the processes of the embodiments of the above-mentioned methods. Among them, any reference to memory, storage, database or other media used in the embodiments provided in this application can include non-volatile and / or volatile memory. Non-volatile memory can include read-only memory (ROM), programmable ROM (PROM), electrically programmable ROM (EPROM), electrically erasable programmable ROM (EEPROM) or flash memory. Volatile memory can include random access memory (RAM) or external cache memory. As an illustration and not limitation, RAM is available in many forms, such as static RAM (SRAM), dynamic RAM (DRAM), synchronous DRAM (SDRAM), double data rate SDRAM (DDRSDRAM), enhanced SDRAM (ESDRAM), synchronous link (Synchlink) DRAM (SLDRAM), memory bus (Rambus) direct RAM (RDRAM), direct memory bus dynamic RAM (DRDRAM), and memory bus dynamic RAM (RDRAM).
[0135] The technical features of the above embodiments may be arbitrarily combined. To make the description concise, not all possible combinations of the technical features in the above embodiments are described. However, as long as there is no contradiction in the combination of these technical features, they should be considered to be within the scope of this specification.
[0136] The above-mentioned embodiments only express several implementation methods of the present application, and the descriptions thereof are relatively specific and detailed, but they cannot be understood as limiting the scope of the invention patent. It should be pointed out that, for a person of ordinary skill in the art, several variations and improvements can be made without departing from the concept of the present application, and these all belong to the protection scope of the present application. Therefore, the protection scope of the patent of the present application shall be subject to the attached claims.
Claims
1. A marketing data processing method, It is characterized in that include: Detect whether the application pushes marketing data to the message queue; When it is detected that the application pushes the marketing data to the message queue, acquiring the marketing data; Based on distributed computing technology, the marketing data is used to generate a distributed data set through RDD technology; Dividing the distributed data set into a plurality of partition data subsets according to a preset strategy; Each of the partitioned data subsets is distributed to available computing nodes via a network for computing and processing.
2. The method according to claim 1, It is characterized in that The message queue is a Kafka message queue.
3. The method according to claim 1, It is characterized in that The step of distributing each of the partitioned data subsets to available computing nodes for computing and processing via a network comprises: Obtaining the storage location of each partition data subset; Based on the storage location of each partition data subset, each partition data subset is distributed to an available computing node closest to the partition data subset through a network for computing and processing.
4. The method according to claim 1, It is characterized in that The preset strategy includes clustering rules.
5. The method according to claim 1, It is characterized in that The step of distributing each of the partitioned data subsets to available computing nodes for computing and processing via a network comprises: Each partition data subset is distributed to available computing nodes for computing and processing by using broadcast variables over the network.
6. The method according to claim 5, It is characterized in that The step of distributing each of the partitioned data subsets to available computing nodes for computing and processing via a network comprises: Detecting whether the data amount of each partition data subset is less than a preset data amount; When the data amount of the partitioned data subset is smaller than the preset data amount, the partitioned data subset smaller than the preset data amount is distributed to available computing nodes for computing and processing via the network using broadcast variables.
7. A marketing data processing device, It is characterized in that include: Data push detection module, used to detect whether the application pushes marketing data to the message queue; A marketing data acquisition module, configured to acquire the marketing data when detecting that the application pushes the marketing data to the message queue; A distributed data set generation module, used to generate a distributed data set from the marketing data using RDD technology based on distributed computing technology; A data subset partitioning module, used to partition the distributed data set into a plurality of partition data subsets according to a preset strategy; The data subset distribution module is used to distribute each of the partitioned data subsets to available computing nodes through the network for computing and processing.
8. The device according to claim 7, It is characterized in that The data subset distribution module comprises: A storage location acquisition unit, used to acquire the storage location of each partition data subset; The distribution unit is used to distribute each partition data subset to an available computing node closest to the partition data subset through a network based on the storage location of each partition data subset for computing and processing.
9. A computer device comprising a memory and a processor, wherein the memory stores a computer program. It is characterized in that When the processor executes the computer program, the steps of the method according to any one of claims 1 to 6 are implemented.
10. A computer-readable storage medium having a computer program stored thereon, It is characterized in that When the computer program is executed by a processor, the steps of the method according to any one of claims 1 to 6 are implemented.
Citation Information
Patent Citations
Distributed stream-oriented computation based spatial data processing method and system
CN105512297A
Distributed streaming data processing method and system in cloud environment
CN107391719A
Short text batch matching method and device
CN111259117A
Distributed computing method for realizing digital energy management system
CN116560850A
