Computing cross products using map reduce
By generating cross products based on join fields and controlling key-value pairs in MapReduce operations, the system addresses inefficiencies in large dataset processing, optimizing resource use and speeding up computations.
Patent Information
- Application Number
- KR1020267021724
- Authority / Receiving Office
- KR · KR
- Patent Type
- Patents
- Current Assignee / Owner
- Priority Date
- 2019-01-28
- Filing Date
- 2020-01-24
- Publication Date
- 2026-11-11
- Estimated Expiration
- 2040-01-24
AI Technical Summary
Conventional MapReduce systems face inefficiencies in generating cross products from large datasets, leading to excessive computational, memory, and storage resource consumption, particularly due to data skew and impractical partitioning, which results in slow and inefficient use of resources.
The system generates multiple cross products based on join fields, controlling the number of key-value pairs sent to reducers through MapReduce operations, distributing computational load efficiently across available resources.
This approach reduces resource consumption by generating cross products based on specific values, optimizing computational, memory, and storage usage, enabling faster processing and more efficient resource utilization.
Smart Images

Figure 112026081734421-PAT00002_ABST
Abstract
Description
Technology Field
[0001] The present disclosure relates to the field of data processing systems, and in particular, to calculating cross products using a map reduce framework. Background Technology
[0002] Large-scale data processing involves extracting data of interest from raw data within one or more datasets and processing the raw data into useful data products. Large-scale data processing in parallel and distributed processing environments typically involves the distribution of data and computations among multiple disks and processing devices to efficiently utilize total storage space and computational power. Brief explanation of the drawing
[0003] Various embodiments of the present disclosure will be more fully understood from the detailed description given below and from the accompanying drawings of various embodiments of the present disclosure. FIG. 1 illustrates an exemplary system architecture according to embodiments of the present disclosure. FIG. 2 is a flowchart illustrating a method for generating external products based on a join field according to embodiments of the present disclosure. FIG. 3 is a flowchart illustrating a method for performing a set of MapReduce operations on a dataset to generate external products according to embodiments of the present disclosure. FIG. 4a illustrates a first operation of a set of MapReduce operations for generating external products from a dataset based on join fields according to embodiments of the present disclosure. FIG. 4b illustrates a second operation of a set of MapReduce operations for generating external products from a dataset based on a join field, according to embodiments of the present disclosure. FIG. 4c illustrates a third operation of a set of MapReduce operations for generating external products from a dataset based on join fields, according to embodiments of the present disclosure. FIG. 4d illustrates a diagram of a fourth operation of a set of MapReduce operations for generating external products from a dataset based on a join field, according to embodiments of the present disclosure. FIG. 4e illustrates a fifth operation of a set of MapReduce operations for generating external products from a dataset based on a join field, according to embodiments of the present disclosure. FIG. 4f illustrates a map phase of a first operation of a set of map-reduce operations for generating external products from a dataset including time ranges, according to embodiments of the present disclosure. FIG. 5 is a block diagram illustrating an exemplary computer system according to embodiments of the present disclosure. Specific details for implementing the invention
[0004] The following description provides numerous specific details, such as examples of specific systems, components, methods, etc., to provide a good understanding of the various embodiments of the present disclosure. However, it will be obvious to those skilled in the art that at least some of the embodiments of the present disclosure may be practiced without these specific details. In other cases, to avoid making the present disclosure unnecessarily obscured, widely known components or methods are not described in detail or are provided in the form of simple block diagrams. Accordingly, the specific details presented are merely illustrative. Specific embodiments may be modified from these illustrative details and may nevertheless be considered to be within the scope of the present disclosure.
[0005] Aspects of the present disclosure relate to external generation using a MapReduce framework. Modern data centers typically contain thousands of hosts and operate collectively to service requests from an even larger number of remote clients. During operation, components of these data centers can generate a significant volume of machine-generated data. Typically, the data can be transformed into useful data products, and the transformed data can be used in downstream processes, such as as input to a trained machine learning model, or to perform specific operations, such as similarity analysis or scoring analysis.
[0006] One such data transformation is the cross product (also referred to as the "Cartesian product"). The cross product may refer to a set of values derived from an operation (e.g., a cross operation) that pairs every individual value of a dataset with every individual other value of the same dataset or with every individual value of one or more other datasets. For example, dataset A may contain four entries: {value 1, value 2, value 3, value 4}. The cross product of dataset A pairs each value of dataset A with every individual remaining value of dataset A. The cross product of dataset A includes a set of values: {[value 1, value 2], [value 1, value 3], [value 1, value 4], [value 2, value 3], [value 2, value 4], [value 3, value 4]}. Generating external products using large datasets can consume a large amount of computer resources, such as computation, memory, and storage resources.
[0007] MapReduce is a programming framework for processing and generating large datasets in parallel using computer cluster(s). A MapReduce job includes map tasks and reduce tasks. A map task may include one or more map operations. A reduce task may include one or more reduce operations. Map tasks perform filtering and saving of the dataset, and reduce tasks perform summary operations.
[0008] In some cases, a single cross product derived from all the data in a dataset may not be a useful data product. Rather, a useful data product may include multiple cross products generated from the dataset, each of which is based on a specific value (e.g., joined to a specific key). For example, a dataset may have five entries with the following values: [User 1:IP1], [User 2:IP1], [User 3:IP1], [User 1:IP2], [User 2:IP2]. The cross product for the entire dataset pairs each of the five values with every other value. Multiple cross products for datasets based on specific values (e.g., joined to specific keys) generate cross products for values having "IP1" {[User 1:IP1, User 2:IP1], [User 1:IP1, User 3:IP1], [User 2:IP1, User 3:IP1]}, and other cross products for "IP2" {[User 1:IP2], [User 2:IP2]}.
[0009] In some conventional systems, MapReduce can be used to generate a single outer product for all values in a dataset(s). Datasets can be large, and generating an outer product on a large dataset results in an even larger dataset. For an outer product of a dataset with one million entries, the resulting outer product can have one trillion entries. Using MapReduce to generate a single outer product for all values in datasets, particularly large datasets, can be inefficient and consume a large amount of computational, memory, and storage resources. In other conventional systems, a dataset can be partitioned into multiple datasets by value, and accordingly, each dataset contains an entry with a specific value. A MapReduce job (or set of jobs) can be executed on each dataset to generate an outer product for each dataset. However, for large datasets, partitioning the datasets in the above manner can generate thousands or even millions of smaller datasets. MapReduce jobs (or sets of jobs) can be generated for each of the smaller datasets, which may be impractical or unacceptable in itself. Separate MapReduce jobs are typically executed serially, which can be slow and result in inefficient use of computer resources. Additionally, when using MapReduce to generate the outer product, performing efficient parallel processing can be difficult. For example, data can be skewed, and consequently, data associated with a specific value (e.g., a key) may be much larger than other data associated with a different value (e.g., a key).Data skew can lead to inefficient use of computational resources in the MapReduce framework because some processing nodes may spend a significant amount of time processing large data blocks, while others remain idle after processing small data blocks.
[0010] Aspects of the present disclosure solve the above and other difficulties by generating multiple cross products for a dataset, each based on a join field. The join field may indicate that the keys of key-value pairs should be generated based on a specific data field of entries in the dataset (e.g., joined to a specific key). If the values within the data field identified by the join field are identical, the keys of the resulting key-value pairs are identical. Cross products can be generated for groups of key-value pairs having identical keys, and accordingly, multiple cross products for the dataset are each based on a specific value.
[0011] In some embodiments, a set of MapReduce operations transforms a dataset into multiple groups of key-value pairs, wherein each group of key-value pairs shares the same key. The set of MapReduce operations performed on the dataset may further generate multiple cross products, wherein each cross product is for a group of key-value pairs having the same keys, and cross products are not generated for key-value pairs that do not share the same keys.
[0012] In the embodiments, a set of MapReduce tasks modifies the keys of a group of key-value pairs to control the number of key-value pairs sent to any one reducer. By controlling the number of key-value pairs sent to any one reducer, the computational load for generating cross products is distributed among the available reducers, which enables cross products to be computed faster and allows for more efficient use of computation, memory, and storage resources.
[0013] Accordingly, the technique described herein enables the generation of multiple cross products from a dataset using a set of MapReduce operations. The foregoing reduces computational resources (e.g., processing resources), memory resources, and storage resources by generating cross products based on join fields, thereby generating cross products based on specific values (e.g., joined to specific keys) rather than a single cross product for the entire dataset, and by modifying key names to control the number of key-value pairs processed by reducers in downstream MapReduce operations.
[0014] FIG. 1 illustrates an exemplary system architecture (100) according to embodiments of the present disclosure. The system architecture (100) (also referred to herein as “system”) includes client devices (110A and 110B) (generally referred to herein as “client device(s) (110”)), a network (105), a data storage (106), a collaboration platform (120), a server (130), and a computer cluster (150). It should be noted that the system architecture (100) is provided for example rather than for limitation. In embodiments, the system architecture (100) may include the same, fewer, more, or different components configured in the same or different ways.
[0015] In one embodiment, the network (105) may include a public network (e.g., the Internet), a private network (e.g., a local area network (LAN) or a wide area network (WAN)), a wired network (e.g., an Ethernet network), a wireless network (e.g., an 802.11 network, a Wi-Fi® network, or a wireless LAN (WLAN)), a cellular network (e.g., a Long Term Evolution (LTE) network), routers, hubs, switches, server computers, or a combination thereof.
[0016] In one embodiment, the data storage (106) may be memory (e.g., random access memory), a cache, a drive (e.g., a hard drive), a flash drive, a database system, or other types of components or devices capable of storing data. The data storage (106) may also include a number of storage components (e.g., a number of drives or a number of databases) that may also span across a number of computing devices (e.g., a number of server computers).
[0017] In embodiments, the server (130) may be one or more computing devices (e.g., a rack-mount server, a server computer, a cluster of physical servers, etc.). In embodiments, the server (130) may be included in the collaboration platform (120), may be an independent system, or may be part of another system or platform. The server (130) may include an external module (140).
[0018] In some embodiments, the collaboration platform (120) may be one or more computing devices (e.g., rackmount server, router computer, server computer, personal computer, mainframe computer, laptop computer, tablet computer, desktop computer, etc.), data storage devices (e.g., hard disks, memories, databases), networks, software components, or hardware components that can be used to perform operations of the collaboration platform (120) and provide users with access to the collaboration platform (120).
[0019] In embodiments, the collaboration platform (120) may also include a website (e.g., a webpage) or application back-end software that can be used to provide users with access to content provided by the collaboration platform (120). For example, users may access the collaboration platform (120) using a collaboration application (114) on client devices (110). It should be noted that the collaboration applications (114A and 114B) may generally be referred to herein as the collaboration application(s) (114). In some embodiments, the collaboration application (114) may be two instances of the same application.
[0020] In embodiments, the collaboration platform (120) may be a type of social network providing connections between users, or a type of user-generated content system that enables users (e.g., end users or consumers) to create content for the platform, wherein the created content may also be consumed by other users of the system. In embodiments of the present disclosure, "user" may be represented as a single individual. However, other embodiments of the present disclosure include "user" (e.g., creator user) which is an entity controlled by an automated source or a set of users. For example, in a user-generated content system, a set of individual users united as a community or group may be considered a "user."
[0021] In one embodiment, the collaboration platform (120) may be a gaming platform such as an online gaming platform or a virtual gaming platform. For example, the gaming platform may provide single-player or multi-player games to a community of users who can access games (122A to 122Z) or interact with games (122A to 122Z) using client devices (110) via a network (105). In embodiments, the games (122) (also referred to herein as “video games”, “online games”, or “virtual games”) may be, for example, two-dimensional (2D) games, three-dimensional (3D) games (e.g., 3D user-generated games using a constructor module (126)), virtual reality (VR) games, or augmented reality (AR) games. In embodiments, users, such as players, may participate in gameplay with other players. In the embodiments, the game (122) can be played in real time with other users of the game (122).
[0022] In some embodiments, the game (122) may include an electronic file that can be executed or loaded using software, firmware, or hardware configured to present game content (e.g., digital media items) to an entity. In some embodiments, the game (122) may be executed and rendered using a game engine (124). In some embodiments, the game (122) may have a common set of rules or a common goal, and the environments of the game (122) share a common set of rules or a common goal. In some embodiments, different games may have different rules or goals.
[0023] It should be noted that the collaboration platform (120) hosting the games (122) is provided for the purposes of example rather than limitation. In some embodiments, the collaboration platform (120) may host one or more media items. Media items may include, but are not limited to, digital videos, digital movies, digital photos, digital music, audio content, website content, social media updates, e-books, e-magazines, digital newspapers, digital audiobooks, e-journals, web blogs, RSS (real simple syndication) feeds, e-comic books, software applications, etc. In embodiments, a media item may be an electronic file that can be executed or loaded using software, firmware, or hardware configured to present the digital media item to a user.
[0024] In some embodiments, the collaboration platform (120) or client devices (110) may include a game engine (124). In embodiments, the game engine (124) may be used for the development or execution of games (122). For example, the game engine (124) may include, among other features, a rendering engine ("renderer") for 2D, 3D, VR or AR graphics, a physics engine, a collision detection engine (and collision response), a sound engine, a scripting function, an animation engine, an artificial intelligence engine, a networking function, a streaming function, a memory management function, a threading function, a scene graph function, or video support for cinematics. Components of the game engine (124) may generate commands (e.g., rendering commands, collision commands, physics commands, etc.) that help compute and render the game (122). In some embodiments, the game engine (124) of the client devices (110) may operate independently, in collaboration with the game engine (124) of the collaboration platform (120), or in a combination of both.
[0025] In embodiments, the collaboration platform (120) may include a creator module (126). In embodiments, the creator module (126) may enable users of the collaboration platform (120) to become creator users who design or create environments within an existing game (122), create new games, or create new game objects within games or environments.
[0026] In embodiments, the creator module (126) may enable the user to create, modify, or customize characters. In embodiments, characters (or generally game objects) are composed of components that are automatically combined to assist the user in editing, and one or more of the components may be selected by the user. One or more characters (also referred to herein as “avatar” or “model”) may be associated with the user (also referred to herein as “playing user”), wherein the user may control the characters to facilitate interaction between the game (122) and the user. In embodiments, the characters may include components such as body parts (e.g., hair, arms, legs, etc.) and accessories (e.g., T-shirts, glasses, decorative images, tools, etc.). In embodiments, the body parts of the customizable characters include, among other things, head types, body part types (arms, legs, torsos, and hands), face types, hair types, and skin types. In embodiments, customizable accessories include clothing (e.g., shirts, pants, hats, shoes, glasses, etc.), weapons, or other tools. In embodiments, the user may also control the scale of the character (e.g., height, width, or depth) or the scale of the character's components. In embodiments, the user may control the proportions of the character (e.g., blocky, anatomical, etc.). In some embodiments, the character may not include character game objects (e.g., body parts, etc.), but it should be noted that the user may control the character (without character game objects) to facilitate interaction with the game (e.g., a puzzle game where there are no rendered character game objects but the user still controls the character to control in-game actions).
[0027] In embodiments, a collaboration platform (120) running a creator module (126) includes a user-interfacing website or application (e.g., a collaboration application (114)), wherein users (also referred to herein as “creating users,” “creators,” “owners,” or “owner users”) can access online computing resources (e.g., cloud resources) hosted by the collaboration platform (120) for the purposes of building, managing, editing, and interacting with privately owned games (122) or gaming environments. In embodiments, the creator module (126) includes tools available to users to create and instantiate three-dimensional virtual games or environments. In embodiments, the creator module (126) is available to users who wish to create and manage their own private virtual games (122). In embodiments, users can access the creator module (126) using the collaboration application (114). In embodiments, the constructor module (126) may use a user interface (also referred to herein as a “developer interface”) through the collaboration application (114) to enable users to access the functionality of the constructor module (126). In embodiments, the developer interface may be part of the collaboration application (114). For example, the developer interface of the collaboration application (114) may enable user access to a library of game objects that may be selected by the user to build a game environment or to build a game (122). The user may make these game objects public through the developer interface, and accordingly, users of the collaboration platform (120) may be able to use the game.
[0028] In embodiments, the collaboration platform (120) may include a messaging module (128). In embodiments, the messaging module (128) may be a system, application, or module that enables users to exchange electronic messages through a communication system such as a network (105). The messaging module (128) may be associated with a collaboration application (114) (e.g., may be a module of the collaboration application (114) or may be a separate application). In embodiments, users may interface with the messaging module (128) and exchange electronic messages among users of the collaboration platform (120). The messaging module (128) may be, for example, an instant messaging application, a text messaging application, an email application, a voice messaging application, a video messaging application, or a combination thereof, among other things.
[0029] In embodiments, the messaging module (128) can facilitate the exchange of electronic messages between users. For example, a user may be logged into a messaging application on a client device (110A), and another user may be logged into a messaging application on a client device (110B). The two users may initiate a conversation, such as an instant messaging conversation. The messaging module (128) may help facilitate the messaging conversation by transmitting and receiving electronic messages between users of the collaboration platform (120). In another embodiment, the two users may engage in an in-game dialogue with each other using their respective messaging applications, wherein the dialogue may be part of a view that includes gameplay.
[0030] In the embodiments, the client devices (110A to 110B) may each include computing devices, such as personal computers (PCs), mobile devices (e.g., laptops, mobile phones, smartphones, tablet computers, or netbook computers), network-connected televisions, gaming consoles, etc. In some embodiments, the client devices (110A to 110B) may also be referred to as "user devices." In the embodiments, one or more client devices (110) may connect to the collaboration platform (120) at any given moment via the collaboration application (114). It should be noted that the number of client devices (110) is provided as an example rather than a limitation. In the embodiments, any number of client devices (110) may be used.
[0031] In the embodiments, each client device (110) may include an instance of a collaboration application (114). In one embodiment, the collaboration application (114) may be an application that enables users to use and interact with the collaboration platform (120), such as controlling a virtual character in a virtual game hosted by the collaboration platform (120), or viewing or uploading content such as games (122), images, video items, web pages, documents, etc. In one example, the collaboration application (114) may be a web application (e.g., an application that works with a web browser) that can access, search for, present, or navigate content served by a web server (e.g., a virtual character in a virtual environment). In another example, the collaboration application (114) may be a native application (e.g., a mobile application or a gaming program) that is installed and executed locally on the client device (110) and enables users to interact with the collaboration platform (120). The collaboration application (114) can render, display, or present content (e.g., a web page, a media viewer) to the user. In an embodiment, the collaboration application (114) may also include an embedded media player (e.g., a Flash® player) embedded in a web page.
[0032] Generally, the functions described as being performed by the collaboration platform (120) in one embodiment may also be performed by client devices (110A to 110B) or a server (130) in other embodiments, where appropriate. Additionally, functionality attributed to a specific component may be performed by different or multiple components operating together. The collaboration platform (120) may also be accessed as a service provided to other systems or devices through appropriate application programming interfaces (APIs).
[0033] In embodiments, the collaboration platform (120) may generate a large amount of data from the operation of the collaboration platform (120). For example, the collaboration platform (120) may have millions of users participating in user sessions daily to play or create games (122). The large amount of raw data associated with user sessions may be stored in one or more databases associated with the data store (106). A session (also referred to herein as a “user session”) may refer to a period of time that begins when an application (e.g., a collaboration application (114)) is opened to access the collaboration platform (120) and ends when the application is closed. In some embodiments, a session may span a period of time (e.g., a time range) that begins when the application is opened and the user is interacting with the collaboration platform (120). A session may end when the user is inactive for a critical period of time (e.g., even if the application is still open). Session information may include context information describing a specific session (e.g., start and end timestamps, client device type, internet protocol address used to access the collaboration platform (120), etc.) and user activity information describing user interaction with the collaboration platform (120) (e.g., user inputs for controlling character actions, text messages, etc.).
[0034] In the embodiments, the cross product of a dataset may be determined using a cross product module (140). The cross product module (140) may use a set of MapReduce tasks to determine the cross product of one or more datasets. A MapReduce task may refer to two phases of MapReduce (e.g., a map phase and a reduce phase). In the mapping phase, one or more map operations (e.g., map task(s)) retrieve data (e.g., key-value pairs) from input data files according to the mapping operations and generate intermediate data values. In the reduce phase, one or more reduce operations (e.g., reduce task(s)) generate output data by merging or otherwise combining the intermediate data values (e.g., combining intermediate values that share the same key) according to the reduce operations. A set of MapReduce tasks may refer to two or more MapReduce tasks that are typically performed in series. For example, two map-reduce operations performed in series may include a first map-reduce operation (e.g., a mapping phase and a reduce phase) that generates an output used as an input to a second map-reduce operation (e.g., another mapping phase and another reduce phase).
[0035] In some embodiments, a system for large-scale processing of data in a parallel processing environment comprises one or more computer clusters (150). It should be noted that the computer cluster (150) is exemplified as a single cluster for the purposes of the examples, rather than by limitation. The computer cluster (150) may comprise one or more computer clusters. In embodiments, the computer cluster (150) comprises one or more interconnected nodes (132 and 134A through 134N) for performing common tasks so that the computer cluster (150) can be viewed as a single computer system. For example, the computer cluster (150) comprises a master node (132) (generally referred to as "node (132)") and worker nodes (134A through 134N) (generally referred to as "node(s) (134)" or "worker node(s) (134)"). Each node (132 and 134) of the computer cluster (150) may include, but is not limited to, any data processing device, such as a processor, a desktop computer, a laptop computer, a mainframe computer, a personal digital assistant, a server computer, a hand-held device, or one or more dies of a multi-die processor, or any other device configured to process data. The nodes (132 and 134) of the computer cluster (150) may be connected to each other through a network such as the network (105). Each node (132 and 134) may run an instance of an operating system on its own.
[0036] In the embodiments, each node (132 and 134) of the computer cluster (150) may have its own physical or virtual memory. The memory may include, but is not limited to, main memory, such as read-only memory (ROM), flash memory, dynamic random access memory (DRAM), or static random access memory (SRAM). Each node of the computer cluster (150) may have data stored on local storage (not shown), such as local storage disks. The computer cluster (150) and each node (132 and 134) of the computer cluster (150) may additionally implement various network-accessible server-based functionalities (not shown) or include other data processing equipment.
[0037] In some embodiments, the master node (132) may control the aspects of the MapReduce job. For example, the master node (132) may determine how many map operations to use, how many reduce operations to use, which processes and processing devices (e.g., nodes) to use to perform the operations, where to store intermediate and output data, and how to respond to processing failures. The master node (132) may instruct one or more worker nodes (134) to perform various operations of the MapReduce job. It should be noted that a single MapReduce job may be executed in parallel on one or more nodes (134) of the computer cluster (150).
[0038] Nodes (134) of a computer cluster may perform map operations, reduce operations, or both. An individual node (134) may perform one or more mapping operations in parallel or in series. An individual node (134) may perform one or more reduce operations in parallel or in series. "Mapper" may refer to a node (134) that performs one or more map operations. "Reducer" may refer to the same or different nodes (134) that perform one or more reduce operations. In some embodiments, a single node (134) may include one or more mappers, one or more reducers, or both.
[0039] In the embodiments, the computer cluster (150) can run a MapReduce framework. The computer cluster (150) can run a specific MapReduce framework, such as Apache TM Hadoop®, Hazelcast®, MongoDB®, Infinispan, or Apache TM Spark TM It can be configured to execute.
[0040] A computer cluster (150) may be associated with one or more queues (136). A queue (136) may include a data structure that stores elements. A queue (136) may assist the computer cluster (150) with scheduling information associated with one or more MapReduce jobs.
[0041] In some embodiments, elements stored in the queue (136) may include markers (138). In some examples, the markers (138) include actual data units on which the computer cluster (150) performs one or more MapReduce operations. In other examples, the markers (138) may identify the locations of data units stored in the data store (106). For example, each marker (138) may be associated with one or more rows of data in a database. Each marker (138) may identify a database, a start address of the data (e.g., a start row in the database), and an end address of the data (e.g., an end row in the database). For example, the marker (138) may be associated with rows 1 through 10,000 in the database. Each marker (138) may identify a fixed-size address range. For example, the first marker can identify the 1st to 10,000th row of the database, and the second marker can identify the 10,001st to 20,000th row of the database.
[0042] In the embodiments, elements in the queue (136) may be maintained in order, and operations on the data structure may include adding elements to the data structure and removing elements from the data structure. For example, the queue (136) may be a First-In-First-Out (FIFO) queue, wherein the first element added to the queue will be the first element to be removed from the queue.
[0043] In some embodiments, the queue (136) is hosted by a computer cluster (150), such as a master node (132). In other embodiments, the queue (136) may be hosted by other components. For example, the queue (136) may be hosted by a component outside the computer cluster (150). Data of the queue (136) may be stored in memory (e.g., random access memory), a cache, a drive (e.g., a hard drive), a flash drive, a database system, or other types of components or devices capable of storing data.
[0044] In some embodiments, a user using a client device (110) may submit a request for one or more MapReduce jobs to be performed by a computer cluster (150). A master node (132) of the computer cluster (150) may receive MapReduce jobs, determine MapReduce operations to be performed, and request one or more worker nodes (134) to perform various MapReduce operations. In embodiments, aspects of the present disclosure may be implemented by an external module (140) executed by the master node (132). In other embodiments, an external module (140) executed by the master node (132), the worker nodes (134), or both may implement aspects of the present disclosure.
[0045] For the sake of example rather than limitation, the external module (140) is described as being implemented in the master node (132). In other embodiments, the external module (140) may be partially or wholly implemented in the collaboration platform (120). In other embodiments, the external module (140) may be partially or wholly implemented in one or more client devices (110). In other embodiments, the external module (140) operating in one or more of the client device (110), computer cluster (150), or collaboration platform (120) may work together to perform the operations described herein. Although embodiments of this disclosure are discussed with respect to collaboration platforms, the embodiments may also be generally applicable to any type of platform that generates or stores data. The external module (140) may help facilitate the operations described herein, such as the operations described with respect to FIGS. 2 through 4. In some embodiments, the external module (140) may be part of another application, such as a plug-in. In some embodiments, the external module (140) may be a separate application running on the computing device.
[0046] In situations where the systems discussed herein collect or use personal information regarding users, users may be provided with the opportunity to control whether the collaboration platform (120) collects user information (e.g., information regarding the user's social networks, social actions or activities, occupation, user preferences, or user's current location), or to control whether or how to receive content that may be more relevant to the user from the collaboration platform (120). Additionally, certain data may be processed in one or more ways so that personally identifiable information is removed before being stored or used. For example, the user's identity may be processed so that personally identifiable information regarding the user cannot be determined, or the user's geographic location from which location information is obtained may be generalized (e.g., to the city, ZIP code, or state level) so that the user's specific location cannot be determined. Thus, users can control how information regarding users is collected and how it is used by the collaboration platform (120).
[0047] FIG. 2 is a flowchart illustrating a method (200) for generating external effects based on a join field according to embodiments of the present disclosure. The method (200) may be performed by processing logic comprising hardware (e.g., circuits, dedicated logic, programmable logic, microcode), software (e.g., instructions executed on a processing device to perform hardware simulation), or a combination thereof. In some embodiments, an external module (140) executed in a computer cluster (150) (e.g., at one or more of a node (132) or nodes (134)) may perform some or all operations. In other embodiments, an external module (140) executed in a collaboration platform (120), a client device (110A), a client device (110B), a server (130), or a combination thereof may perform some or all operations. It should be noted that in some embodiments, the method (200) may include the same, different, fewer, or more number of operations performed in any order.
[0048] In block (205) of method (200), the processing logic performing method (200) receives a request to perform a set of MapReduce operations to generate cross products from a dataset based on a join field. The join field indicates that each cross product should be generated from a corresponding subset of the dataset. Each subset is associated with the same keys, and different subsets are associated with different keys.
[0049] In some embodiments, the join field indicates that each external product must be generated on a key basis such that each external product is generated from a corresponding subset associated with the same key, rather than from data in a dataset associated with different keys.
[0050] In some embodiments, a request (e.g., a single request) is received from a client device (110) for a set of MapReduce tasks for generating crosses of a dataset. The dataset contains one or more entries, and each entry of the dataset contains data specific to a specific entry. The request may include a join field parameter (e.g., "IP") indicating that crosses should be generated for a join field (e.g., "IP") such that crosses are generated for each subset of entries of the dataset associated with the same key, rather than across entries of the dataset associated with different keys. For example, each entry in a subset of the dataset contains data "IP1", and each entry in another subset of the dataset contains data "IP2". An external product is generated for a subset of data associated with "IP1", and another external product is generated for a subset of data associated with "IP2", but external products that will pair entries containing "IP1" with entries containing "IP2" are not generated for the dataset.
[0051] In block (210), in response to receiving a request to perform a set of MapReduce operations to generate cross products of a dataset based on join fields, the processing logic performs a set of MapReduce operations on the dataset to generate cross products. To perform the set of MapReduce operations, the processing logic generates groups of key-value pairs from corresponding subsets of the dataset. Each key-value pair in each group of key-value pairs contains the same key.
[0052] It should be noted that entries in a dataset can be transformed into key-value pairs using one or more mapping operations of a MapReduce job. In some embodiments, some data of an entry is used as a key, while the entire data of the entry is retained and becomes the value in the key-value pair. A join field indicates which data of the entry should be the key. If the key data is the same in multiple entries, these entries will generate key-value pairs that are the same or have identical keys. Performing a set of MapReduce jobs is further described with respect to FIGS. 3 and FIGS. 4a through 4f.
[0053] In block (215), the processing logic stores the final output data of the set of map-reduce tasks, and the final output data includes cross products for each group of key-value pairs. The cross product pairs each value of each group of key-value pairs with each remaining value of each group of key-value pairs to form pairs of values (e.g., value pairs).
[0054] FIG. 3 is a flowchart illustrating a method (300) for performing a set of MapReduce operations on a dataset to generate external effects according to embodiments of the present disclosure. The method (300) may be performed by processing logic comprising hardware (e.g., circuits, dedicated logic, programmable logic, microcode), software (e.g., instructions executed on a processing device to perform hardware simulation), or a combination thereof. In some embodiments, an external effect module (140) executed in a computer cluster (150) (e.g., at one or more of a node (132) or nodes (134)) may perform some or all of the operations. In other embodiments, an external effect module (140) executed in a collaboration platform (120), a client device (110A), a client device (110B), or a server (130), or a combination thereof, may perform some or all of the operations. It should be noted that in some embodiments, the method (300) may include the same, different, fewer, or more number of operations performed in any order.
[0055] In operation (305), the processing logic performing method (300) performs the first map phase of the first operation of the set of map-reduce operations to generate first intermediate data using the dataset. The first intermediate data includes groups of key-value pairs. Key-value pairs of a specific group of key-value pairs have the same keys. Key-value pairs between different groups of key-value pairs have non-identical keys.
[0056] In operation (310), the processing logic performs a first reduce phase of a first operation of a set of map-reduce operations to generate first output data using first intermediate data. The first output data comprises a first group of groups of key-value pairs having first modified keys, and the first modified keys represent the number and sort order of key-value pairs within the first group of key-value pairs. The first operation of the set of operations is further described with respect to FIG. 4a.
[0057] In some embodiments, the dataset may have entries containing time ranges. In embodiments where the dataset has entries containing time ranges, to perform the first map phase of the first operation of the set of MapReduce operations to generate first intermediate data using the dataset, the processing logic identifies a parameter associated with a request to perform the set of MapReduce operations that generate a plurality of cross products. In some embodiments, the parameter represents a time unit. For each entry of the dataset, the processing logic identifies the time range indicated in the entry. The processing logic increments the time range by time units. The increments start from the earliest time identified by the time range and continue to the last time identified by the time range. In each increment, the processing logic generates a timestamp that reflects the time within the time range in the increment. The processing logic generates one or more key-value pairs for the entry based on the increment. The key of one or more key-value pairs identifies the timestamp of the increment, and the value of one or more key-value pairs identifies the data of the entry. The processing logic generates groups of key-value pairs, wherein each key-value pair in each group of key-value pairs contains the same timestamp (e.g., the same key). The above mapping operation is further explained with respect to FIG. 4f.
[0058] In operation (315), the processing logic performs the second map phase of the second operation of the set of MapReduce operations to generate second intermediate data using the first output data of the first operation. The second intermediate data includes a first group of key-value pairs having first modified keys. For clarity, it may be noted that operation (315) and subsequent operations (320 to 350) are described for the first group of key-value pairs to generate a cross product for the first group of key-value pairs. Similar operations may be performed for other groups of key-value pairs, where it may be understood that a cross product is generated for each of the groups of key-value pairs. In total, a plurality of cross products are generated for the dataset, each based on a specific value (e.g., joined to a specific key).
[0059] In operation (320), the processing logic performs a second reduce phase of a second operation of a set of map-reduce operations to generate second output data using second intermediate data. The second output data includes first subgroups of a first group of key-value pairs having second modified keys. The first modified keys of the first group of key-value pairs are modified to generate second modified keys that identify the first subgroups of the first group of key-value pairs.
[0060] In some embodiments, to perform a second reduce phase of a second operation of a set of MapReduce operations to generate second output data using second intermediate data, the processing logic sorts key-value pairs of a first group of key-value pairs in a sorted order identified by first modified keys. The processing logic identifies the number of key-value pairs within the first group of key-value pairs indicated by the first key of the sorted key-value pairs. The processing logic determines the number of key-value pairs for each of the subgroups of key-value pairs (associated with the first group of key-value pairs). The number of key-value pairs within the first subgroups does not exceed the maximum number of key-value pairs identified in the request. The processing logic generates second modified keys for the first subgroups of key-values so that the second modified keys identify a specific subgroup from the first subgroups. The second MapReduce operation of the set of MapReduce operations is further described with respect to FIG. 4b.
[0061] In operation (325), the processing logic performs the third map phase of the third operation of the set of map-reduce operations to generate third intermediate data using the second output data of the second operation. The third intermediate data includes first subgroups of key-value pairs.
[0062] In operation (330), the processing logic performs a third reduce phase of a third operation of a set of map-reduce operations to generate third output data using third intermediate data. The third output data includes second subgroups of key-value pairs generated from first subgroups of key-value pairs. The second subgroups of key-value pairs include the first subgroups of key-value pairs and duplicate key-value pairs of the first subgroups of key-value pairs, wherein at least the keys of the duplicate key-value pairs are modified. A third map-reduce operation of a set of map-reduce operations is further described with respect to FIG. 4c.
[0063] In operation (335), the processing logic performs the fourth map phase of the fourth operation of the set of map-reduce operations to generate fourth intermediate data using the third output data of the third operation. The fourth intermediate data includes second subgroups of key-value pairs.
[0064] In operation (340), the processing logic performs the fourth reduce phase of the fourth task of the set of MapReduce tasks to generate the fourth output data using the fourth intermediate data. Each reducer of the fourth reduce phase receives each key-value pair of the fourth intermediate data having the same keys. In each reducer, each value of each key-value pair is paired with all the respective remaining values of each key-value pair to generate new values (e.g., cross product) having new keys. The fourth output data includes new key-value pairs from each of the reducers of the fourth task. The fourth MapReduce task of the set of MapReduce tasks is further described with respect to FIG. 4d.
[0065] In operation (345), the processing logic performs the fifth map phase of the fifth operation of the set of map-reduce operations to generate fifth intermediate data using the fourth output data of the fourth operation. The fifth intermediate data includes new key-value pairs from each of the reducers of the fourth operation.
[0066] In operation (350), the processing logic performs the fifth reduce phase of the fifth operation of the set of MapReduce operations to perform a deduplication operation to remove duplicate key-value pairs from new key-value pairs from each of the reducers of the fourth operation and to provide a plurality of cross products for a first group of key-value pairs. As mentioned above, similar operations (315 to 350) may be performed for a first group of key-value pairs and other groups of key-value pairs (generated in the first operation) to generate a plurality of cross products for the dataset. Each of the cross products is a cross product for a subset of the original dataset (e.g., a non-nested subset).
[0067] FIG. 4a illustrates a first operation of a set of MapReduce operations for generating cross-products from a dataset based on join fields, according to embodiments of the present disclosure. The first MapReduce operation (400) illustrates a mapper (410), and reducers (414A) and (414B) (generally referred to as "reducer(s) (414)"). A request is received to perform a set of MapReduce operations for generating cross-products from a dataset based on join fields. In the illustrated example, an input (411) is received by the mapper (410). The input (411) may include a dataset for performing a set of MapReduce operations. As illustrated, the dataset includes 10 entries, and each entry includes data. For example, entry (416) includes the data "User 1:IP2". The join field can be included as a parameter in a request to execute a set of MapReduce operations that generate crossproducts from a dataset. In the current example, the join field is "IP" (e.g., Internet Protocol address), which indicates that crossproducts should be generated for subsets of the dataset, each having a specific value. For instance, all entries with "IP1" are included in the subset for which crossproducts are generated, and all entries with "IP2" are included in the other subset for which other crossproducts are generated. No crossproducts are generated between entries containing "IP1" and entries containing "IP2".
[0068] In the embodiments, the mapper (410) performs a map task using a dataset (e.g., input (411)) and generates a plurality of groups of key-value pairs. For example, the mapper (410) uses data within each entry to identify a key based on a join field "IP". The mapper (410) identifies a first subset of entries in the dataset containing "IP1" and generates a group of key-value pairs (413A) based on the first subset of entries. The mapper (410) identifies a second subset of entries in the dataset containing "IP2" and generates a group of key-value pairs (413B) based on the second subset of entries.
[0069] It should be noted that in some embodiments, the dataset may contain any number of entries. For example, the dataset may contain millions of entries. Thousands of groups of key-value pairs may be generated from the dataset, where each group of key-value pairs is based on the same key. Each group of key-value pairs may ultimately have thousands to millions of key-value pairs, all having the same keys. Additionally, it should be noted that in FIGS. 4a through 4f, each map phase of the corresponding map operation may use one or more mappers, and each reduce phase of the corresponding reduce operation may use one or more reducers.
[0070] As mentioned above, entries in a dataset can be converted into key-value pairs using one or more mapping operations of the map task. In some embodiments, some data of an entry is used as a key (e.g., "IP1" or "IP2"), and the entire data of the entry is retained and becomes the value in the key-value pair. A join field (e.g., "IP") indicates which data of the entry should be the key (e.g., key data). If the key data (e.g., "IP1" or "IP2") is the same in multiple entries, these entries will generate key-value pairs for a group of key-value pairs (e.g., group of key-value pairs (413A) or group of key-value pairs (413B)).
[0071] The output (412) (e.g., first intermediate data) of the mapper (410) includes a group (413A) of key-value pairs and a group (413B) of key-value pairs. The output (412) is used as an input (not shown) for the reduce phase of the first map-reduce operation (400). The key-value pairs of each group are sent to different reducers (414A and 414B) according to the key, and accordingly, key-value pairs of group (413A) having the same keys ("IP1") are sent to reducer (414A), and key-value pairs of group (413B) having the same keys ("IP2") are sent to reducer (414B). In embodiments, key-value pairs having the same keys are sent to a specific reducer.
[0072] In the reduce phase, the reducer (414) receives the output (412) from the mapper (410). The reducers (414) modify the output (412) so that the keys of the groups of key-value pairs (413A and 413B) are modified (e.g., first modified keys) so that the output (415A) and output (415B) are sorted and re-grouped (e.g., creation of subgroups) in the next reduce phase of the second map operation (420).
[0073] For example, a group of key-value pairs (413A) is passed to a reducer (414A). The reducer (414A) performs a reduce task to count the total number of key-value pairs within the group of key-value pairs (413A). One of the key-value pairs in the group of key-value pairs (413A) is modified to a sort ID (417), and accordingly, in the next reduce phase of the second map operation (420), the group of key-value pairs (413A) is sorted so that the position of the key-value pair having the sort ID (417) in the sort order is known.
[0074] In the present example, the sort ID (417) contains a value that allows a selected key-value pair of the group of key-value pairs (413A) to become the first or initial key-value pair after the sort operation. In the present example, the key "IP1" of the last key-value pair (419) of the group of key-value pairs (413A) is modified to ("0") to identify that the key-value pair (419) should be the first key-value pair in the sort order. Other key-value pairs of the group of key-value pairs (413A) are modified to other sort IDs. The other sort IDs of the other key-value pairs are numbers greater than the sort ID (417) ("0") to identify that the other key-value pairs are sorted after the last key-value pair (419) in the sort order. Other keys of the other key-value pairs are modified to "1", which indicates that they should be sorted after the key-value pair (419). It should be noted that the key-value pair (419) may contain additional information (e.g., group size ID (418)) that can be used by the next reduce phase of the second map operation (420). Knowing the location (placement in sort order) of the key-value pair (419) allows the next reduce phase of the second map operation (420) to know the location to look for to extract information (e.g., group size ID (418)) from the modified key of the key-value pair (419).
[0075] In the embodiments, the reducer (414A) counts the number of key-value pairs within the group of key-value pairs (413A). The key of the key-value pair (419) is further modified to indicate the number of key-value pairs within the group of key-value pairs (413A) (e.g., group size ID (418)). In the example, there are 5 key-value pairs, and the key of the key-value pair (419) is further modified to "5" to indicate the number of key-value pairs within the group of key-value pairs (413A).
[0076] It should be noted that the modified keys of the group of key-value pairs (413A) are also referred to as the first modified keys. The first modified keys represent the number and sort order of key-value pairs within the group of key-value pairs (413A). It should be additionally noted that similar operations may be performed for the group of key-value pairs (413B). For clarity, the subsequent MapReduce operations described with respect to FIGS. 4b through 4f describe the Map and Reduce tasks performed for the group of key-value pairs (413A). It may be understood that similar Map and Reduce tasks may be performed for the group of key-value pairs (413B) even if not explicitly mentioned.
[0077] FIG. 4b illustrates a second operation of a set of MapReduce operations for generating external products from a dataset based on join fields, according to embodiments of the present disclosure. The second MapReduce operation (420) illustrates a mapper (430) and a reducer (424). The second MapReduce operation (420) divides a group of key-value pairs (413A) into subgroups so that processing in subsequent MapReduce operations of the set of MapReduce operations becomes more efficient. The second MapReduce operation (420) generates subgroups of a group of key-value pairs (413A) having a size less than or equal to the maximum number of key-value pairs.
[0078] The mapper (430) receives the output (415A) of the reducer (414A) of the first MapReduce job (400) of the set of MapReduce jobs. The output (415A) contains a first group (413A) of key-value pairs having first modified keys. The output (415A) from the reducer (414A) of the first MapReduce job (400) is used as an input to the mapper (430). The mapper (430) uses the input and generates second intermediate data such as the output (421). In this example, the mapper (430) reads the group (413A) of key-value pairs without modifying the group (413A) of key-value pairs and passes it to the reducer (424). The output (421) of the mapper (430) is used as an input (422) to the reducer (424). Logic can be implemented so that all key-value pairs having the same prefix (e.g., "IP1") containing information from the key before the colon are sent to the same reducer.
[0079] In the embodiments, the reducer (424) performs the reduce phase of the second map-reduce operation (420) to generate the output (423) using intermediate data (e.g., output (421)) from the mapper (430). The reducer (424) modifies the first modified keys of the group of key-value pairs (413A) to generate second modified keys that identify subgroups (e.g., subgroups (425A, 425B, and 425C)) of the group of key-value pairs (413A).
[0080] In the embodiments, the reducer (424) sorts a group of key-value pairs (413A) in sort order using first modified keys, in particular sort IDs. The sort ID (417) represented as "0" after the first colon in the key of the key-value pair (419) of the group of key-value pairs (413A) is sorted to become the first key in the sort order. The remaining key-value pairs of the group of key-value pairs (413A) have larger sort IDs and are ordered after the key-value pair (419).
[0081] In the embodiments, the reducer (424) identifies the number of key-value pairs in the group of key-value pairs (413A), indicated by the key of the first key-value pair, i.e., the key of the key-value pair (419). For example, the reducer (424) knows which position in the sort order contains the key-value pair (in this case, the first position) having the key with the group size ID (418). The reducer (414A) parses the key of the first key-value pair (e.g., key-value pair (419)) to identify the group size ID (418) (e.g., "5") that identifies the number of key-value pairs in the group of key-value pairs (413A).
[0082] In embodiments, the reducer (424) determines the number of key-value pairs for each of the subgroups (425A, 425B, and 425C) (generally referred to as "subgroup(s) (425)"). The subgroups (425) are non-nested subgroups of the group of key-value pairs (413A). In embodiments, a parameter (e.g., a maximum number parameter) indicates the maximum number of key-value pairs that each of the subgroups (425) may contain, and the number of key-value pairs within any of the subgroups (425) does not exceed the maximum number of key-value pairs identified by the maximum number parameter. In some embodiments, the initial request to perform a set of MapReduce operations that generate cross products includes the maximum number parameter. In other embodiments, the maximum number parameter may be determined in advance and may be part of the script or source code of the set of MapReduce operations. In the current example, the maximum number parameter indicates that the maximum number of key-value pairs within the subgroup (425) cannot exceed two key-value pairs.
[0083] In the embodiments, the reducer (424) generates second modified keys for subgroups of key-value pairs (e.g., subgroups (425)) such that the second modified keys identify a specific subgroup from subgroups (425) and the specific subgroup does not exceed the maximum number of key-value pairs. For example, subgroup (425A) contains the maximum number of key-value pairs (e.g., two key-value pairs). The keys of subgroup (425A) are modified to read as "0-2" after a colon, which indicates that the key-value pairs of subgroup (425A) belong to subgroup "0" of three subgroups. Similarly, the keys of subgroup (425B) are modified to read as "1-2" after a colon, which indicates that the key-value pairs of subgroup (425B) belong to subgroup "1" of three subgroups. The key of subgroup (425C) is modified to be read as "2-2" after the colon, which indicates that the key-value pair of subgroup (425C) belongs to subgroup "2" of the three subgroups. It should be noted that subgroup (425C) contains only one key-value pair because there are no additional key-value pairs available to fill the subgroup.
[0084] FIG. 4c illustrates a third operation of a set of map-reduce operations for generating external products from a dataset based on join fields according to embodiments of the present disclosure. The third map-reduce operation (440) illustrates a mapper (431) and reducers (433A, 433B, and 433C) (generally referred to as "reducer(s) (433)"). The third map-reduce operation (440) creates duplicate copies of key-value pairs of subgroups (425) and combines the copies of key-value pairs with the key-value pairs of each subgroup (425). In some cases, the keys of the copied key-value pairs are modified. Duplicate the key-value pairs of subgroups (425) and modify at least the keys of the copied key-value pairs, in the fourth MapReduce job (450), so that key-value pairs having the same values (but different keys) are sent to different reducers. The fourth MapReduce job (450) will generate an external product for a group of key-value pairs (e.g., a group of key-value pairs (413A)). By duplicating the key-value pairs of subgroups (425) and modifying at least the keys of the copied key-value pairs in the third MapReduce job (440), the processing load is more evenly distributed across the reducers of the fourth MapReduce job (450), and even if the operation is distributed across multiple reducers, each value of the group of key-value pairs (413A) can be paired with each remaining value of the group of key-value pairs (413A). Each of the reducers in the fourth map-reduce job (450) can receive a predetermined maximum number of key-value pairs (at least partially based on the previous generation of subgroups (425) in the map-reduce job (420) for including a maximum number of key-value pairs).
[0085] In the embodiments, to perform the map phase of the third map-reduce operation (440), the mapper (431) receives an output (423) from the reducer (424) of the second map-reduce operation (420). The output (423) becomes the input of the mapper (431) and includes subgroups (425) of a group (413A) of key-value pairs. In a specific example, the mapper (431) does not modify the subgroups (425) of the group (413A) of key-value pairs. The subgroups (425) become intermediate data (e.g., output (432)). The intermediate data is read and passed to the reducers (433) so that each subgroup (425A, 425B, and 425C) is passed to a different reducer (433A, 433B, and 433C) based on the subgroup keys. In the embodiments, key-value pairs having the same key are passed to the same reducer.
[0086] In the embodiments, during the reduce phase of the third MapReduce task (440), the reducers (433A, 433B, and 433C) each receive a subgroup (425A, 425B, and 425C). The subgroup (425A, 425B, and 425C) is used as an input (434A, 434B, and 434C) by each reducer (433). The reducers (433A, 433B, and 433C) each generate an output (435A, 435B, and 435C). The outputs (435A, 435B, and 435C) of the reducers (433) each include second subgroups of key-value pairs (436A, 436B, and 436C). Subgroups (436A, 436B, and 436C) are generated from the first subgroups (425A, 425B, and 425C). The subgroups (436A, 436B, and 436C) include key-value pairs from each of the subgroups (425A, 425B, and 425C), and duplicate key-value pairs for each of the subgroups (425A, 425B, and 425C). At least the keys of the duplicate key-value pairs are modified.
[0087] For example, the reducer (433A) receives an input (434A) from the mapper (431). The input (434A) contains key-value pairs from the subgroup (425A). The reducer (433A) passes the key-value pairs from the subgroup (425A) to the output (435A) (e.g., the second and fourth key-value pairs of the subgroup (436A)). The reducer (433A) also duplicates the key-value pairs of the subgroup (425A) and modifies the keys of the duplicate key-value pairs (e.g., the first and third key-value pairs of the subgroup (436A)).
[0088] In another example, the reducer (433C) receives an input (434C) from the mapper (431). The input (434C) contains key-value pairs from a subgroup (425C). The reducer (433C) passes the key-value pairs from the subgroup (425C) to an output (435C), duplicates the key-value pairs from the subgroup (425C), and modifies the keys of both the passed key-value pairs from the subgroup (425C) and the duplicate key-value pairs.
[0089] It should be noted that the reducers (433) modify the keys to include reducer IDs. Reducer IDs can be used so that keys having the same reducer IDs are transferred to the same reducer in subsequent MapReduce operations (e.g., the fourth MapReduce operation (450)). Reducer IDs are indicated after the colon of the keys. For example, in subgroup (436A), the first key-value pair represents reducer ID "0-1", the second key-value pair represents reducer ID "0-2", and the third key-value pair represents reducer ID "0-1". In some embodiments, the generation of keys for subgroups (436) is optimized to reduce the number of redundant extra-records generated in the fourth MapReduce operation (450), while ensuring that every individual value of the group of key-value pairs (413A) is paired with every individual remaining value of the group of key-value pairs (413).
[0090] In one example, key optimization can be exemplified in a reducer (433A). The input (434A) includes a subgroup (425A). Information in the keys of the subgroup (425A) after the colon (i.e., "0-2") indicates the subgroup ID and the number of subgroup IDs. For the subgroup (425A), the subgroup ID is "0" and the number of subgroup IDs is "2" (where the total number of groups is 3—groups "0", "1", and "2"). The reducer (443A) will iterate through the subgroup IDs across all possible subgroup IDs. For example, the reducer (433A) will generate the subgroup ID-numbers of the subgroup IDs as "0-0", "0-1", "0-2". If the repetition of the subgroup ID-number of the subgroup ID does not repeat numbers (e.g., "0-1", "0-2"), the resulting subgroup ID-number of the subgroup ID is used as part of the key for the subgroup (436A). If the repetition of the subgroup ID-number of the subgroup ID repeats numbers (e.g., "0-0"), the resulting subgroup ID-number of the subgroup ID is not used as part of the key for the subgroup (436A).
[0091] FIG. 4d illustrates a fourth operation of a set of MapReduce operations for generating cross products from a dataset based on join fields according to embodiments of the present disclosure. The fourth MapReduce operation (450) illustrates a mapper (441) and reducers (443A, 443B, and 443C) (generally referred to as "reducer(s) (443)"). In the fourth MapReduce operation (450), each reducer (443) generates pairs of values (e.g., cross product pairs) such that each value of a key-value pair in a particular reducer (443) is paired with each respective remainder value of the remaining key-value pairs. In embodiments, the fourth MapReduce operation (450) effectively generates cross products for a group of key-value pairs (413A). In the embodiments, the output of the reduce phase of the fourth MapReduce job (450) may include duplicate key-value pairs (e.g., external pairs) that can be removed in the fifth MapReduce job (460). The outputs (445A, 445B, and 445C) of each reducer (443) (generally referred to as "output(s) (445)") include new key-value pairs having values that include paired values (e.g., external pairs).
[0092] In the embodiments, the outputs (435) of the third MapReduce operation (440), which include subgroups (436) of groups (413A) of key-value pairs, are transmitted to the mapper (441) of the fourth MapReduce operation (450) to be used as inputs to the mapper (441). In the map phase of the fourth MapReduce operation (450), the mapper (441) uses the outputs (435) of the third MapReduce operation (440) to generate intermediate data. The intermediate data includes subgroups (436) (e.g., second subgroups of key-value pairs). The mapper (441) reads the input (e.g., subgroups (436) of key-value pairs) and transmits each of the subgroups (436) of key-value pairs to each of the reducers (443). In the embodiments, key-value pairs having the same keys are transmitted to the same reducer, and accordingly, subgroup (436A) is transmitted to reducer (443A), subgroup (436B) is transmitted to reducer (443B), and subgroup (436C) is transmitted to reducer (443C). Subgroups (436A, 436B, and 436C) are each used as inputs (444A, 444B, and 444C) in each of the respective reducers (443).
[0093] In the embodiments, the reducers (443) perform the reduce phase of the fourth map-reduce operation (450) to generate outputs (445A), (445B), and (445C) (generally referred to as "output (445)"). In each of the reducers (443), to generate new values with new keys (e.g., new key-value pairs that are external pairs), each value of each key-value pair of the subgroup (436) is paired with each remaining value of each key-value pair of the subgroup (436).
[0094] For example, the reducer (443C) receives an input (444C) containing subgroups (436C) that all share the same key "IP1:0-2". The values of the subgroups (436C) are all different. The reducer (443C) generates new key-value pairs as shown in the output (445C), where each value of the subgroup (436C) is paired with each other value of the subgroup (436C). As illustrated, the key-value pairs of the output (445C) include "[User 1:IP1, User 4:IP1]", "[User 1:IP1, User 5:IP1]", and "[User 4:IP1, User 5:IP1]".
[0095] In some embodiments, the new keys of the new key-value pairs exemplified in the output (445) use unique IDs associated with the entry(s) of the dataset from which the values of the new key-value pairs originated (e.g., derived). For example, the first key-value pair in the output (445A) of the reducer (443A) includes the key "Q1:Q5". "Q1" represents the unique ID associated with the first entry of the dataset from which the value "User 1:IP1" was derived (e.g., input (411) of FIG. 4a), and "Q5" represents the unique ID associated with the fifth entry of the dataset from which the value "User 5:IP1" was derived. Using unique IDs to associate the values of the new key-value pairs in the output (445) with their entries in the original dataset helps to eliminate duplicate data (e.g., eliminate duplicate data). For example, there may be situations where the values of different key-value pairs (e.g., the external pairs of new key-value pairs in the output (445)) are identical but originate or are derived from different entries in the dataset. In such cases, an administrator or other user may want to retain duplicate values derived from different entries.
[0096] In some embodiments, a unique ID identifying a specific entry in the dataset from which the value of a key-value pair is obtained may be associated with the value through a set of MapReduce operations. For example, the unique ID associated with the value may be inserted into an entry in the dataset, and accordingly, in the first MapReduce operation (400), the unique ID becomes part of the value of the key-value pair and effectively moves through the value across a number of MapReduce operations as described herein. In the fourth MapReduce operation (450), the unique ID may be retrieved from each value of the key-value pair and used as part of a new key. In some cases, the unique ID may be removed from the value during the reduce phase of the fourth MapReduce operation (450).
[0097] In some embodiments, a physical node may run multiple reducers. A memory unit (e.g., a cache) may be established by the MapReduce framework, and accordingly, each physical node may access a predetermined amount of memory in the memory unit to store data. In some embodiments, the memory unit associated with a physical node may be configured to store keys (or key-value pairs) generated by reducers associated with a specific physical node. The memory unit may be used for deduplication, and accordingly, the cache may be checked to determine if a key already exists before writing a new key-value pair to the output of the reducer. If the key exists in the memory unit, the key-value pair is not written to the output of the reducer. If the key does not exist in the memory unit, the key-value pair is written to the output of the reducer.
[0098] FIG. 4e illustrates a fifth operation of a set of map-reduce operations for generating cross products from a dataset based on join fields, according to embodiments of the present disclosure. The fifth map-reduce operation (460) illustrates a mapper (461) and a reducer (463). In the fifth map-reduce operation (460), duplicate key-value pairs are removed (e.g., duplicate removal), and cross products (without duplicate key-value pairs) for a group of key-value pairs (413A) are stored in a data store.
[0099] In the embodiments, during the map phase of the fifth MapReduce job (460), the mapper (461) receives the output (445) of the reducers (443) of the fourth MapReduce job (450). The output (445) is used as an input to the mapper (461). The mapper uses the input to generate intermediate data. The intermediate data includes new key-value pairs generated by the reducers (443) of the fourth MapReduce job (450). The mapper (461) reads the data and passes the data to the reducer (463) without modification.
[0100] In the embodiments, the reduce phase of the fifth MapReduce operation (460) is performed by a reducer (463). The reduce phase performs a deduplication operation to remove duplicate key-value pairs from the new key-value pairs. The output (462) provides the cross product for a subset of the dataset (all entries having "IP1") from which the group of key-value pairs (413A) was derived. The output (462) is stored in a data store. It should be noted that a single reducer (463) is illustrated for the purpose of clarity rather than limitation. In some embodiments, each key-value pair within the output (462) of the mapper (461) having the same key is sent to a specific reducer. For example, key-value pairs having keys "Q1:Q5" are sent to a first reducer, and key-value pairs having keys "Q2:Q3" are sent to a second reducer. Each reducer can perform duplicate removal on the received key-value pairs.
[0101] Additionally, it should be noted that the set of operations described herein includes a first MapReduce operation (400), a second MapReduce operation (420), a third MapReduce operation (440), a fourth MapReduce operation (450), and a fifth MapReduce operation (460). In other embodiments, one or more of the MapReduce operations may be modified. For example, FIG. 4f illustrates an alternative mapping phase that may be implemented in the first MapReduce operation (400). In other examples, a cache that helps perform a deduplication operation may be used for the MapReduce operation (450). In some embodiments, one or more operations of the MapReduce operations of the set of MapReduce operations may be combined. For example, the operations of the MapReduce operation (420) and the MapReduce operation (440) may be combined into a single MapReduce operation. In some embodiments, not all MapReduce tasks of the set of MapReduce tasks are performed. For example, in some implementations, the MapReduce task (460) may be replaced with the cache-based deduplication operation described for the MapReduce task (450).
[0102] FIG. 4f illustrates a map phase of a first operation of a set of map-reduce operations for generating cross products from a dataset containing time ranges according to embodiments of the present disclosure. In embodiments, the map phase (470) may be used as an alternative to the map phase illustrated in the first map-reduce operation (400) of FIG. 4a. In some embodiments, the entries of the dataset from which cross products are to be generated may include time data types such as time ranges. For example, an entry (473) of a dataset (e.g., input (472)) includes the time range "12:01-12:02". In some cases, it may be beneficial to generate cross products of the data to understand whether the data overlaps in time. For example, a collaboration platform (120) may determine whether multiple users are playing a game simultaneously. The map phase (470) can generate key-value pairs for different times within a time range (e.g., 12:02, 12:03). In the embodiments, the map phase increments the time range by time units, and a new key-value pair is generated at each timestamp of the increment.
[0103] In the embodiments, a request is received to perform a set of MapReduce tasks to generate cross products from a dataset based on a join field. In the illustrated example, an input (472) is received by the mapper (471). The input (472) may include a dataset to perform a set of MapReduce tasks. As illustrated, the dataset includes two entries containing data. For example, entry (473) contains the data "User 2 Session:12:01-12:02". A join field may be included as a parameter in the request to perform a set of MapReduce tasks. In the present example, the join field is "time range", which indicates that keys should be generated based on the "time range field" of the entries in the dataset, and cross products should be generated for a subset of the dataset associated with the same key (e.g., the same timestamp).
[0104] In some embodiments, the map phase (470) identifies a time unit parameter associated with a request to generate external products. The time unit parameter represents a time unit. In the present example, the time unit is one minute. The time unit can be set to any value, such as hours, days, etc.
[0105] In the embodiments, for each entry of the dataset, the mapper (471) identifies data within the time range data field within the entry. The data within the time range field may be referred to as the time range. The mapper (471) increments the time range by time units. The increments start from the earliest time identified by the time range and continue to the last time identified by the time range. In each increment, a timestamp is generated that reflects the time within the time range in the increment. The mapper (471) generates one or more key-value pairs for the entry based on the increments. The key of one or more key-value pairs identifies the timestamp of the increment. The value of one or more key-value pairs identifies the data of the entry. In the embodiments, the mapper (471) generates groups of key-value pairs. Each key-value pair in each group of key-value pairs contains the same timestamp.
[0106] For example, entry (476) includes the time range "12:01-12:03", and entry (473) includes the time range "12:01-12:02". The time unit is one minute. For entry (476), the mapper starts with the earliest time identified by the time range "12:01-12:03", i.e., "12:01", and increments the earliest time by one minute (e.g., 12:02). "12:02" is a timestamp generated by reflecting the time within the time range in the increment. The time range "12:01-12:03" is incremented again by one minute to "12:03". "12:03" is a timestamp and is the last time identified by the time range. The timestamps "12:02" and "12:03" are the keys, and the values are the data within each entry (e.g., "User 1 Session: 12:01-12:03"). The resulting key-value pairs are grouped by key into groups of key-value pairs (475A and 475B) and passed to the reducer phase of the MapReduce job (400) of FIG. 4a.
[0107] FIG. 5 is a block diagram illustrating an exemplary computer system (500) according to embodiments. The computer system (500) executes one or more sets of instructions that cause a machine to perform any one or more of the methods discussed herein. The set of instructions, instructions, etc. may refer to instructions that, when executed by the computer system (500), cause the computer system (500) to perform one or more operations of an external module (140). The machine may operate as a server or client device in a client-server network environment, or as a peer machine in a peer-to-peer (or distributed) network environment. The machine may be a personal computer (PC), tablet PC, set-top box (STB), personal digital assistant (PDA), mobile phone, web device, server, network router, switch or bridge, or any machine capable of executing a set of instructions (sequentially or otherwise) that specifies actions to be performed by the machine. Additionally, although only a single machine is exemplified, the term “machine” will also be understood to include any collection of machines that execute sets of instructions individually or collectively to perform any one or more of the methods described herein.
[0108] A computer system (500) includes a processing device (502), a main memory (504) (e.g., read-only memory (ROM), flash memory, dynamic random access memory (DRAM), such as synchronous DRAM (SDRAM) or Rambus DRAM (RDRAM), etc.), a static memory (506) (e.g., flash memory, static random access memory (SRAM), etc.), and a data storage device (516), and these communicate with each other via a bus (508).
[0109] The processing device (502) represents one or more general-purpose processing devices, such as a microprocessor, a central processing unit, etc. More specifically, the processing device (502) may be a complex instruction set computing (CISC) microprocessor, a reduced instruction set computing (RISC) microprocessor, a very long instruction word (VLIW) microprocessor, or a processing device implementing other instruction sets or a combination of instruction sets. The processing device (502) may also be one or more special-purpose processing devices, such as an application-specific integrated circuit (ASIC), a field programmable gate array (FPGA), a digital signal processor (DSP), a network processor, etc. The processing device (502) is configured to execute instructions of the system architecture (100) and the external module (140) to perform the operations discussed herein.
[0110] The computer system (500) may further include a network interface device (522) that provides communication with other machines through a network (518) such as a local area network (LAN), intranet, extranet, or the Internet. The computer system (500) may also include a display device (510) (e.g., a liquid crystal display (LCD) or a cathode ray tube (CRT)), an alphanumeric input device (512) (e.g., a keyboard), a cursor control device (514) (e.g., a mouse), and a signal generating device (520) (e.g., a speaker).
[0111] The data storage device (516) may include a non-transient computer-readable storage medium (524), and on the non-transient computer-readable storage medium (524), a set of instructions of a system architecture (100) and an external module (140) that implement any one or more of the methods or operations described herein are stored. The set of instructions of the system architecture (100) and the external module (140) may also reside in a processing device (502) and / or in a main memory (504) wholly or at least partially during their execution by the computer system (500), and the main memory (504) and the processing device (502) also constitute computer-readable storage media. The set of instructions may additionally be transmitted or received through a network (518) via a network interface device (522).
[0112] Although an example of a computer-readable storage medium (524) is illustrated as a single medium, the term “computer-readable storage medium” may include a single medium or multiple media (e.g., a centralized or distributed database and / or associated caches and servers) that store sets of instructions. The term “computer-readable storage medium” may include any medium capable of storing, encoding, or carrying sets of instructions for execution by a machine and for causing the machine to perform any one or more of the methods of the present disclosure. The term “computer-readable storage medium” may include, but is not limited to, solid-state memories, optical media, and magnetic media.
[0113] In the foregoing description, numerous details are provided. However, it will be apparent to a person skilled in the art who has an interest in the present disclosure that the present disclosure can be practiced without these specific details. In some cases, to avoid obscuring the present disclosure, widely known structures and devices are not described in detail but are shown in block diagram form.
[0114] Parts of the detailed description have been presented regarding symbolic representations and algorithms of operations on data bits within computer memory. These descriptions and representations of algorithms are means used by a person of ordinary skill in the field of data processing technology to most effectively convey the content of their work to another person of ordinary skill in the relevant field. Here and generally, an algorithm is conceived as a self-consistent sequence of operations leading to a desired result. A sequence of operations requires physical manipulation of physical quantities. These quantities generally, though not necessarily, take the form of electrical or magnetic signals that can be stored, transmitted, combined, compared, and manipulated in other ways. Primarily for reasons of common use, it has sometimes proven convenient to refer to these signals as bits, values, elements, symbols, characters, terms, numbers, etc.
[0115] However, it should be noted that all these and similar terms will be associated with appropriate physical quantities and are merely convenient labels applied to such quantities. Unless specifically stated otherwise, it is understood that throughout the description, discussions utilizing terms such as "hosting," "determining," "receiving," "providing," "transmitting," "identifying," "monitoring," "increasing," "performing," etc., refer to actions and processes of a computer system or similar electronic computing device that manipulate and convert data represented as physical (e.g., electronic) quantities within computer system memories or registers into other data similarly represented as physical quantities within computer system memories or registers or other such information storage, transmission, or display devices.
[0116] The present disclosure also relates to an apparatus for performing the operations of the present invention. Such an apparatus may be specifically configured for the required purposes, or the apparatus may include a general-purpose computer that is selectively activated or reconfigured by a computer program stored in the computer. Such a computer program may be stored on any type of disk including a floppy disk, an optical disk, a compact disk read-only memory (CD-ROM), a magneto-optical disk, a read-only memory (ROM), a random access memory (RAM), an erasable programmable read-only memory (EPROM), an electrically erasable programmable read-only memory (EEPROM), a magnetic or optical card, or any type of media suitable for storing electronic instructions, but not limited thereto.
[0117] In this application, the words “yes” or “exemplary” are used to mean serving as an example, case, or illustration. Any mode or design described herein as “yes” or “exemplary” is not to be interpreted as being more desirable or advantageous than other modes or designs. Rather, the use of the words “yes” or “exemplary” is intended to present concepts in a specific manner. As used in this application, the term “or” is intended to mean an inclusive “or” rather than an exclusive “or.” That is, unless otherwise specified or evident from the context, it is intended to mean any of the natural inclusive substitutions of “X includes A or B.” That is, if X includes A; if X includes B; or if X includes both A and B, then “X includes A or B” is satisfied under either of the foregoing cases. Additionally, singular forms as used in this application and the appended claims may generally be interpreted to mean "one or more" unless otherwise specified or evident from the context relating to the singular form. Furthermore, the use of terms such as "implementation," "one implementation," "an example," or "one embodiment" throughout is not intended to mean the same implementation or embodiment unless described as such. One or more implementations or embodiments described herein may be combined in a particular implementation or embodiment. Terms such as "first," "second," "third," "fourth," etc., as used herein, are meant as labels to distinguish different elements and may not necessarily have an ordinal meaning according to their numerical designations.
[0118] It should be understood that the foregoing description is intended to be illustrative and not limiting. Other embodiments will be obvious to a person skilled in the art upon reading and understanding the foregoing description. Accordingly, the scope of the present disclosure may be determined with respect to the appended claims, together with the full scope of equivalents granted by the appended claims.
[0119] In additional embodiments, one or more processing devices for performing the operations of the embodiments described above are disclosed. Additionally, in embodiments of the present disclosure, a non-transient computer-readable storage medium stores instructions for performing the operations of the described embodiments. Also, in other embodiments, systems for performing the operations of the described embodiments are also disclosed.
Claims
Claim 1 A method comprising: receiving by a processing device a request to perform a set of MapReduce tasks to generate a plurality of cross products from a dataset based on a join field ― said join field indicates that each of said cross products is to be generated from data of one subset of said dataset ―; in response to receiving the request to perform the set of MapReduce tasks, performing the set of MapReduce tasks on said dataset to generate final output data including said cross products ― said performing the set of MapReduce tasks includes distributing the MapReduce operations of said set of MapReduce tasks to each node for parallel processing ―; and storing said final output data of said set of MapReduce tasks. Claim 2 A method according to claim 1, wherein each subset of the plurality of subsets is associated with the same keys, and performing the set of map-reduce operations generates groups of key-value pairs from the plurality of subsets of the dataset, and each key-value pair of each group of key-value pairs includes the same keys. Claim 3 A method according to claim 1, wherein the join field indicates that each of the plurality of external fields must be generated by key such that each of the plurality of external fields is generated from a corresponding subset associated with the same key. Claim 4 In paragraph 2, the step of performing the set of map-reduce operations on the dataset to generate the plurality of external products comprises: performing a first map phase of the first operation of the set of map-reduce operations to generate first intermediate data using the dataset — the first intermediate data comprises groups of key-value pairs, and key-value pairs from different groups of the groups of key-value pairs have non-identical keys —; and performing a first reduce phase of the first operation of the set of map-reduce operations to generate first output data using the first intermediate data, wherein the first output data comprises a first group of groups of key-value pairs having first modified keys, and the first modified keys represent the number and sort order of key-value pairs within the first group of key-value pairs. Claim 5 In paragraph 4, the step of performing the first map phase of the first task of the set of MapReduce tasks to generate the first intermediate data using the dataset comprises: identifying a parameter associated with a request to generate the plurality of cross products — said parameter represents a time unit —; for each entry of the dataset, identifying a time range indicated in the entry and incrementing the time range by said time units — said increments start from the earliest time identified by the time range and continue to the last time identified by the time range, and in each increment, a timestamp is generated reflecting the time within the time range in the increment —; and generating one or more key-value pairs for the entry based on said increments — said key of the one or more key-value pairs identifies the timestamp of said increment, and said value of the one or more key-value pairs identifies the data of said entry —; A method comprising the step of generating groups of key-value pairs, wherein each key-value pair of each group of key-value pairs comprises the same timestamp. Claim 6 In claim 4, the step of performing the set of map-reduce operations on the dataset to generate the plurality of external products comprises: performing a second map phase of the second operation of the set of map-reduce operations to generate second intermediate data using the first output data of the first operation — the second intermediate data comprises a first group of key-value pairs including the first modified keys —; and further comprising the step of performing a second reduce phase of the second operation of the set of map-reduce operations to generate second output data using the second intermediate data, wherein the second output data comprises first subgroups of the first group of key-value pairs having second modified keys, and the first modified keys of the first group of key-value pairs are modified to generate the second modified keys that identify the first subgroups of the first group of key-value pairs. Claim 7 In claim 6, the step of performing a second reduce phase of the second operation of the set of map-reduce operations to generate the second output data using the second intermediate data comprises: sorting key-value pairs of a first group of key-value pairs in the sorting order identified by the first modified keys; identifying the number of key-value pairs within the first group of key-value pairs represented by the first key-value pair of the sorted key-value pairs; determining the number of key-value pairs for each of the first subgroups of key-value pairs for the first group of key-value pairs — wherein the number of key-value pairs within the first subgroups does not exceed the maximum number of key-value pairs —; and generating the second modified keys for the first subgroups of key-value pairs such that the second modified keys identify a specific subgroup from the first subgroups and the specific subgroup does not exceed the maximum number of key-value pairs. Claim 8 In claim 6, the step of performing the set of map-reduce operations on the dataset to generate the plurality of external products comprises: performing the third map phase of the third operation of the set of map-reduce operations to generate third intermediate data using the second output data of the second operation — the third intermediate data comprises first subgroups of the key-value pairs —; and further comprising the step of performing the third reduce phase of the third operation of the set of map-reduce operations to generate third output data using the third intermediate data, wherein the third output data comprises second subgroups of key-value pairs generated from the first subgroups of the key-value pairs, the second subgroups of the key-value pairs comprise duplicate key-value pairs of the first subgroups of the key-value pairs, and at least the keys of the duplicate key-value pairs are modified. Claim 9 In claim 8, the step of performing the set of map-reduce operations on the dataset to generate the plurality of external products comprises: performing a fourth map phase of the fourth operation of the set of map-reduce operations to generate fourth intermediate data using the third output data of the third operation — the fourth intermediate data comprises second subgroups of the key-value pairs —; and further comprising the step of performing a fourth reduce phase of the fourth operation of the set of map-reduce operations to generate fourth output data using the fourth intermediate data, wherein each reducer of the fourth reduce phase receives each key-value pair of the fourth intermediate data having the same keys, and in each reducer, each value of each key-value pair is paired with all remaining values of each key-value pair to generate new values having new keys, and the fourth output data comprises new key-value pairs from each of the reducers of the fourth operation. Claim 10 In claim 9, the step of performing the set of map-reduce operations on the dataset to generate the plurality of cross products further comprises: performing the fifth map phase of the fifth operation of the set of map-reduce operations to generate fifth intermediate data using the fourth output data of the fourth operation — the fifth intermediate data includes the new key-value pairs from each of the reducers of the fourth operation —; and performing the fifth reduce phase of the fifth operation of the set of map-reduce operations to perform a deduplication operation to remove duplicate key-value pairs from the new key-value pairs from each of the reducers of the fourth operation and to provide the cross product of the plurality of cross products for a first group of the key-value pairs. Claim 11 A system comprising: memory; and a processing device coupled to said memory, wherein the processing device receives a request to perform a set of MapReduce tasks to generate a plurality of cross products from a dataset based on a join field ― said join field indicates that each of said cross products is to be generated from data of one subset of said dataset ―; in response to receiving the request to perform the set of MapReduce tasks, perform the set of MapReduce tasks on said dataset to generate a final output data including said cross products ― said performing the set of MapReduce tasks includes distributing the MapReduce operations of said set of MapReduce tasks to each node for parallel processing ―; and a system storing said final output data of said set of MapReduce tasks. Claim 12 In paragraph 11, each subset of the plurality of subsets is associated with the same keys, and performing the set of MapReduce operations generates groups of key-value pairs from the plurality of subsets of the dataset, and each key-value pair of each group of key-value pairs includes the same keys, a system. Claim 13 A system according to claim 11, wherein the join field indicates that each of the plurality of externals must be generated by key such that each of the plurality of externals is generated from a corresponding subset associated with the same key. Claim 14 In claim 12, to perform the set of map-reduce operations on the dataset to generate the plurality of external products, the processing device further performs a first map phase of the first operation of the set of map-reduce operations to generate a first intermediate data using the dataset — the first intermediate data comprises groups of key-value pairs, and key-value pairs from different groups of the groups of key-value pairs have non-identical keys — and performs a first reduce phase of the first operation of the set of map-reduce operations to generate a first output data using the first intermediate data, wherein the first output data comprises a first group of groups of key-value pairs having first modified keys, and the first modified keys represent the number and sort order of key-value pairs within the first group of key-value pairs, a system. Claim 15 In paragraph 14, to perform the first map phase of the first task of the set of MapReduce tasks to generate the first intermediate data using the dataset, the processing device further identifies a parameter associated with a request to generate the plurality of external products — the parameter represents a time unit —; for each entry of the dataset, identifies a time range indicated in the entry, and increments the time range by the time units — the increments begin from the earliest time identified by the time range and continue to the last time identified by the time range, and in each increment, a timestamp is generated reflecting the time within the time range in the increment —; generates one or more key-value pairs for the entry based on the increments — the key of the one or more key-value pairs identifies the timestamp of the increment, and the value of the one or more key-value pairs identifies the data of the entry —; generates groups of the key-value pairs, and each of each of the groups of the key-value pairs Key-value pairs are systems containing the same timestamp. Claim 16 In claim 14, to perform the first map phase of the first operation of the set of map-reduce operations to generate the first intermediate data using the dataset, the processing device further performs the second map phase of the second operation of the set of map-reduce operations to generate the second intermediate data using the first output data of the first operation — the second intermediate data includes a first group of key-value pairs including the first modified keys — and performs the second reduce phase of the second operation of the set of map-reduce operations to generate the second output data using the second intermediate data, wherein the second output data includes first subgroups of the first group of key-value pairs having second modified keys, and the first modified keys of the first group of key-value pairs are modified to generate the second modified keys identifying the first subgroups of the first group of key-value pairs. Claim 17 In claim 16, to perform the first map phase of the first task of the set of map-reduce tasks to generate the first intermediate data using the dataset, the processing device further performs the third map phase of the third task of the set of map-reduce tasks to generate the third intermediate data using the second output data of the second task — the third intermediate data comprises the first subgroups of the key-value pairs —, performs the third reduce phase of the third task of the set of map-reduce tasks to generate the third output data using the third intermediate data, the third output data comprises the second subgroups of key-value pairs generated from the first subgroups of the key-value pairs, the second subgroups of the key-value pairs comprise the first subgroups of the key-value pairs and duplicate key-value pairs of the first subgroups of the key-value pairs, at least the keys of the duplicate key-value pairs are modified, and a fourth intermediate using the third output data of the third task A system comprising: performing a fourth map phase of a fourth task of a set of map-reduce tasks to generate data, wherein the fourth intermediate data comprises a second subgroup of the key-value pairs, and performing a fourth reduce phase of a fourth task of a set of map-reduce tasks to generate fourth output data using the fourth intermediate data, wherein each reducer of the fourth reduce phase receives each key-value pair of the fourth intermediate data having the same keys, and in each reducer, to generate new values having new keys, each value of each key-value pair is paired with all remaining values of each key-value pair, and the fourth output data comprises new key-value pairs from each of the reducers of the fourth task. Claim 18 In claim 17, to perform the first map phase of the first task of the set of map-reduce tasks to generate the first intermediate data using the dataset, the processing device further performs the fifth map phase of the fifth task of the set of map-reduce tasks to generate the fifth intermediate data using the fourth output data of the fourth task — the fifth intermediate data comprises the new key-value pairs from each of the reducers of the fourth task — and performs the fifth reduce phase of the set of map-reduce tasks to perform a deduplication operation to remove duplicate key-value pairs from the new key-value pairs from each of the reducers of the fourth task and to provide the cross product of the plurality of cross products for the first group of the key-value pairs. Claim 19 A non-transient computer-readable medium comprising instructions, wherein the instructions cause the processing device to perform operations in response to execution by the processing device, the operations comprising: receiving by the processing device a request to perform a set of MapReduce operations to generate a plurality of cross products from a dataset based on a join field ― wherein the join field indicates that each of the plurality of cross products is to be generated from data of one subset of a plurality of subsets of the dataset ―; performing the set of MapReduce operations on the dataset to generate final output data including the plurality of cross products in response to receiving the request to perform the set of MapReduce operations ― wherein performing the set of MapReduce operations includes distributing the MapReduce operations of the set of MapReduce operations to respective nodes for parallel processing ―; and storing the final output data of the set of MapReduce operations. Claim 20 In paragraph 19, each subset of the plurality of subsets is associated with the same keys, and performing the set of MapReduce operations generates groups of key-value pairs from the plurality of subsets of the dataset, and each key-value pair of each group of key-value pairs includes the same keys, a non-transient computer-readable medium.
Citation Information
Patent Citations
Mapreduce for distributed database processing
US20080086442A1
Systems and Methods for Processing Machine Learning Algorithms in a MapReduce Environment
US20120226639A1
Data processing method and distributed processing system
US20130179466A1
Data Processing Apparatus and Method for Processing Serial Tasks
US20160364273A1
Sharing intermediate data in map-reduce
US20180285440A1
