Event mining in social networks

Abstract

Procedure (100,500) for detecting an event from a social stream, wherein the procedure (100,500) comprises the steps of one: Receiving a social stream from a social network, where the social stream contains at least one object; and where the object contains a text, sender information of the text and recipient information of the text (101,510); Assigning the object to a cluster based on a similarity value between the object and the cluster (102,520); Monitoring changes in at least one of the clusters (103,540); and Triggering an alarm when changes in at least one of the clusters exceed a first threshold (104,550), where at least one of the steps is performed using a computer unit, wherein the procedure further includes maintaining changes in the clusters as history data of the social stream (530), wherein in the maintenance step a sketch-supported technique is used, and wherein the sketch-supported technique is used to estimate a structural similarity value between the object and the clusters.

Description

BACKGROUND OF THE INVENTION Area of ​​the invention

[0001] The present invention relates to event mining and, more precisely, the detection of new events from a social stream. Description of the state of the art

[0002] The problem of text mining has been extensively studied within the information gathering community due to the ubiquity of text data availability in a wide variety of scenarios, such as the web, social networks, newsfeeds, and many others. Much of this text data originates in the context of temporal applications, such as newsfeeds and social media streams, where text arrives as a continuous and massive influx of documents. Streaming applications pose a particular challenge for such problems because it is often necessary to process the data in a single pass, and not all data can be stored on disk for reprocessing.

[0003] A key challenge in the context of temporal and streaming text data is that of online event detection, which is closely related to the problem of topic detection and tracking. This challenge is also closely related to stream splitting and attempts to identify new thematic trends within the text stream and their significant development. The idea is that important and newsworthy real-life events (such as the recent unrest in the Middle East) are often captured as temporal bursts of closely related documents within a social stream. The challenge can exist in both supervised and unsupervised scenarios. In the unsupervised case, it is assumed that no training data is available to guide the stream's event detection process. In the supervised case, historical data on events is available to guide the event detection process.

[0004] A method for evaluating social flows is known from publication US 2010 / 0119053 A1.

[0005] From publication US 2007 / 0226212 A1, a method for detecting outliers in data based on clusters is known, whereby individual data points are assigned to clusters based on similarities with other data points. BRIEF SUMMARY OF THE INVENTION

[0006] Accordingly, one aspect of the present invention provides a method for detecting an event from a social stream. The method comprises the steps of: receiving a social stream from a social network, wherein the social stream contains at least one object and the object contains text, sender information of the text, and recipient information of the text; assigning the object based on a similarity value between the object and the clusters; monitoring changes in at least one of the clusters; and triggering an alarm when the changes in at least one of the clusters exceed a first threshold, wherein at least one of the steps is performed using a computer unit.

[0007] Another aspect of the present invention provides a system that recognizes an event from a social stream. The system includes: a receiver module for receiving a social stream from a social network, wherein the social stream contains at least one object and the object contains text, sender information of the text, and recipient information of the text; a cluster module for assigning the object to a cluster based on a similarity value between the object and the clusters;

[0008] A monitoring module for monitoring changes in at least one of the clusters; and a triggering module for triggering an alarm when the changes in at least one of the clusters exceed a first threshold. BRIEF DESCRIPTION OF THE DRAWINGS Fig. Figure 1 shows a flowchart illustrating a method 100 for detecting an event in a social network according to a preferred embodiment of the present invention. Fig. Figure 2 shows a system for detecting an event in a social network according to a preferred embodiment of the present invention. Fig. Figure 3 shows a hardware configuration for realizing or executing at least one embodiment of the present invention. Fig. Figure 4 shows a flowchart illustrating a method for assigning an object to an existing cluster or creating a new cluster during a partitioning step 102 according to a preferred embodiment of the present invention. Fig. Figure 5 shows a flow chart illustrating a method 500 for maintenance according to a further preferred embodiment of the invention. Fig. Figure 6 shows a detailed overall algorithm for maintenance according to a further preferred embodiment of the invention. Fig. Figure 7 illustrates, according to a preferred embodiment of the present invention, the effectiveness results of the clustering algorithm with respect to cluster purity. Fig. Figure 8 shows, according to a preferred embodiment of the present invention, the efficiency of the clustering approach with an increasing number of clusters. Fig. Figure 9 illustrates the results of the monitored event detection method according to a preferred embodiment of the present invention. DETAILED DESCRIPTION OF PREFERRED EXECUTION FORMS

[0009] The foregoing and further features of the present invention will become apparent from a detailed description of embodiments thereof, which are shown in conjunction with the attached drawings. Identical reference numerals refer to the same or similar parts in the attached drawings of the invention.

[0010] As will be apparent to those skilled in the art, aspects of the present invention can be implemented as a system, method, or computer program product. Accordingly, aspects of the present invention can be implemented as a complete hardware embodiment, a complete software embodiment (including firmware, resident software, microcode, etc.), or in an embodiment that combines software and hardware aspects, all of which may be generally referred to herein as a "circuit," "module," or "system." Furthermore, aspects of the present invention can be implemented as a computer program product, which may be contained in one or more computer-readable media with computer-readable program code included thereon.

[0011] Any combination of one or more computer-readable media can be used. A computer-readable storage medium can be, for example, but is not limited to, a system, device, or unit of an electronic, magnetic, optical, electromagnetic, infrared, or semiconductor nature, or any suitable combination thereof.More specific examples of computer-readable storage media may include (but are not limited to): an electrical connection with one or more wires, a portable computer disk, a hard disk, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disc read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the foregoing.In the context of this document, a computer-readable storage medium can be any physical medium that can contain or store a program for use by or in conjunction with a system, device or unit to execute instructions.

[0012] Computer program code for performing operations for aspects of the present invention can be written in any combination of one or more programming languages, including an object-oriented programming language such as Java, Smalltalk, C++, or similar languages, and conventional procedural programming languages ​​such as the programming language "C" or similar programming languages. The program code can be executed entirely on the user's computer, partially on the user's computer, or as a standalone software package and partially on the user's computer. Aspects of the present invention are described below with reference to flowcharts and / or block diagrams of methods, devices (systems), and computer program products according to embodiments of the invention.It is understood that each block of the flowchart and / or block diagrams, as well as combinations of blocks in the flowchart and / or block diagrams, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, a dedicated computer, or other programmable data processing device to create a machine such that the instructions executed by the processor of the computer or other programmable data processing device produce a means of implementing the functions / actions specified in the block or blocks of the flowchart and / or block diagram.

[0013] These computer program instructions may also be stored in a computer-readable medium capable of directing a computer, other programmable data processing device, or other unit to operate in a particular manner, such that the instructions stored in the computer-readable medium produce a manufactured article including instructions to perform the function / action specified in the block or blocks of the flow chart and / or block diagram.

[0014] The computer program instructions can also be loaded onto a computer, other programmable data processing device, or other units to induce a series of operational steps to be performed on the computer, other programmable device, or other units in order to generate a process realized on the computer such that the instructions executed on the computer or other programmable device provide processes for realizing the functions / actions specified in the block or blocks of the flowchart and / or block diagram.

[0015] The flowchart and block diagrams in the figures illustrate the architecture, functionality, and operation of possible implementations of systems, methods, and computer program products according to various embodiments of the present invention. In this respect, each block in the flowchart or block diagrams can represent a module, segment, or code section containing one or more executable instructions for implementing the specified logical function(s). It should also be noted that in some alternative implementations, the functions specified in the block may occur in a different order than that shown in the figures. For example, two consecutively depicted blocks may actually be executed essentially simultaneously, or the blocks may sometimes be executed in reverse order, depending on the functionality involved.It is also noted that each block of the block diagrams and / or flowchart representations and combinations of blocks in the block diagrams and / or flowchart representations can be implemented by dedicated hardware-based systems or combinations of dedicated hardware and computer instructions that perform the specified functions or actions.

[0016] The terminology used herein serves solely to describe particular embodiments and is not to be construed as limiting the invention. The singular forms "ein," "eine," and "der," "die," "das," and their declensions used herein are intended to include the plural forms as well, unless otherwise clearly indicated in the context. Furthermore, it is understood that the terms "contains" and / or "containing" in this document indicate the presence of specified features, integers, steps, processes, elements, and / or components, but do not exclude the presence or addition of one or more features, integers, steps, processes, elements, components, and / or groups thereof.

[0017] The corresponding structures, materials, actions, and correspondences of all means or steps, as well as functional elements, in the following claims are intended to include all structures, materials, or actions for carrying out the function in combination with other claimed elements, as individually claimed. The description of the present invention has been provided for illustrative and descriptive purposes but is not to be considered exhaustive or limited to the invention as disclosed. Many modifications and variations are obvious to the person skilled in the art without deviating from the scope and spirit of the invention.The embodiment was chosen and described to best explain the basic ideas of the invention and its practical application, and to enable other skilled persons to understand the invention for various embodiments with various modifications suitable for the particular use under consideration.

[0018] A social network is a social structure comprised of actors (or organizations) called "nodes" that are linked (connected) by one or more specific types of interdependence, such as friendship, kinship, shared interests, financial exchange, dislike, sexual relationships, or relationships of belief, knowledge, or prestige. The social network may also include messaging in an online chat service or an email network, through which messages are exchanged between pairs of nodes.

[0019] Because such social networks are dynamic, a number of interesting issues arise within them and are associated with a network structure in a stream. Each node represents an actor in the social network, and each message sent within the social network is text content associated with an edge in the network. Obviously, multiple messages can be sent between the same pair of nodes over time. In this case, the thematic content of the documents, their temporal distribution, and the graphical structure of the dynamic network of interactions can be used to identify interesting events and their development. Messages sent between a closely linked group of nodes can be more informative about a particular event of social interest than a set of messages that are more diffusely related from a structural perspective.If the social network is viewed as a diagram whose edges correspond to messages sent between entities, then such messages are structurally well-connected. This relates to the problem of community recognition, where finding structurally connected regions of the social network is a key challenge. At the same time, the content and topics of the documents should also play a significant role in the event recognition process.

[0020] The network location of an event can play a crucial role in its significance. For example, a major event specific to a particular university might correspond to news primarily among students and faculty of that university, whereas a more global event, such as the unrest in the Middle East, might correspond to content-related news with a broader geographical scope. In the first case, communication is likely to occur between the more closely connected group of entities, while in the latter, the news may be more global, with a focus on a Middle Eastern-related location within the social network. The ability to identify such distinct events with varying levels of location and scale within a dynamic network presents an extreme challenge in large-scale, extensive social networks.

[0021] A key problem in the streaming scenario is the assumption that, due to the limitations of processing and storage capacity for handling such large volumes of data, it cannot be stored on disk for repeated processing. In other words, all splitting or event detection algorithms must operate under the assumption of a single streaming pass, where only (memory-limited) summary data can be extracted from the incoming stream, and the raw stream is lost thereafter. Furthermore, the algorithm must be fast, as it should be able to detect events from the large volume of the social network stream in real time. Therefore, the key challenges for event detection in social streams are as follows: (a) The ability to leverage both the content and the (graphical) structure of the interactions for event detection.(b) The ability to use temporal information in the event detection process. For example, a trend of structurally and content-wise closely related text documents that were not previously encountered may correspond to a new event in the social stream. (c) The ability to handle very large and extensive volumes of text documents within the constraints of scenarios using a single streaming pass.

[0022] The problem of identifying events in streams is closely related to the problem of stream splitting, for which several methods have been proposed for different types of data. In the context of social networks, the content of such streams is text. An event in such a text stream is essentially the beginning of a new activity pattern, which can be modeled as the starting point of a new cluster in the data. Several such methods have been proposed by the text mining community in the context of topic detection and tracking. This also applies to the problem of splitting and topic modeling in dynamic text streams.

[0023] In social networks, however, a great deal of structure is available for identifying important events within the network. For example, an event corresponding to unrest in the Middle East can often correspond to text streams exchanged between members closely linked based on geographic proximity. While the use of links to identify clusters and patterns has been extensively studied by the social networking community, these methods are typically designed for static networks. Some partitioning methods have recently been designed for dynamic networks, although they do not utilize the content of the underlying network for the mining process. Conversely, some recent network pattern detection methods utilize both content and structure, although these methods are not defined for the problem of event detection in a temporal scenario.Therefore, we are designing a method that can use content, structural, and temporal information in a holistic way to identify relevant clusters and events in social flows, with the method being able to solve the unique challenge of identifying important events in the network.

[0024] Fig. Figure 1 illustrates a flowchart containing process steps according to one embodiment of the invention. In step 101, a social stream is received from a social network, wherein the social stream contains at least one object. As mentioned above, a social network is a social structure of actors (or organizations) called "nodes" that are linked (connected) by one or more specific types of mutual dependency, such as friendship, kinship, common interests, financial exchange, dislike, sexual relationships, or relationships of belief, knowledge, or prestige. The social network may further include messages in an online chat messaging service or an email network in which messages are sent between pairs of nodes.The social network generates a social stream, where the social stream is a continuous and temporal sequence of objects S1...S. r ...contains, so that each object S i This corresponds to a content-driven interaction between social entities and includes explicit content information and linking information between entities. The object S i contains a text document T i , which corresponds to the content of the interaction of an entity in the social network with one or more other entities. The object S i further contains an origin node q i ∈ N, where T is the sender of the message. i It deals with other nodes. The object S i Finally, it contains a set of one or more receiver nodes R. i ⊆ N, which is for all recipients of the message T i from node q i correspond. Thus, the message T i from the origin node qi to each node r ∈ R i sent. It is assumed that each edge (q i\ , r) belongs to sentence A.

[0025] An object in step 101, namely S i , is represented by the tuple (q i , R i , T i ). It should be noted that the above definition of social flow encompasses a number of different natural scenarios across various types of social networks. For example, in the social network “Twitter”, the document T i the content of the tweet, and the node q i corresponds to the tweeting actor. The sentence R i This corresponds to the recipients of the tweet. An email interaction network can also be considered a social network with the exact same interpretation as above. A similar argument applies to chat interaction networks. In many of these cases, the recipient set R icontain only a single node, and as a consequence, there is a stream of content-driven edge interactions. Finally, in many social networks, a post on one user's wall to another can correspond to an edge, where the document T i corresponds to the content of the post.

[0026] The social stream in step 101 typically contains comprehensive information about trends that can lead to changes in both the content and the structural location of the network in which interactions can occur. The embodiments of the present invention begin by describing an unsupervised event detection technique that continuously identifies incoming interactions in the form of clusters and uses them to report events in the data stream.

[0027] After receiving the social media feed from the social network, step 102 will be taken. Fig. 1. A splitting of the received social stream was carried out, with further details of step 102 described later. Objects from the social stream, S1 ... S r , are continuously divided into k clusters C1...C k divided so that each object S i Objects belong to at most one of cluster C. Furthermore, they are assigned to different clusters using a similarity value that captures both the content of the exchanged messages and the dynamic structure of the social network implied by the different messages. The calculation of the similarity value is described in more detail in [reference to be added]. Fig. 4 provided.

[0028] The clusters are created using both the content and link information. Because the clusters are created dynamically, they can change considerably over time with the evolving data stream, and new points are added to the clusters. Furthermore, in some cases, an incoming object may differ sufficiently from the current clusters. In this case, it can be placed in its own cluster, and one of the clusters from the set C1...C k be removed. Such an event can be interesting, especially if the newly created cluster initiates a new activity pattern in which further stream objects are subsequently added. At the same time, the events may not be entirely new, but rather their relative distribution across clusters may reflect significant changes in the patterns of incoming objects.

[0029] In step 103, changes in the clusters are monitored in at least one of the clusters. In the present embodiment of the invention, there are two types of new events, referred to as novel events and evolution events, to describe these different scenarios. The arrival of a data point S i is referred to as a novel event if it is defined as a single point within a newly created cluster C i is placed. The creation time for Cluster C i is with t(C i ). In this case, the event is the one associated with data point S. i the underlying story or theme, and not the data point itself.

[0030] The emergence of a new event can manifest itself as the novel event itself, but it can also affect the relative occurrence of existing data points across different clusters. For example, the event "Middle East unrest" might manifest either as the creation of a new cluster or as a significant addition of new data points to the clusters most closely related to that topic. This is because it is often impossible for existing clusters to closely correspond to a sudden surge of objects related to a particular topic. The sudden surge of objects within an event is called a development event. A development event is confined to a specific time horizon and represents a change in the relative activity for that particular cluster.

[0031] To determine whether the new event is a novel event or a developmental event, a fractional cluster presence function is used in step 103. For the fractional cluster presence of cluster C... i During the period (t1,t2), this refers to the proportion of data records arriving from the social stream during the period (t1,t2) that belong to cluster C. i belong. This proportional occurrence is denoted by F(t1,t2,C). i ). This sudden outbreak is characterized by a change in the proportional occurrence of data points in clusters, and such an outbreak will define a development event.

[0032] In step 103, a higher rate is determined at which data points in the previous time window of length H have arrived at the cluster compared to other data points before H. Additionally, the parameter α is used as an initial threshold to measure this development rate. The parameter α can be a predefined value and can also be configured by the user. A development event above the horizon H at the current time t c is considered to be at a first threshold α for cluster C i "occurred" is defined as the ratio of the relative occurrence of points in cluster C. i above the horizon (t c - H,t c ) before time t c - H is greater than a first threshold α. The following equation is provided below for better understanding: F(tc−H,tc,Ci)F(t(Ci),tc−H,Ci)≥α

[0033] In equation 1, it is assumed that t c - 2 · H ≥ t(C i ), where the value of t c - 2 · H is greater than the cluster creation time t(C) i ), to establish the aforementioned development ratio. This ensures that H time units are used when calculating the denominator of this ratio during step 103.

[0034] In step 104, an alarm is triggered if the changes in at least one of the clusters exceed a first threshold. In one embodiment of the invention, the first threshold is expressed as parameter α, which is a predefined value and can be configured by the user. As explained above, the event detection algorithm uses a time horizon of α for the input used in the event detection process. To perform event detection and trigger an alarm whenever the ratio of α to α exceeds a certain threshold, the algorithm uses a specific time horizon. F(tc−H,tc,Ci)F(t(Ci),tc−H,Ci) For C, the threshold of α is exceeded. This suggests a significant change in the underlying social flow, which is detected by a significant change in the ratios of flow objects assigned to the different clusters.

[0035] Fig. Figure 2 shows a system for detecting an event from a social stream according to an embodiment of the present invention. The system 200 includes a receiver module 220 that receives a social stream 210, wherein the social stream is a continuous and temporal sequence of objects S1...S r ...contains, so that each object S i corresponds to a content-based interaction between social entities and includes explicit content information and linking information between entities.

[0036] In the Fig. In the embodiment shown in 2, the system 200 also includes a cluster module 230, which continuously extracts objects from the social stream, S1...S r ... in k cluster C1...C k divides so that each object S i to at most one of cluster C r belongs. As mentioned above, the clusters are created using both the content and link information. Because the clusters are created dynamically, they can change considerably over time with the evolving flow, and new points are added to the clusters. Furthermore, in some cases, an incoming object may differ sufficiently from the current clusters. In this case, it can be placed in its own cluster, and one of the clusters from the set C1...C k be removed.

[0037] In the Fig. In the embodiment shown in Figure 2, the system 200 also includes a monitoring module 240 that monitors changes in the clusters. The monitoring module also determines, using the proportional cluster occurrence function described above, whether the new event is a novel event or an evolutionary event. Finally, the system includes a trigger module 250, which triggers an alarm when the changes in at least one of the clusters exceed a first threshold. The first threshold can be a predefined value and can be configured by the user.

[0038] Fig. Figure 3 illustrates a hardware configuration of a data processing system 1600 (e.g., the receiver module 202, the cluster module 203, the monitoring module 204, the trigger module 205) that implements and / or executes the embodiments of the present invention. The data processing system 1600 can include all modules in Fig. 2 included in the embodiment of the present invention. The hardware configuration preferably includes at least one processor or central processing unit (CPU) 1611. The CPUs 1611 are connected via a system bus 1612 to random access memory (RAM) 1614, read-only memory (ROM) 1616, an input / output (I / O) adapter 1618 (for connecting peripheral units, such as disk units 1621 and tape drives 1640, to the bus 1612), a user interface adapter 1622 (for connecting a keyboard 1624, a mouse 1626, a speaker 1628, a microphone 1632 and / or other user interface units to the bus 1612), and a data transfer adapter 1634 for connecting the system 1600 to a data processing network, the Internet, an intranet, or a personal network. Personal area network (PAN), etc., and a display adapter 1636 for connecting the bus 1612 to a display unit 1638 and / or a printer 1639 (e.g. a digital printer or similar).

[0039] In another embodiment, the receiver module 202, the cluster module 203, the monitoring module 204, and the trigger module 205 are implemented as hardware on reconfigurable hardware, e.g., an FPGA (Field Programmable Gate Array) or a CPLD (Complex Programmable Logic Device), using a hardware description language (Verilog, VHDL, Handel-C, or System C). In yet another embodiment, the topic modeling module 140, the interconnection modeling module 150, and the community modeling module 160 are implemented on a semiconductor chip, e.g., an ASIC (Application-Specific Integrated Circuit), using a semi-user-specific design approach, i.e., designing a chip using standard cells and a hardware description language.

[0040] In another embodiment, the receiver module 202, the cluster module 203, the monitoring module 204, and the trigger module 205 are implemented as software using one or more programming languages, e.g., C, C++, Java, .NET, Perl, Python, etc. In another embodiment, the receiver module 202, the cluster module 203, the monitoring module 204, and the trigger module 205 are stored in a computer-readable medium, e.g., a CD (Compact Disc), a DVD (Digital Versatile Disc), an HDD (Hard Disk Drive), or an SSD (Solid State Drive), as an instruction, machine language, or assembly language that is executed by a processor, e.g., Intel®, Core®, IBM® PowerPC®, or AMD® Opteron®.

[0041] Fig. Figure 4 illustrates in detail a method for assigning an object to an existing cluster or creating a new cluster during partitioning step 102, according to a preferred embodiment of the present invention. Fig. 4 includes the steps of: assigning the object to an existing cluster if a similarity value between the object and the existing cluster is greater than a second threshold; creating a new cluster with the object if the similarity value between the object and the existing cluster is less than a second threshold; and replacing an outdated cluster with the new cluster.

[0042] In step 410 of Fig. 4. A second threshold is then determined to determine whether the similarity value between the object and the existing cluster is greater than a certain threshold. The similarity value between the object S i and Cluster C r (Sim(Si ,C r )) is composed of both structural components SimS(S i ,C r ) as well as content components SimC(S i ,C r ) calculated. For the content component, a TF-IDF-based similarity is calculated between the text document T. i , which relates to the content of an entity's interaction with S in the social network i corresponds to, and the word summary W r used. The structural components are defined by using the node summary V. r and the node R i ∪ {q i} calculated in the social stream. The node summary V is that of cluster C. i assigned nodes, and it contains the nodes j i1 , j i2 ... j is together by v i1 ,v i2 ...v is designated node frequencies. At q i This is the origin node q. i∈ N, where T is the sender of the message. i at other nodes. In R i This refers to the set of one or more receiver nodes R. i ⊆ N, which is for all recipients of the message T i from node q i correspond. First, B(S) i ) = (b1,b2,..b sr ) as the bit vector representation of R i ∪ {q i} specified, which has one bit for each node in V. The bit value is 1 if the corresponding node in R i ∪ {q i} is contained, and is otherwise 0. The structural similarity between object S i and the frequency-weighted node set of cluster C r is defined in equation 2: SimS(Si,Cr)=∑t=1Srbt⋅vrt‖Ri∪{qi}‖⋅(∑t=1Srvrt)

[0043] In contrast to the usual use of the L2 norm, equation 2 uses the L1 norm of the node frequency vector in the denominator to penalize the creation of excessively large clusters. This results in more balanced clusters.

[0044] After calculating both the structural components SimS(S i ,C r ) as well as the content components SimC(S i ,C r ), the similarity value Sim(S) i ,C r ) is calculated as a linear combination of the structural and content-based similarity values. Equation 3 illustrates the above calculation for the similarity value Sim(S). i ,C r ): Sim(Si,Cr)=λ⋅SimS(Si,Cr)+(1−λ)⋅SimC(Si,Cr)

[0045] In equation 3, parameter λ is the adjustment parameter, which lies in the range (0,1). The parameter λ is defined by the user.

[0046] Continue with step 410 of Fig. 4 is used when calculating the similarity value between the object and the existing cluster (Sim(S)). i ,C r )) determines whether the similarity value is greater than a second threshold. The second threshold can be a user-defined value or a value of µ - 3·σ, where µ is the expected value and σ is the standard deviation of all similarity values ​​of incoming stream objects (S1...S1). r ...) to cluster summaries, the details of which will be explained later. If in step 420 of Fig. 4. If the similarity value of the next cluster is above the second threshold, the incoming stream object S i assigned to its next cluster focus. If, on the other hand, the similarity of S i with a value smaller than the second threshold, in step 430 of Fig. 4 a single-element cluster containing only a single object S i It contains, and generates corresponding cluster summary statistics. Step 440 of Fig. 4 by replacing the most outdated cluster from the current collection C1...C k through the new single-element cluster. The most outdated cluster is defined as the one that has been updated the least. In the case of a null cluster (one that has never been updated), it is automatically considered the most outdated cluster. Any bindings are arbitrarily broken.

[0047] Fig. Figure 5 illustrates a flowchart that demonstrates the process of maintaining changes in the cluster as historical social flow data. Step 530 of Fig. Figure 5 illustrates that changes in the clusters are maintained using a clustering algorithm. The clustering algorithm is assumed to take the number of clusters k as an input and maintains the structural and content information in the underlying clusters in the form of node and word frequencies within the cluster. The clusters are named C1...C k designated. The one belonging to Cluster C i The assigned set of nodes is denoted by V, and the set of words assigned to it is denoted by W. i The V set is referred to as the node summary, while the W set is referred to as the node summary. i This is referred to as word summarization. Summarization characterization can be used to assign incoming objects of the social stream to clusters, as will be described later. Sentence V contains node j. i1 , j i2 ... j is together by v i1 , vi2 ...v is designated node frequencies. The word set W i contains the word identifier l i1 ,l i2 ...l is together with through ϕ i1 , ϕ i2 ...ϕ is designated node frequencies.

[0048] The incoming object of the social stream S i is included with each cluster summary ψ i (C i ) stored. The cluster summary ψ i (C i ) contains a node summary and a content summary. The node summary contains a V i = {j i1 , j i2 ...j is} designated set of nodes and one with η i = v i1 , v i2 ...v is designated set of corresponding node frequencies, where it is assumed that the node set V is a number of S i Contains a node. The table of contents includes one with W. i = {li1 , l i2 ...l is} designated set of word identifiers and one designated with Φ i = ϕ i1 ,ϕ i2 ...ϕ iu designated set of corresponding word frequencies, assuming that W i a number of u i It contains words. Thus, the cluster summary ψ i (C i ) with ψ i (C i )=(V i ,η i ,W i ,Φ i ). In the preferred embodiment of the present invention, the set of clusters C1 ... C k together with the cluster summaries ψ1 (C1)...ψ k (C k ) stored. As new objects arrive in the social stream, the clusters are continuously updated and maintained. Simultaneously, changes in the underlying clusters are continuously tracked and maintained to trigger alerts for new events.

[0049] After the object Si its next cluster focus C r The corresponding cluster summary ψ has been assigned. r (C r ) updated. In particular, new nodes are added to S i , which are not included in the node summary V, added to V, and the frequency of the nodes of S i , which in V r The number of nodes included is increased by 1. It is noted that the nodes in S i both the source node q i as well as the target node R I correspond. In other words: the sentence R i ∪ {q i} is used to express the sentence V r and to update its membership frequencies. The same approach is used with the words in object S. i of the social flow on the words in W rapplied. The only difference in this case is that the word frequencies are increased not just by 1, but by their frequency of occurrence in the underlying document.

[0050] In step 530 of Fig. 5. The expected value and standard deviation of the (nearest) similarity values ​​of the incoming objects to the clusters are also maintained as historical data. For this purpose, the present embodiment continuously stores the zeroth, first, and second-order moments M0, M1, and M2 of the nearest similarity values. These values ​​can be easily maintained in the stream scenario, as they can be maintained additively via the data stream. The expected value µ and the standard deviation σ can be expressed with respect to these moments as follows: μ=M1 / M0,σ=M2 / M0−(M1 / M0)2

[0051] The detailed overall algorithm for cluster maintenance is in Fig. Figure 6 illustrates this. The historical data can be used when monitoring changes in at least one of the clusters, as described in step 103 of Fig. Figure 1 shows that historical data can still be created and maintained from the results of assigning an object to an existing cluster or creating a new cluster, as shown in Fig. Figure 4 shows that historical data can be used in supervised event detection, where the historical data is made available to guide the event detection process.

[0052] In the case of supervised event detection, it is assumed that access exists to the past history of the stream in which event ℇ is known to have occurred. Furthermore, information is available about at least a subset of the tweets in the social media stream that are relevant to this specific event. This constitutes the reference base that can be used for more accurate event detection.

[0053] To perform supervised event detection, changes have also been made to the clustering section of the algorithm. A major change is that replacing old clusters or creating new ones is not permitted if a new incoming point does not naturally fit into any cluster. Instead, it is always assigned to the nearest cluster, arbitrarily breaking any existing bindings. This is done to ensure stability in the clustering characteristics and is essential for characterizing events with respect to the clusters in the underlying data in a temporally consistent manner over time.

[0054] The relative distribution of event-specific stream objects across clusters is used as a signature specific to the event. This signature can be used to perform real-time detection. The assumption in the supervised case is that the training data on the social stream objects associated with the event is available in the training history data. The event signature ℇ is defined as either an event signature or a horizon signature.

[0055] The event signature of a social stream is a k-dimensional vector V(ε) that contains the (average) relative distribution of event-specific stream objects across clusters. In other words, the i component of V(ε) is the proportion of event-specific (training) stream objects assigned to cluster i.

[0056] The event signature provides a useful characterization of the relative thematic distribution during a significant event. For example, during a period of unrest in the Middle East (event ℇ), some clusters are likely to be much more active than others, and this can be captured in the vector V(ε), provided the reference basis for it is available. Event signatures can be compared to horizon signatures, which are defined in essentially the same way as event signatures, except that they are over the more recent time horizon (t). c - H,t c ) of length H are defined.

[0057] The horizon signature, which is a different type of event signature, over the last time period (t c - H,t c ) is a k-dimensional vector representing the relative distribution of objects in the social stream that occur during the time interval (t c - H,t c) have arrived, contains on cluster.

[0058] To perform supervised event detection, the dot product of the horizon signature with the known event signature (calculated using the reference base) and an alarm level equal to this value are calculated. The balance between false positives and false negatives is determined by the threshold chosen to decide when such an event has actually occurred.

[0059] The main challenge of this algorithm is that node-based statistics can be quite large. As a result, the corresponding similarity calculations and maintenance can be tedious. For example, the number of nodes in V can be r The number of variables can be on the order of tens of millions, which can make the algorithm extremely slow. Therefore, a sketch-based technique is being developed to speed up the calculations.

[0060] The sketch-based technique is a natural method for compressing the count information in the underlying data, allowing the broad characteristics of the dominant numbers to be maintained in a space-efficient manner. In this embodiment of the present invention, the count-min sketch is used to maintain node counts in the underlying clusters. The count-min sketch employs a hash approach to track the node counts in the underlying data stream. The function w = [ln(1 / δ)] is used for pairwise independent hash functions. Each of the independent hash functions is random integers in the range h = [0, eThe hash functions are assigned uniformly, where e is the base of the natural logarithm. The data structure contains a two-dimensional array with w · h cells, each with a length of h and a width of w. Each hash function corresponds to one of w 1-dimensional arrays, each containing h cells. In standard applications of the Count-min sketch, the hash functions are used to update the numbers of the distinct cells in this two-dimensional data structure. For example, consider a one-dimensional data stream containing elements extracted from a comprehensive set of domain values. For example, in the embodiment of the present invention, this domain of values ​​corresponds to the distinct node identifiers in the social network. When a new element of the data stream is received, each of the w hash functions is applied to assign it a number in [0...h - 1]. The number of each of the sets of w cells is incremented by 1.To estimate the number of an element, we determine the set of w cells to which each of the w hash functions is assigned, and calculate the minimum value among all these cells. Let c. t The true value of the estimated number. It is stipulated that the estimated number must be at least equal to c. t This is because only non-negative numbers are considered, and collisions between hash cells can lead to overestimation. Furthermore, it turns out that an upper probabilistic limit for the estimation can be determined. It has been shown that for a data stream with T arrivals, the estimate can be determined with a probability of at least 1-δ and at most c. t +ε·T.

[0061] To use the count-min sketch to improve the node count estimation process, we maintain a sketch table for each cluster in the data. The sketch table is used to maintain the frequency counts of the nodes in the incoming data stream. Specifically, the sketch table for cluster j is populated with U j denoted. If desired, we can use the same set of w hash functions for the different clusters. The main condition is that the set of w hash functions should be independent of each other. For each incoming object S i The sketch table for the cluster is updated, with the cluster being assigned to the sketch table based on the similarity measure. The w different hash functions are applied to the (string representation of the identifier of the) nodes in R. i ∪ {q i} is applied, and 1 is added to the numbers of the corresponding cells. Thus, for the incoming object R, i each of the hash functions on the different |R i | = 1 node is applied, and the corresponding cells must be updated. This is equivalent to applying (|R i |+1)·w Hash function instantiations and corresponding cell updates.

[0062] Furthermore, the sketch-based structure can also be used to calculate the similarity value SimS(S). i ,C r ) to effectively estimate. We note that this similarity calculation must be performed for each cluster C, and its corresponding sketch table U, in order to determine the closest cluster to the incoming object based on the composite similarity measure. The denominator of SimS(S i ,C r ) can be accurately estimated, since the object S iis known and therefore also the value of |Ri∪{qi}| can be determined. The value of ∑t=1Srvrt can also be determined, since the value is equal to the sum of all values ​​in the sketch table cells in U, for each of the w hash functions. Thus, this value can be obtained exactly by summing the h cells for each specific hash function. On the other hand, the counter must be estimated approximately. It should be noted that the counter is essentially the sum of the estimated values ​​of the frequencies in R. i ∪ {q i} contained node is defined.

[0063] The frequency of each such node can be estimated. Specifically for each in R i ∪ {q iFor each node in the array, the corresponding cluster-specific frequency can be obtained by applying the hash function to the identifier of each node. It should be noted that the value of the corresponding hash cell will always be an overestimate due to collisions between the different node identifiers for the same hash cell. Therefore, the minimum of these values ​​over the w different hash functions will also be an overestimate, although it will be much closer and more stable due to the use of different hash functions. How do we sum these estimated frequency values ​​over the different nodes in R? i ∪ {q i}. This is essentially the estimate of the numerator. This estimate of the numerator can be used in conjunction with the exact knowledge of the different denominator values ​​to estimate SimS(S). i ,C r ) to create. Let EstSimS(Si ,C r ) a representation of the estimated similarity of S i with C r using the sketch-based approach. Then we can show the following result:

[0064] Lemma 1: If a sketch table of length h and width w is used, for a small value ∈>|Ri|+1 / h the estimated value of similarity EstSimS(S i ,C r ) with a probability of at least 1−(|Ri+1|h⋅∈)w bound to the following area: SimS(Si,Cr)≤EstSimS(Si,Cr)≤SimS(Si,Cr)+∈

[0065] Proof: As explained previously in Equation 2, the structural similarity is given by the following equation: SimS(Si,Cr)=∑t=1Srbt⋅vrt‖Ri∪{qi}‖⋅(∑t=1Srvrt)

[0066] It is noted that when using the sketch-based process, the numerator is approximated (overestimated), whereas the denominator can be known exactly. It is evident that due to the overestimation of the numerator, SimS(S i ,C r ) ≤ EstSimS(S i ,C r It remains to be shown that with a probability of at least 1−(|Ri+1|h⋅∈)w EstSimS(S i ,C r ≤ Sims(S i ,C r ) +ε is.

[0067] Let SimSN(S i ,C r ) and EstSimSN(S i ,C r ) the estimation of the numerator in the sketch-based approach. Since the denominator can be calculated exactly, we then have: EstSimS(Si,Cr)=EstSimSN(Si,Cr)|Ri|+1⋅(∑t=1Srvrt)

[0068] Furthermore, we have: SimS(Si,Cr)=SimSN(Si,Cr)|Ri|+1⋅(∑t=1Srvrt)

[0069] To prove the result in the lemma, the limits of the approximation in the numerator must therefore be proven. More precisely, it must be proven that the following occurs with a probability of at least 1−(|Ri+1|h⋅ε)w applies. EstSimSN(Si,Cr)≤SimSN(Si,Cr)+ε⋅|Ri|+1⋅(∑t=1Srvrt) ε⋅|Ri|+1⋅(∑t=1Srvrt) In equation 9, B is replaced. As with equation 9, the above limit of probability must be shown for the condition that: EstSimSN(Si,Cr)−SimSN(Si,Cr)≤B

[0070] The expected value of EstSimSN(S i ,C r )- SimSN(S i ,C r ) using any given hash function, reads (|Ri|+1)⋅(∑t=1Srvrt) / h. This result is obtained because the errors are calculated using |R i| + 1 needs to sum different frequency estimates and the expected number of collisions for each of these cell-based estimates (∑t=1Srvrt) / h is. Then we can use the Markov inequality to show that the probability that, when using a 1-hash function, the condition in equation 10 is met with a probability of at most (|Ri|+1)⋅(∑t=1Srvrt) / hB⋅h is violated. The probability of this can be calculated using w independent hash functions to at most ((|Ri|+1)⋅(∑t=1Srvrt)B⋅h)w generalize. Therefore, the condition in equation 10 holds with a probability of at least 1−((|Ri|+1)⋅(∑t=1Srvrt)B⋅h)w This is fulfilled. By replacing the value of B in the above equation, the desired result can be obtained.

[0071] The above result suggests that the value of similarity can be estimated quite accurately using the modest memory footprint of a Sketch table. For example, consider a tweet with |R| ≈ 100 and a similarity estimation limit of Consider this: If a sketch table with h = 200,000 and w = 5 (typical values) is used, it will require a memory of only 1 million cells, which is on the order of megabytes. The similarity estimate is accurate with a probability of at least 1-(1 / 20). 5 > 1-10 -6 within of the true value. Since these theoretical limits are quite loose, estimates in practice are much better.

[0072] The following sections explain the effectiveness and efficiency of the embodiments of the present invention. Two real datasets were used as examples to assess the effectiveness of the embodiments of the present invention.

[0073] The algorithm was tested on the following datasets. The first set is a social Twitter® stream, which is a stream of tweets obtained through a crawler search of the Twitter® social network. Each social object contains the network structure and content of a tweet. Specifically, each object contained the text of the tweet, as well as the sender and recipients of the tweet. The tweets were collected for the period from May 9, 2011, 05:00 GMT to May 10, 2011, 18:30 GMT. The tweets were extracted using the Twitter® Garden Stream API, available through the Twitter4j library (http: / / dev.finritter.com / docs / finritter-libraries). The stream contained a total of 1,628,779 tweets distributed across 47,351,520 nodes.The nodes contain the sender and recipients, either as direct mentions of senders or, in the case of sent messages, as their followers. When extracting followers, the experiment included only users who follow fewer than 1,000 other people and have fewer than 1,000 followers themselves. Kwak et al. report dramatic shifts in behavior among users with social graph values ​​outside this range; such users might be market participants, celebrities with professional publicists, or news media sources, and are therefore atypical of the general population. The experiment also removed short URLs, stop words, and emoticons from each of the received tweets. On average, each stream object contained approximately 84 nodes per tweet.

[0074] The second example set is an Enron email stream, which is an Enron email dataset converted into a stream using timestamps embedded in the emails. Each object contained the email text and the network structure corresponding to the sender and recipient(s). In this sense, the network structure of an email is very similar to a tweet with a single sender and multiple recipients. The Enron email stream contained a total of 517,432 emails. Emails without valid sender and recipient email addresses were removed. Calendar invitations, duplicate emails, and the email history at the end of each email were also filtered out. After the filtering process, a total of 349,911 emails remained, distributed across 29,083 individuals. On average, each email had 3.62 recipients.

[0075] Both datasets were tested for clustering and event detection methods, as well as for effectiveness. For efficiency, only the clustering method was used, as the largest proportion of time in event detection was devoted to the clustering process. To test effectiveness, the experiments used a number of class tags associated with objects in the social media stream. For the Twitter® stream, these class tags were hashtags associated with the tweets. The most frequent hashtags in the stream often correspond to events within the stream, such as the earthquake in Japan, and represent significant characteristics of the underlying objects that should be grouped into a cluster. These hashtags often represent the significant events in the stream. It should be noted that not every tweet may contain hashtags, and therefore the hashtags were only assigned to a subset of the tweets.In the case of the Enron email stream, the class identifiers were determined by the most frequently occurring tokens in the subject line. These tokens were as follows: meeting, agreement, gas, energy, power, report, update, request, conference, letter, deal, credit, california, trading, contract, project, presentation, houston, announcement

[0076] All emails containing one of the aforementioned tokens in the subject line were tagged with the corresponding class identifier. Thus, for both data streams, a subset of stream objects was tagged with a class identifier. Clearly, higher-quality clusters tend to group objects with similar identifiers into the same cluster. Therefore, the purity of dominant classes in each cluster was calculated. Each cluster was tagged with the highest occurrence, and the proportion of (tagged) cluster objects belonging to that identifier was calculated. This value was then weighted and averaged across the different clusters, with the weight of a cluster being proportional to the number of (tagged) objects in it.

[0077] The efficiency of the social stream clustering process was also tested. To test the efficiency, the number of social stream objects processed per unit of time was calculated, and the numbers for the different algorithms were presented.

[0078] The effectiveness of the event detection algorithm was tested. For cases involving the unsupervised algorithm, a case study is presented to illustrate the interesting events found by the technique. This provides an intuitive understanding of the technique's effectiveness. On the other hand, a supervised algorithm is required to generate the reference base. For this purpose, hashtags in the Twitter stream were used to generate a 0-1 bitstream corresponding to the time of the actual occurrence of the events. A hashtag corresponding to "#japan" was used to generate the events in the Twitter data stream. Specifically, at each timestamp, we observed the preceding window of length h and counted the number of occurrences of a particular hashtag within that window.If the number of occurrences of the hash tag was at least 3, a bit of 1 was generated at that timestamp to indicate that the event actually occurred.

[0079] The supervised event detection algorithm generates a continuous alarm level. It is possible to use a threshold t at this continuous alarm level to generate a 0-1 bitstream, which corresponds to the algorithmic prediction of the event's occurrence time. By using different thresholds at this alarm level, different balances between precision and recall can be achieved. Let S F (t) the set of timestamps at which an alarm is generated when a threshold of t is used at the actual alarm level. Let S GThe reference basis set of timestamps at which the event actually occurs. Then Precision(t) and Re call(t) can be calculated as follows: Precisiont(t)=|SF(t)∩SG|SF(t) Recall(t)=SF(t)∩SGSG

[0080] A graph relating accuracy to hit rate was plotted by varying the value of t and examining the balance between accuracy and hit rate using different thresholds for the generated alarm level. The higher the curve, the better the quality of the result.

[0081] Different algorithm settings were tested to determine the effect of content and network structure on accuracy and efficiency. By setting λ to the extreme values ​​of 0 and 1, the algorithm's performance can be tested. The performance overview can be tested for either the network-only algorithm or the text-only algorithm for the clustering process. A combination scheme was also tested in which the value of λ was set to 0.5. This scheme provides equal weighting for text and content in the clustering and event detection processes. For the combination scheme, we tested the algorithm with and without the Sketch technique to assess the accuracy and efficiency of the effects of using Sketch on the scheme.Therefore, variation also serves as a good starting point, since the use of pure text content is currently the only natural alternative for this problem.

[0082] Fig. Figure 7 illustrates the effectiveness results of the clustering algorithm with respect to cluster purity. The effectiveness of the approach was further tested with increasing cluster numbers. The results for the Twitter and Enron streams are shown in Fig. Figures 7(a) and 7(b) illustrate this. The Sketch table length h was set to 262,213, while the Sketch table width w was set to 2 for the social Twitter stream, and these values ​​were set to 16,369 and 2, respectively, for the Enron stream. In each case, the number of clusters was plotted on the x-axis and the cluster purity on the y-axis.

[0083] For the two data streams, a trend was observed regarding the relative performance of the different algorithms. In all cases, the algorithm using only text performed the worst. The trends between the purely network-based approach and the combined approach depended on the degree of granularity used for clustering. For both data streams, the network-based approach was found to be superior when using a small number of clusters. When a larger number of clusters were used, the combined methods performed better than the purely network-based approach. This is because the network location information is sufficient to effectively partition the clusters when the granularity is relatively coarse.In such cases, adding text does not improve the quality of the underlying clusters and can even be detrimental to the clustering quality. However, as the number of clusters increases, the combination approach demonstrates better performance because the cluster granularity is much higher, resulting in a greater need for attributes to differentiate between the various clusters. This is particularly evident in the case of the Enron data stream, where the gap between the combination approach and the purely network-based approach is quite significant. In all cases, the use of sketches was found to result in a loss of accuracy. However, this loss of accuracy is not very significant, especially considering that the sketch-based approach was considerably faster.It is also important to note that the performance of the purely text-based approach (which is the starting point) was worst in all scenarios.

[0084] These results also seem to suggest that the network information in the social stream provides much more powerful clustering information compared to the text information. This is not particularly surprising for sources such as the Twitter stream, where the text tends to be noisy and often contains non-standard acronyms or other text that is difficult to use meaningfully. However, it was found to be quite interesting and surprising that these trends also held true for a source like the Enron stream, where the text was relatively clean and usually quite informative regarding the underlying class labels.

[0085] Cluster purity was also visualized over the course of the stream. This provides a dynamic view of the performance of the clustering process as the stream progresses. Fig. 7(c) and Fig. Figure 7(d) shows the results for the Twitter and Enron streams. The number of clusters was fixed at 750. For the Twitter stream, the sketch table length h was fixed at 262,213, and for Enron, it was set to 16,369. The sketch table width w was fixed at 2 in both cases. The x-axis shows the stream's progression for each 1,000 processed objects, and the y-axis shows the cluster purity in the last 1-hour time window. The relative trends between the different methods are similar to those of previous observations, although the main observation is that cluster purity generally decreases as the stream progresses. This is because the number of class tags in the stream generally increases as new classes, tags, and events are encountered.As a result, cluster purity generally decreases as the stream progresses.

[0086] Finally, in Fig. 7(e) and Fig. 7(f) The sensitivity of the approach was tested with Sketch table length and width. The sensitivity results for the Twitter and Enron data streams decreased with increasing Sketch table length in Fig. Figures 7(e) and 7(f) respectively are shown. The sketch table length is plotted on the x-axis, while the cluster purity is plotted on the y-axis. The number of clusters was set to 500 and the sketch table width to 2 in this case. To provide a baseline for the relative effectiveness of training the sketch structure, the results are also shown in the same figure for the method that does not use the sketch. It is clear that the cluster purity increases with increasing sketch table length. This is because a greater sketch table length reduces the number of collisions in the sketch table and therefore improves the overall accuracy. Furthermore, as the sketch table length increases, the accuracy of the sketch table-based approach approaches the accuracy of the method that does not use a sketch table.In such cases, the collisions are sufficiently reduced to the point where there is a reduction in accuracy due to the use of the sketch table.

[0087] In Fig. 7(g) and Fig. In section 7(h), the sensitivity results with increasing Sketch table width are tested and shown for the Twitter and Enron data streams. The number of clusters was set to 500, and the Sketch table length was set to 262,213 for the Twitter stream and 16,369 for the Enron stream. Although the effectiveness results improved with increasing Sketch table width, the purity results were not as sensitive to Sketch table width compared to Sketch table length. This is because increasing the number of hash functions does not drastically reduce the number of collisions between the different elements, although it does provide additional stability.

[0088] Fig. Figure 8 shows the efficiency of the clustering approach with an increasing number of clusters. The parameter settings corresponded to those used in the effectiveness results. Fig. 8(a) and Fig. Figure 8(b) also shows the efficiency results for the Twitter and Enron streams. The x-axis represents the number of clusters, while the y-axis represents the number of stream objects processed in each hour. The results show that the network-based approach was slower than the text-based approach. This is because a text-based approach required processing a large number of individual nodes. However, the Sketch-based approach significantly improved the speed of both streams because the number of operations required for similarity calculations was reduced by the Sketch representation. Furthermore, the processing rate did not necessarily decrease with a decreasing number of clusters, as the larger number of clusters, resulting in more isolated clusters with smaller object counts in each cluster, demonstrated faster similarity calculations between the incoming objects and the underlying clusters.

[0089] Fig. 8(c) and Fig. Figure 8(d) shows the efficiency results as the current progresses. The current progress is plotted on the x-axis, and the processing rate is plotted on the y-axis. Fig. 8(c) and Fig. Figure 8(d) illustrates that the processing rate decreases for all the different methods as the stream progresses, since the clusters contain a larger number of objects. This increases the complexity of the similarity calculations, as the number of attributes in each cluster (in terms of the number of text words or nodes) also increases. However, the slowdown of the process levels off at a certain point when the number of attributes in each cluster stabilizes.

[0090] Fig. 8(e), Fig. 8(f), Fig. 8(g) and Fig. Figure 8(h) shows the sensitivity of the approach using the hash table length and width. The sensitivity results using the hash table length and width for the two data streams are shown in Fig. 8(e), Fig. 8(f), Fig. 8(g) and Fig. Figure 8(h) also illustrates this. The figures clearly show that the runtime is not very sensitive to the hash table length, and most variations are actually random. On the other hand, the hash table length affects the number of hash functions that need to be computed. Therefore, the runtime generally decreases with increasing hash table width. These results seem to suggest that a larger hash table width is not particularly beneficial, as it does not significantly improve purity but substantially affects efficiency. Therefore, additional memory should be used to increase the hash table length rather than its width.

[0091] Furthermore, we provide a case study of the unsupervised event detection problem. Both developmental and novel events were detected using this approach. Typically, such events were associated with a specific news story, country, language, or sentiment.

[0092] The first event concerns the nuclear crisis in Japan. Specifically, the relevant segment of the social stream corresponds to the instance in which the Japanese Prime Minister instructed the Chubu Electric Company to shut down the Hamaoka nuclear power plant. This event generated considerable discussion in the social stream, with the underlying actors debating the positive and negative aspects of this instruction. The corresponding section of the social stream contained structural nodes geographically centered on Japan and generally generated its own clusters based on the underlying network and content information. The frequent text content in these clusters included the following words: nuclear, hamaoka, plant, concerns, dilemma, gaswat, hairline, halt, shut, heavy, japan, neglect, operation (1.152)

[0093] An interesting aspect of the algorithm was its ability to recognize events whose content was in a foreign language. This is because the event recognition algorithm does not use any procedures specific to English, and its network structure is blind to the use of a particular language. For example, on May 9, 2011, the Minister of Finance in Indonesia issued an order to purchase 7% of the shares in the company "PT Newmont Nusa Tenggara." This triggered discussion threads on Twitter, which were captured due to their related content and network structure. The most frequent text content associated with the cluster most related to this event was as follows: keputusan, menkeu, beli, newmont, saham, wapres, didukung, challenge, minskade, metro, jak, kimiad, menos, minskade (0.9964)

[0094] In the previous example, the entire text content in this event consists of words in a foreign language.

[0095] As demonstrated above, the unsupervised approach to event detection was able to uncover interesting and novel events in the underlying social stream. Such interaction can be very useful in diagnosing important social sentiments related to real-world events.

[0096] Fig. Figure 9 illustrates the results of the supervised event detection procedure on the Twitter stream. To generate the supervised events, we set the horizon to 5 minutes and determined the durations in which events corresponding to the nuclear crisis in Japan occurred in the data stream. In each case, we set the number of clusters to 750 to detect the underlying events. Fig.Figure 9 illustrates in detail the conflict between accuracy and hit rate. Hit rate is plotted on the x-axis, while accuracy is shown on the y-axis. In each case, the event detection algorithm was implemented with different variations of the algorithm. It is clear that using only text provided less accurate event detection than all methods that used network structure in the event detection process. In particular, the method that used both the network structure and the text content yielded the most accurate results. Using sketches degraded the detection accuracy to some extent, but the approach was still more accurate than event detection using only text. It is also evident that the magnitudes of both accuracy and hit rate were quite high.In the case of the nuclear incident in Japan, for example, an accuracy of 0.525 with a hit rate of 0.62 was obtained when both network and text were used. On the other hand, the scheme that used either text or network achieved an accuracy of approximately 0.3 with a hit rate of approximately 0.6.

[0097] Thus, using a combination of network and text significantly improved the accuracy of the event detection algorithm. These results seem to suggest that combining network and text content for clustering and event detection can provide useful and accurate results in a wide variety of scenarios.

Claims

[1] Procedure (100,500) for detecting an event from a social stream, wherein the procedure (100,500) comprises the steps of one: Receiving a social stream from a social network, where the social stream contains at least one object; and where the object contains a text, sender information of the text and recipient information of the text (101,510); Assigning the object to a cluster based on a similarity value between the object and the cluster (102,520); Monitoring changes in at least one of the clusters (103,540); and Triggering an alarm when changes in at least one of the clusters exceed a first threshold (104,550), where at least one of the steps is performed using a computer unit, wherein the procedure further includes maintaining changes in the clusters as history data of the social stream (530), wherein in the maintenance step a sketch-supported technique is used, and wherein the sketch-supported technique is used to estimate a structural similarity value between the object and the clusters. [2] Method (100,500) according to claim 1, wherein the similarity value is determined by calculating a value selected from a group consisting of a structural similarity value, a content-based similarity value, a temporal similarity value and combinations thereof. [3] Method (100, 500) according to claim 1, wherein the assignment step (520) further comprises the step of: Assigning the object to an existing cluster if a similarity value between the object and the existing cluster is greater than a second threshold. [4] Method (100, 500) according to claim 1, wherein the assignment step (520) further comprises the steps of: Creating a new cluster with the object if the similarity value between the object and the existing cluster is less than a second threshold; and Replacing an outdated cluster with the new cluster. [5] Method (100,500) according to claim 3, wherein the second threshold is calculated from the expected value and standard deviation of the similarity value. [6] Method (100,500) according to claim 1, wherein the historical data are used for monitored event detection. [7] Method (100, 500) according to claim 6, wherein the step of triggering an alarm (550) further comprises the step of: Using an event signature and a horizon signature. [8] System (200) for detecting an event from a social stream, wherein the system (200) features: a receiving module (220) for receiving a social stream from a social network, where the social stream contains at least one object; and where the object contains a text, sender information of the text and recipient information of the text; a cluster module (230) for assigning the object based on a similarity value between the object and the cluster; a monitoring module (240) for monitoring changes in at least one of the clusters; and a trigger module (250) for triggering an alarm when changes in at least one of the clusters exceed a first threshold, wherein the system further includes a maintenance module for maintaining changes in the cluster as historical data of the social stream, wherein the maintenance module uses a sketch-based technique, and wherein the sketch-based technique is used to estimate a structural similarity value between the object and the clusters. [9] System (200) according to claim 8, wherein the similarity value is determined by calculating a value selected from a group consisting of a structural similarity value, a content-based similarity value, a temporal similarity value and combinations thereof. [10] System (200) according to claim 8, wherein the cluster module (230) further comprises: An existing cluster module for assigning the object to an existing cluster if a similarity value between the object and the existing cluster is greater than a second threshold. [11] System (200) according to claim 8, wherein the cluster module (230) further comprises: a new cluster module to create a new cluster with the object if the similarity value between the object and the existing cluster is less than a second threshold; and A replacement module to replace an outdated cluster with the new cluster. [12] System (200) according to claim 10, wherein the second threshold is calculated from the expected value and standard deviation of the similarity value. [13] System (200) according to claim 8, wherein the historical data are used for monitored event detection. [14] System (200) according to claim 13, wherein the trigger module (250) further comprises: A signature module for using an event signature and a horizon signature. [15] A computer-readable storage medium which constitutes a computer-readable program code with computer-readable instructions which, when executed, cause a computer to perform the steps of claim 1.

Images