Method, device, electronic equipment and storage medium for data stream anomaly detection

Abstract

The application provides a data stream anomaly detection method and device, electronic equipment and storage medium, which can be applied to traffic risk control anti-fraud of e-commerce, live broadcast, travel, video and other industry internet applications. The risk of the data stream can be identified by combining behavior and relationship chain, and then it is judged whether the data stream is black production behavior, so as to improve the accuracy and coverage of black production identification. The data stream anomaly detection method comprises: obtaining a data stream, the data stream at least comprising IP information and device information; performing feature vector processing on the data stream to obtain a target relationship chain feature vector and a target behavior feature vector; performing fusion processing on the target relationship chain feature vector and the target behavior feature vector to obtain a fusion feature vector; inputting the fusion feature vector into at least one pre-trained risk identification model to obtain a risk identification result of the data stream output by the at least one risk identification model.

Description

TECHNICAL FIELD

[0001] Embodiments of the present application relate to the technical field of network security, and more particularly, to a data flow anomaly detection method and device, electronic equipment and storage medium. BACKGROUND

[0002] Black production refers to a black industry that uses illegal means to obtain profits by taking the Internet as a medium and network technology as a means. In order to realize batch cheating and brushing, network black production usually initiates a large number of requests in a short time through the same Internet Protocol (IP) address, device number, mobile phone number, and user account, which seriously affects normal network activities. How to identify the risk of data flow to determine whether the data flow is a black production behavior is a problem to be solved. SUMMARY

[0003] The present application provides a data flow anomaly detection method, device, electronic equipment, chip and computer readable storage medium, which embeds the IP and device relationship chain features into vectors, so that the behavior features can be fused with the relationship chain features, thereby enabling the combination of behavior and relationship chain to identify the risk of data flow, and further determining whether the data flow is a black production behavior, improving the accuracy and coverage of black production identification.

[0004] Other characteristics and advantages of the present application will become apparent from the following detailed description, or will be learned by practice of the present application.

[0005] According to an aspect of the present application, a data flow anomaly detection method is provided, comprising:

[0006] Obtaining a data flow, the data flow comprising at least IP information and device information;

[0007] Performing feature vector processing on the data flow to obtain a target relationship chain feature vector and a target behavior feature vector; wherein the target relationship chain feature vector is an IP relationship chain feature vector, and the target behavior feature vector is an IP behavior feature vector; or the target relationship chain feature vector is a device relationship chain feature vector, and the target behavior feature vector is a device behavior feature vector;

[0008] Performing fusion processing on the target relationship chain feature vector and the target behavior feature vector to obtain a fusion feature vector;

[0009] Inputting the fusion feature vector into at least one pre-trained risk identification model to obtain a risk identification result of the data flow output by the at least one risk identification model.

[0010] According to an aspect of the present application, a data flow anomaly detection device is provided, comprising:

[0011] An acquisition module is configured to acquire a data stream, which comprises at least IP information and device information;

[0012] A determination model is configured to perform eigenvector processing on the data stream to obtain a target relationship chain eigenvector and a target behavior eigenvector; the target relationship chain eigenvector is an IP relationship chain eigenvector, and the target behavior eigenvector is an IP behavior eigenvector; or the target relationship chain eigenvector is a device relationship chain eigenvector, and the target behavior eigenvector is a device behavior eigenvector;

[0013] A fusion module is configured to perform fusion processing on the target relationship chain eigenvector and the target behavior eigenvector to obtain a fusion eigenvector;

[0014] An input module is configured to input the fusion eigenvector into at least one pre-trained risk identification model;

[0015] The acquisition module is further configured to acquire a risk identification result of the data stream output by the at least one risk identification model.

[0016] According to an aspect of the present application, an electronic device is provided, comprising a processor and a memory for storing a computer program, the processor being configured to invoke and run the computer program stored in the memory to perform the steps of the method for detecting data stream anomaly.

[0017] According to an aspect of the present application, a chip is provided, comprising a processor configured to invoke and run a computer program from a memory, so that the processor performs the steps of the method for detecting data stream anomaly.

[0018] According to an aspect of the present application, a computer readable storage medium is provided for storing a computer program, the computer program causing a computer to perform the steps of the method for detecting data stream anomaly.

[0019] Based on the above technical solution, the IP and device relationship chain features are vector embedded, so that the behavior features can be fused with the relationship chain features, thereby enabling the behavior and relationship chain to jointly identify the risk of the data stream, and further determining whether the data stream is a black production behavior, and improving the black production identification accuracy and coverage.

[0020] Other characteristics and advantages of the embodiments of the present application will become apparent from the following detailed description, or will be learned by practice of the present application.

[0021] It should be understood that the above general description and the following detailed description are only exemplary and do not limit the present application. BRIEF DESCRIPTION OF DRAWINGS

[0022] The accompanying drawings, which are incorporated herein and form a part of the specification, illustrate embodiments consistent with the present application and, together with the description, further serve to explain the principles of the application. It is to be understood that the drawings are designed solely for purposes of illustration to be used in conjunction with the description in

[0023] Figure 1 A diagram illustrating an application scenario of a method of data stream anomaly detection according to an embodiment of the present application is shown;

[0024] Figure 2 A diagram illustrating an architecture of data stream anomaly detection according to an embodiment of the present application is shown;

[0025] Figure 3 A flowchart illustrating a method of data stream anomaly detection according to an embodiment of the present application is shown;

[0026] Figure 4 A flowchart illustrating a method of determining a target relationship chain feature vector and a target behavior feature vector according to an embodiment of the present application is shown;

[0027] Figure 5 A flowchart illustrating a method of training an IP dimension risk identification model according to an embodiment of the present application is shown;

[0028] Figure 6 A structural diagram illustrating training of an IP dimension risk identification model according to an embodiment of the present application is shown;

[0029] Figure 7 A flowchart illustrating a method of determining a target relationship chain feature vector and a target behavior feature vector according to another embodiment of the present application is shown;

[0030] Figure 8 A flowchart illustrating a method of training a device dimension risk identification model according to an embodiment of the present application is shown;

[0031] Figure 9 A structural diagram illustrating training of a device dimension risk identification model according to an embodiment of the present application is shown;

[0032] Figure 10 A block diagram illustrating an apparatus of data stream anomaly detection according to an embodiment of the present application is shown;

[0033] Figure 11 A structural diagram of a computer system of an electronic device suitable for implementing embodiments of the present application is shown. DETAILED DESCRIPTION

[0034] Example implementations will now be described more fully with reference to the accompanying drawings. Example implementations can be implemented in any

[0035] Moreover, the described features, structures, or characteristics can be combined in any suitable manner in one or more example implementations. In the following description, numerous specific details are provided for a thorough understanding of example implementations. However, those of ordinary skill in the relevant art will recognize that one or more example implementations can be practiced without one or more of the specific details, or

[0036] Some of the block diagrams in the drawings are functional entities that do not necessarily correspond to physically or logically separate entities. These functional entities can be implemented in software, or in one or more hardware modules or integrated circuits, or in different networks, processor devices, or microcontroller devices.

[0037] With the research and progress of artificial intelligence technology, artificial intelligence technology is researched and applied in many fields, such as common smart home, smart wearable device, virtual assistant, smart sound box, smart marketing, unmanned vehicle, automatic driving, unmanned aerial vehicle, robot, smart medical treatment, smart customer service, and the like. It is believed that with the development of technology, artificial intelligence technology will be applied in more fields and play an increasingly important value. The present application can perform multi-speaker scene recognition and multi-speaker scene recognition network training based on artificial intelligence technology.

[0038] Artificial intelligence (AI) is to use a digital computer or a machine controlled by a digital computer to simulate, extend and expand human intelligence, perceive environment, acquire knowledge and use knowledge to obtain the best results. In other words, artificial intelligence is a comprehensive technology of computer science, which tries to understand the essence of intelligence and produce a new intelligent machine that can react in a similar way to human intelligence. Artificial intelligence is to study the design principles and implementation methods of various intelligent machines, so that machines have the functions of perception, reasoning and decision-making, that is, machines have learning ability.

[0039] Machine learning (ML) is a multidisciplinary field involving probability theory, statistics, approximation theory, convex analysis, and algorithm complexity theory, among others. It specifically studies how computers can simulate or implement human learning behavior to acquire new knowledge or skills and reorganize existing knowledge structures to continuously improve their performance. Machine learning is the core of artificial intelligence and the fundamental way to endow computers with intelligence; its applications span all areas of artificial intelligence. Machine learning and deep learning typically include techniques such as artificial neural networks (e.g., convolutional neural networks), belief networks, reinforcement learning, transfer learning, inductive learning, and instructional learning.

[0040] Artificial intelligence combined with cloud services can also create AI cloud services, commonly known as AI as a Service (AIaaS). This is currently a mainstream service model for AI platforms. Specifically, AIaaS platforms break down several common AI services and provide them as independent or packaged services in the cloud. This service model is similar to opening an AI-themed marketplace: all developers can access and use one or more AI services provided by the platform through Application Programming Interfaces (APIs). Some experienced developers can also use the AI ​​frameworks and infrastructure provided by the platform to deploy and maintain their own dedicated cloud AI services.

[0041] Figure 1 This is an application scenario diagram of the data stream anomaly detection method provided in one embodiment, such as... Figure 1 As shown, this application scenario includes terminal 110 and server 120.

[0042] In some implementations, at least one risk identification model can be trained via server 120. This risk identification model may include some or all of a supervised learning model, a clustering model, and anomaly detection models. After obtaining the trained risk identification model, server 120 can deploy it in a risk identification application. Terminal 110 can install this application. When terminal 110 obtains the data stream, the user can issue a risk identification command through corresponding operations. Terminal 110 can receive the risk identification command, treat the acquired data stream as data to be processed, perform risk identification, and obtain the risk identification result of the data stream.

[0043] The aforementioned risk identification application can be a cybersecurity application, which may also have functions such as data logging, audio and video playback, translation, and data query.

[0044] In other implementations, at least one risk identification model can be trained via terminal 110. This risk identification model may include some or all of a supervised learning model, a clustering model, and anomaly detection models. After terminal 110 acquires the data stream, the user can issue a risk identification command through corresponding operations. Terminal 110 can receive the risk identification command, treat the acquired data stream as data to be processed, perform risk identification, and obtain the risk identification result for the data stream.

[0045] It is understood that the above application scenario is merely an example and does not constitute a limitation on the data stream anomaly detection method provided in the embodiments of this application. For example, at least one trained risk identification model can be stored in server 120. Server 120 can receive data streams sent by terminal 110, perform risk identification on the data streams, obtain the risk identification results of the data streams, and then return them to terminal 110.

[0046] Server 120 can be a standalone physical server, a server cluster or distributed system composed of multiple physical servers, or a cloud server providing basic cloud computing services such as cloud services, cloud databases, cloud computing, cloud functions, cloud storage, network services, cloud communication, middleware services, domain name services, security services, CDN, and big data and artificial intelligence platforms. Terminal 110 can be a smartphone, tablet, laptop, desktop computer, smart speaker, in-vehicle computer, or smartwatch, but is not limited to these. Terminal 110 and server 120 can be directly or indirectly connected via wired or wireless communication, and this application does not limit this connection.

[0047] Figure 2 This illustration schematically shows an architecture diagram of data stream anomaly detection provided in one embodiment of this application, such as... Figure 2 As shown, the input consists of data streams generated during online activities such as registration, login, coupon redemption, group buying, discounting, product ordering, video-on-demand, and commenting. These data streams typically contain user account information, IP address information, and device information. After undergoing account risk identification, IP risk identification, device risk identification, and behavioral risk identification respectively, model fusion and policy fusion are used to finally output a risk level and risk label, thereby determining whether the data stream originates from malicious activities. This application is specifically applicable to two scenarios: IP risk identification and device risk identification.

[0048] To facilitate a better understanding of the embodiments of this application, the current methods for identifying black market activities will be explained.

[0049] Currently, risk control teams typically identify the IPs and devices used by cybercriminals from three aspects:

[0050] 1. Maintain a blacklist of IPs, devices, and accounts: These blacklists may originate from IPs and devices exhibiting abnormal behavior within the business itself, or from the IPs' own profile tags such as data center IPs and open port information; since the IPs and devices controlled by black market operators are not fixed, the blacklists have time limits.

[0051] 2. Identify black market activities through behavioral analysis: In order to achieve mass fraud and traffic boosting, black market operators usually launch a large number of requests in a short period of time. Therefore, risk control teams can identify black market fraud by the number of requests from the same IP, device number, mobile phone number, or user number in a short period of time.

[0052] 3. Identifying cybercrime through relationship chains: In order to bypass the risk control of business partners, cybercriminals have developed corresponding tools to modify, steal, and forge IP addresses, device numbers, and phone numbers. When cybercriminals frequently switch IP addresses and device numbers, a relationship chain between IP addresses or device numbers will appear. The risk control team can use the relationship chain to uncover cybercriminal groups.

[0053] Current black market detection technologies have the following drawbacks:

[0054] 1. Cybercriminals use techniques such as instant IP address dialing, device ID modification, and SIM card pool account farming to quickly change IP addresses and device IDs, which greatly shortens the validity period of the business's blacklist.

[0055] 2. Cybercriminal IPs may share the same IP range with legitimate users for a period of time, making it difficult to combat them directly using IP blacklists;

[0056] 3. Because black market operators frequently switch IP addresses and device IDs, it is difficult to achieve effective coverage by simply digging into their behavior or relationship chains.

[0057] Based on the above-mentioned technical problems, this application proposes a data stream anomaly detection scheme. By embedding the IP and device relationship chain features into vectors, behavioral features can be fused with relationship chain features. This allows for risk identification of the data stream by combining behavior and relationship chain, thereby determining whether the data stream is related to black market activities. This improves the accuracy of black market identification and achieves higher coverage.

[0058] The specific implementation process of the embodiments of this application will be described in detail below.

[0059] Figure 3 A schematic flowchart of a data stream anomaly detection method 200 according to an embodiment of this application is shown. This data stream anomaly detection method 200 can be executed by a device with computing power, such as the aforementioned terminal 110 or server 120. (Refer to...) Figure 3 As shown, the data stream anomaly detection method 200 may include at least steps S210 to S240, which are described in detail below:

[0060] In S210, a data stream is acquired, which includes at least IP information and device information.

[0061] Specifically, the aforementioned data streams can be generated during online activities such as registration, login, coupon redemption, group buying, helping others cut prices, placing product orders, video on demand, and commenting.

[0062] The aforementioned IP information may be, for example, IP addresses. Specifically, IP information may include, for example, multiple IP addresses or multiple associated IP addresses, or multiple IP addresses or multiple associated IP addresses within a certain period of time; IP information may also include, for example, multiple IP addresses requesting the same service, or multiple IP addresses requesting the same service within a certain period of time.

[0063] The aforementioned device information may be, for example, a device number or a device identifier. Specifically, device information may include, for example, multiple device numbers or device identifiers, or multiple associated device numbers or device identifiers; device information may also include, for example, multiple device numbers or device identifiers within a certain period of time, or multiple associated device numbers or device identifiers within a certain period of time; device information may also include, for example, multiple device numbers or device identifiers requesting the same service, or multiple device numbers or device identifiers requesting the same service within a certain period of time.

[0064] In some embodiments, the data stream described above may also include, but is not limited to, account information and content information. Account information may be, for example, a user account, or information such as a mobile phone number or ID card number associated with the user account. Content information may be, for example, business-related information such as registration, login, coupon redemption, group buying, etc.

[0065] In some embodiments, the data stream described above may be request stream data.

[0066] In some embodiments, abnormal data in the original data stream can be filtered to obtain the aforementioned data stream. Abnormal data may include, for example, data with invalid IP addresses, invalid device numbers, invalid accounts, invalid content, etc. Of course, other types of invalid data are also possible, and this application is not limited to these.

[0067] In some embodiments, the raw data stream can be filtered to obtain the aforementioned data stream. For example, the filtering can be based on possible methods of black market fraud and data manipulation.

[0068] In S220, feature vector processing is performed on the data stream to obtain target relationship chain feature vector and target behavior feature vector; wherein, the target relationship chain feature vector is an IP relationship chain feature vector and the target behavior feature vector is an IP behavior feature vector; or, the target relationship chain feature vector is a device relationship chain feature vector and the target behavior feature vector is a device behavior feature vector.

[0069] In S230, the target relationship chain feature vector and the target behavior feature vector are fused to obtain a fused feature vector.

[0070] In some embodiments, the target relationship chain feature vector and the target behavior feature vector can be concatenated or connected to obtain a fused feature vector.

[0071] Assume the feature vector of the target relation chain is (a1, a2, ..., a n The feature vector of the target behavior chain is (b1, b2, ..., b). m The feature vector fusion process can specifically include: concatenating the target relation chain feature vector with the target behavior chain feature vector to obtain (a1, a2, ..., a...). n ,b1,b2,…,b m The concatenated feature vectors are then regularized to the same range using a regularization algorithm, resulting in (x1, x2, ..., x...). m+n ).

[0072] That is, the feature vector of the target relation chain (a1, a2, ..., a...) n ) and target behavior feature vector (b1,b2,…,b m The fusion yields a fused feature vector of (x1, x2, ..., x). m+n ).

[0073] It should be noted that the target relationship chain feature vector and the target behavior feature vector can also be fused in other ways, and this application does not limit this.

[0074] Specifically, using fused feature vectors as input to the risk identification model can greatly enrich the dimensions of features, combine relationship chain features and behavioral features to jointly identify risks in data streams, and then determine whether the data stream is a black market activity, thereby improving the accuracy and coverage of black market identification.

[0075] In S240, the fused feature vector is input into at least one pre-trained risk identification model to obtain the risk identification result of the data stream output by at least one risk identification model.

[0076] In some embodiments, the at least one risk identification model described above includes a supervised learning model, a clustering model, and an anomaly detection model.

[0077] The labeled data used by the supervised learning model is obtained from a risk database accumulated in the business. The risk database may come from IPs and devices exhibiting abnormal behavior within the business itself, or it may come from the IP's own profile tags, such as data center IPs and open port information.

[0078] For risk identification at the IP level, the risk database could be, for example, an IP blacklist / profile database formed from IPs that exhibit abnormal behavior within the business itself.

[0079] For risk identification at the device level, the risk database could be, for example, a blacklist / profile database of devices that exhibit abnormal behavior within the business itself.

[0080] It should be noted that for supervised learning models, the training data in supervised learning has both features and labels. Through training, the machine finds the relationship between features and labels, so that when it encounters data that only has features but no labels, it can determine the label for it.

[0081] In other words, for supervised learning models, the relationship between independent variables (fused feature vector, X) and dependent variables (label data, Y) is trained to achieve the goal of predicting Y from X.

[0082] It's important to note that clustering models are the process of dividing samples into multiple classes composed of similar objects. During classification, a predefined classification criterion is unnecessary; cluster analysis can automatically classify data based on the sample data. Objects within the same class exhibit high similarity, while objects in different classes show significant dissimilarity.

[0083] It's important to note that anomaly detection models aim to identify objects that differ from the majority of objects—in other words, outliers. Generally, data is defined as having a "normal" model, and anomalies are considered deviations from this normal model.

[0084] In some embodiments, the supervised learning model may be a random forest model, the clustering model may be a Gaussian mixture model, and the anomaly detection model may be an isolation forest model; however, this application is not limited to these.

[0085] In some embodiments, a user instruction is obtained, which is used to indicate a first threshold; and if the risk identification result output by one or more risk identification models in at least one risk identification model is greater than the first threshold, the data stream is determined to be a data stream with risk.

[0086] In some embodiments, if the risk identification result output by one or more risk identification models in at least one risk identification model is greater than a first threshold, the data stream is determined to be a risky data stream. That is, the data stream may be related to illicit activities.

[0087] Specifically, the first threshold can be set manually based on needs; for example, the first threshold can be set to 0.5.

[0088] In some embodiments, the training process of the at least one risk identification model includes:

[0089] Obtain a training sample set. Each training sample in the training sample set includes a fused feature vector obtained by fusing the target relationship chain feature vector and the target behavior feature vector, and its corresponding risk identification label.

[0090] Train at least one risk identification model based on the training sample set.

[0091] Specifically, the training samples include target relationship chain feature vectors that are IP relationship chain feature vectors and target behavior feature vectors that are IP behavior feature vectors; or, the training samples include target relationship chain feature vectors that are device relationship chain feature vectors and target behavior feature vectors that are device behavior feature vectors.

[0092] Specifically, feature vector processing can be performed on the data stream to obtain the target relationship chain feature vector and the target behavior feature vector, which are included in the training samples.

[0093] The number of training samples in the training sample set can be set according to needs. For example, if the model needs to be trained 40 times, with 5000 training samples used each time, then the training sample set can include 5000 training samples, and these 5000 training samples will be used for each model training session. The fused feature vector and its corresponding risk identification label refer to the following: the fused feature vector is the feature vector that needs to be used for risk identification, and the risk identification label is the expected risk identification result obtained after using the risk identification model to identify the fused feature vector.

[0094] The goal of model training is to obtain better model parameters to improve the effectiveness of risk identification. During training, the fused feature vector is input into the risk identification model. Then, the model parameters are adjusted based on the difference between the risk identification result output by the risk identification model and the risk identification label corresponding to the fused feature vector. This process continues until the model convergence condition is met, and the trained risk identification model is finally obtained.

[0095] In some embodiments, the training performance of at least one risk identification model can be verified based on a risk database.

[0096] Specifically, at least one verification sample is selected from the risk database. The verification sample includes a fused feature vector obtained by fusing IP relationship chain feature vector and IP behavior feature vector, and its corresponding risk identification label. When the risk identification model (such as supervised learning model, clustering model and anomaly detection model) is fitted to the training sample, the verification sample is input into the risk identification model. Based on the risk identification result output by the risk identification model and the risk identification label corresponding to the verification sample, the optimal parameter is selected from multiple sets of convergence parameters obtained by the risk identification model during the training process.

[0097] The proposed implementation can be applied to traffic risk control and fraud prevention in internet applications such as e-commerce, live streaming, travel, and video. It can improve the accuracy of identifying online black market activities, enhance traffic quality, optimize marketing costs, and improve the effectiveness of traffic marketing at various traffic stages, including application registration, login, coupon redemption, product ordering, video-on-demand, and comments. For example, in typical e-commerce coupon group-buying activities, it can be used to identify black market groups using multiple accounts to participate in group buying and discounting, as well as bulk registration of new accounts to claim new user coupons.

[0098] The proposed implementation can solve the problem of accuracy and coverage in identifying cybercriminals when they control a large number of IPs, devices, and accounts.

[0099] Therefore, in this embodiment of the application, vector embedding is performed on the IP and device relationship chain features so that the behavioral features can be fused with the relationship chain features. This allows for risk identification of the data stream by combining the behavior and the relationship chain, thereby determining whether the data stream is a black market activity and improving the accuracy and coverage of black market identification.

[0100] In addition, the IP / device relationship chain has been quantified, which allows the model to be trained through the relationship chain. This avoids the problem of manually setting thresholds when using community discovery algorithms to mine relationship chains, and at the same time improves the accuracy of risk identification of data streams.

[0101] Figure 4 A schematic flowchart of a data stream anomaly detection method according to an embodiment of this application is shown. Figure 4 The method described herein details the specific process of obtaining the target relationship chain feature vector and the target behavior feature vector in S220 above. Specifically, the target relationship chain feature vector is the IP relationship chain feature vector, and the target behavior feature vector is the IP behavior feature vector, as shown below. Figure 4 As shown, it may include S2201 to S2204 as follows. Detailed descriptions are as follows:

[0102] In S2201, the data stream is processed by IP-based graphing to obtain an IP relationship graph. The nodes of the IP relationship graph are IP addresses. An edge is created between two IP addresses that have at least one common device, account, or content. The weight of the edge is the number of devices, accounts, or content that the two IP addresses have at least one common device, account, or content.

[0103] In other words, in the IP relationship graph, an edge is created between two IP addresses that have at least one common device, account, or content. The weight of the edge is the number of devices, accounts, or content that the two IP addresses have at least one common content.

[0104] In some embodiments, before constructing the IP relationship graph, it is necessary to extract and deduplicate the IP information and first information (which may specifically include device information, account information, and content information) of each request stream in the data stream to form IP-first information relationship pairs.

[0105] In S2201, it is assumed that an edge is created between two IP addresses that share a common device, with the weight of the edge being the number of devices that both IP addresses share. Specifically, in S2201, the IP and device information of each request stream in the data flow is extracted and deduplicated to form IP-device relationship pairs. An IP relationship graph is constructed using these relationship pairs, with IP addresses as nodes. If two IP addresses share a common device in a relationship pair, an edge is created between the two nodes, with the weight of the edge being the number of devices that both IP addresses share.

[0106] In S2202, a random walk method is used to perform multiple samplings on nodes in the IP relationship graph to obtain IP address sequence samples.

[0107] In some embodiments, after randomly arranging the nodes on the IP relationship graph, a walking path is generated with the randomly selected i-th IP address as the root node. Each step of the walk moves randomly from the current node to an adjacent node. The walk ends when the next adjacent node cannot be found or the path length reaches the upper limit. After performing multiple walk samplings, an IP address sequence sample is obtained.

[0108] The path length limit can be determined based on the IP relationship graph, or it can be set manually according to actual needs.

[0109] Specifically, an embedding vector of a node can be generated using a graph neural network (GNN) node representation algorithm to obtain IP address sequence samples.

[0110] The following example uses the classic deepwalk algorithm (of course, other walk algorithms can also be used, and this application is not limited to this). Nodes are sampled on the IP relationship graph through random walks. Starting from a specific node, each step of the walk randomly moves from the current node to an adjacent node, and this process is repeated to generate a random walk path. After generating a certain number of random walk paths, the node sampling task is complete.

[0111] Specifically, the IP relationship graph can be represented as G = (V, E), where V is the set of nodes and E is the set of edges. After randomly arranging the nodes, traversing all nodes, and selecting a random node v each time... i , with v i Generate a walk path for the root node n is the path taken. The maximum number of nodes on the path is determined, and the process ends when no next adjacent node can be found or the path length reaches the upper limit. The above steps can be repeated γ times to obtain the IP address sequence sample.

[0112] In S2203, the embedding vector of each IP address in the IP address sequence sample is calculated to obtain the IP relationship chain feature vector.

[0113] Specifically, a sequence algorithm can be used to calculate the embedding vector for each IP address in the IP address sequence sample. The sequence algorithm can be, for example, word2vec, node2vec, BERT, etc.

[0114] The following example uses the classic word2vec algorithm. For a random walk path (v1, v2, ..., v...),... n For any intermediate node v i The model predicts node v based on its surrounding nodes. i Thus, node v is obtained. i The embedding vector of each node is obtained. This process is repeated to obtain the embedding vectors of all nodes.

[0115] In S2204, the data stream is aggregated through the IP dimension, and the behavioral statistics under the IP sliding window are calculated to obtain the IP behavioral feature vector.

[0116] In some embodiments, the size of the IP sliding window can be set manually for a time period according to actual needs, such as 1 minute, 10 minutes, 30 minutes, 1 hour, 2 hours, 1 day, etc. This application does not limit this.

[0117] Specifically, the data stream is aggregated through the IP dimension, and behavioral statistics under the IP sliding window are calculated to obtain the behavioral characteristics of the IP.

[0118] In some embodiments, the behavioral statistics under the IP sliding window include at least one of the following:

[0119] Number of requests, number of associated devices, number of associated accounts, number of associated phone numbers.

[0120] In some embodiments, the data stream is aggregated by the IP dimension, and at least one of the following is calculated under the IP sliding window: the number of requests, the number of associated devices, the number of associated accounts, and the number of associated mobile phone numbers; and feature vector processing is performed on at least one of the following: the number of requests, the number of associated devices, the number of associated accounts, and the number of associated mobile phone numbers under the IP sliding window, to obtain the above-mentioned IP behavior feature vector.

[0121] Figure 5 A schematic flowchart illustrating a training method for an IP-dimensional risk identification model according to an embodiment of this application is shown. Figure 6 A schematic structural diagram of the training of an IP-dimensional risk identification model according to an embodiment of this application is shown. Figure 5 and Figure 6 Correspondingly, specifically, such as Figure 5 As shown, it may include S1-1 to S1-11 as follows. Details are as follows:

[0122] S1-1. Obtain the raw data stream.

[0123] Specifically, the aforementioned raw data streams can be generated in online activities such as registration, login, coupon redemption, group buying, helping to cut prices, placing product orders, video on demand, and commenting.

[0124] S1-2. Perform abnormal data filtering on the original data stream to obtain a new data stream, which includes at least IP information and device information.

[0125] Specifically, abnormal data in the original data stream can be filtered to obtain the aforementioned data stream. Abnormal data can include, for example, data with invalid IP addresses, invalid device numbers, invalid accounts, invalid content, etc. Of course, other types of invalid data are also possible; this application does not limit this.

[0126] The aforementioned IP information may be, for example, IP addresses. Specifically, IP information may include, for example, multiple IP addresses or multiple associated IP addresses, or multiple IP addresses or multiple associated IP addresses within a certain period of time; IP information may also include, for example, multiple IP addresses requesting the same service, or multiple IP addresses requesting the same service within a certain period of time.

[0127] The aforementioned device information may be, for example, a device number or a device identifier. Specifically, device information may include, for example, multiple device numbers or device identifiers, or multiple associated device numbers or device identifiers; device information may also include, for example, multiple device numbers or device identifiers within a certain period of time, or multiple associated device numbers or device identifiers within a certain period of time; device information may also include, for example, multiple device numbers or device identifiers requesting the same service, or multiple device numbers or device identifiers requesting the same service within a certain period of time.

[0128] In some embodiments, the data stream described above may also include, but is not limited to, account information and content information. Account information may be, for example, a user account, or information such as a mobile phone number or ID card number associated with the user account. Content information may be, for example, business-related information such as registration, login, coupon redemption, group buying, etc.

[0129] S1-3. Extract IP and device deduplication pairs from the data stream.

[0130] Specifically, the IP and device information of each request stream in the data stream is extracted and deduplicated to form IP-device relationship pairs. This ensures that each IP and each device is unique.

[0131] S1-4. Perform IP-based graph processing on the deduplication of IP and device relationships to obtain an IP relationship graph. In the IP relationship graph, the nodes are IP addresses. An edge is created between two IP addresses that have a common corresponding device. The weight of the edge is the number of devices that the two IP addresses have a common corresponding device.

[0132] S1-5. Perform graph node walking sampling on the IP relationship graph to obtain IP address sequence samples.

[0133] Specifically, after randomly arranging the nodes on the IP relationship graph, a walking path is generated with the randomly selected i-th IP address as the root node. Each step of the walk moves randomly from the current node to an adjacent node. The walk ends when the next adjacent node cannot be found or the path length reaches the upper limit. After performing multiple walk samplings, an IP address sequence sample is obtained.

[0134] For example, an IP relationship graph can be represented as G = (V, E), where V is the set of nodes and E is the set of edges. After randomly arranging the nodes, traversing all nodes, and selecting a random node v each time... i , with v i Generate a walk path for the root node n is the path taken. The maximum number of nodes on the path is determined, and the process ends when no next adjacent node can be found or the path length reaches the upper limit. The above steps can be repeated γ times to obtain the IP address sequence sample.

[0135] S1-6. Calculate the embedding vector of each IP address in the IP address sequence sample to obtain the IP relationship chain feature vector.

[0136] Specifically, a sequence algorithm can be used to calculate the embedding vector for each IP address in the IP address sequence sample. The sequence algorithm can be, for example, word2vec, node2vec, BERT, etc.

[0137] The following example uses the classic word2vec algorithm. For a random walk path (v1, v2, ..., v...),... n For any intermediate node v i The model predicts node v based on its surrounding nodes. i Thus, node v is obtained. i The embedding vector of each node is obtained. This process is repeated to obtain the embedding vectors of all nodes.

[0138] S1-7. Perform IP-level data aggregation processing on the data stream.

[0139] Specifically, the data contained in the data stream is aggregated at the IP dimension in order to calculate behavioral statistics under the IP sliding window.

[0140] S1-8. Calculate the behavioral statistics of the IP under the sliding window to obtain the IP behavior feature vector.

[0141] In some embodiments, the size of the IP sliding window can be set manually for a time period according to actual needs, such as 1 minute, 10 minutes, 30 minutes, 1 hour, 2 hours, 1 day, etc. This application does not limit this.

[0142] Specifically, the data stream is aggregated through the IP dimension, and behavioral statistics under the IP sliding window are calculated to obtain the behavioral characteristics of the IP.

[0143] In some embodiments, the behavioral statistics under the IP sliding window include at least one of the following:

[0144] Number of requests, number of associated devices, number of associated accounts, number of associated phone numbers.

[0145] S1-9. The IP relationship chain feature vector and the IP behavior feature vector are fused using a regularization algorithm to obtain the fused feature vector.

[0146] In some embodiments, the IP relationship chain feature vector and the IP behavior feature vector can be concatenated or connected to obtain a fused feature vector.

[0147] Assume the feature vector of the IP relationship chain is (a1, a2, ..., a nThe feature vector of the IP behavior chain is (b1, b2, ..., b m The feature vector fusion process can specifically include: concatenating the IP relationship chain feature vector with the IP behavior chain feature vector to obtain (a1, a2, ..., a...). n ,b1,b2,…,b m The concatenated feature vectors are then regularized to the same range using a regularization algorithm, resulting in (x1, x2, ..., x...). m+n ).

[0148] That is, the IP relationship chain feature vector (a1, a2, ..., a n ) and IP behavior feature vectors (b1, b2, ..., b m The fusion yields a fused feature vector of (x1, x2, ..., x). m+n ).

[0149] It should be noted that IP relationship chain feature vectors and IP behavior feature vectors can also be fused in other ways, and this application does not limit this.

[0150] S1-10. Input the fused feature vectors into the supervised learning model, clustering model, and anomaly detection model respectively for model training.

[0151] That is, at least one of the risk identification models mentioned above includes a supervised learning model, a clustering model, and an anomaly detection model. Specifically, the supervised learning model uses a random forest model, the clustering model uses a Gaussian mixture model, and the anomaly detection model uses an isolated forest model.

[0152] The labeled data used by the supervised learning model is obtained from a risk database accumulated in the business. The risk database may come from IPs exhibiting abnormal behavior within the business itself, or it may come from the IP's own profile tags, such as data center IPs and open port information.

[0153] Specifically, a risk database could be an IP blacklist / profile database formed from IPs that exhibit abnormal behavior within the business itself.

[0154] If any of the supervised learning model, clustering model, or anomaly detection model reaches a first threshold, the data stream is judged to be risky, meaning it may be related to illicit activities.

[0155] S1-11. Verify the effectiveness of training the supervised learning model, clustering model, and anomaly detection model based on the risk database accumulated in the business.

[0156] In some embodiments, a risk database accumulated in the business is used to validate the model's effectiveness.

[0157] Specifically, at least one verification sample is selected from the risk database. The verification sample includes a fused feature vector obtained by fusing IP relationship chain feature vector and IP behavior feature vector, and its corresponding risk identification label. When the risk identification model (such as supervised learning model, clustering model and anomaly detection model) is fitted to the training sample, the verification sample is input into the risk identification model. Based on the risk identification result output by the risk identification model and the risk identification label corresponding to the verification sample, the optimal parameter is selected from multiple sets of convergence parameters obtained by the risk identification model during the training process.

[0158] Figure 7 A schematic flowchart of a data stream anomaly detection method according to an embodiment of this application is shown. Figure 7 The method described herein details the specific process of obtaining the target relationship chain feature vector and the target behavior feature vector in S220 above. The target relationship chain feature vector is the device relationship chain feature vector, and the target behavior feature vector is the device behavior feature vector, as shown below. Figure 7 As shown, it may include S2205 to S2208 as follows. Detailed descriptions are as follows:

[0159] In S2205, feature vector processing is performed on the data stream to obtain a device relationship graph. The nodes of the device relationship graph are device identifiers. An edge is created between two devices that have at least one of the following common IP addresses, accounts, and content. The weight of the edge is the number of at least one of the following common IP addresses, accounts, and content between the two devices.

[0160] That is, in the device relationship graph, an edge is created between two devices that have at least one common IP address, account, or content. The weight of the edge is the number of at least one common IP address, account, or content between the two devices.

[0161] In some embodiments, before constructing the device relationship graph, it is necessary to extract and deduplicate the device information and second information (which may specifically include IP information, account information and content information) of each request stream in the data stream to form a device-second information relationship pair.

[0162] In S2205, it is assumed that an edge is created between two devices that share a common IP address, with the weight of the edge being the number of common IP addresses between the two devices. Specifically, in S2205, the IP and device information of each request stream in the data flow is extracted and deduplicated to form device-IP relationship pairs. A device relationship graph is constructed using these relationship pairs, with nodes representing device numbers (identifiers). If two devices share a common IP address in a relationship pair, an edge is created between the two devices, with the weight of the edge being the number of common IP addresses between the two devices.

[0163] In S2206, a random walk method is used to perform multiple samplings on nodes in the device relationship graph to obtain device sequence samples.

[0164] In some embodiments, after randomly arranging the nodes on the device relationship graph, a walking path is generated with the randomly selected j-th device as the root node. Each step of the walk moves randomly from the current node to an adjacent node. The walk ends when the next adjacent node cannot be found or the path length reaches the upper limit. After performing multiple walk samplings, a device sequence sample is obtained.

[0165] Specifically, a graph neural network (GNN) node representation algorithm can be used to generate the node embedding vectors, thereby obtaining device sequence samples.

[0166] The following example uses the classic deepwalk algorithm (of course, other walk algorithms can also be used, and this application is not limited to this one). Nodes are sampled on the device relationship graph through random walks. Starting from a specific node, each step of the walk randomly moves from the current node to an adjacent node, and this process is repeated continuously to generate a random walk path. After generating a certain number of random walk paths, the node sampling task is complete.

[0167] Specifically, the device relationship graph can be represented as G = (V, E), where V is the set of nodes and E is the set of edges. After randomly arranging the nodes, traversing all nodes, and selecting a random node v each time... i , with v i Generate a walk path for the root node n is the path taken. The maximum number of nodes on the path is determined, and the process ends when no next adjacent node can be found or the path length reaches the upper limit. The above steps can be repeated γ times to obtain the device sequence sample.

[0168] In S2207, the embedding vector of each device in the device address sequence sample is calculated to obtain the device relationship chain feature vector.

[0169] Specifically, a sequence algorithm can be used to calculate the embedding vector for each device in the device sequence sample. Sequence algorithms can include word2vec, node2vec, BERT, etc.

[0170] The following example uses the classic word2vec algorithm. For a random walk path (v1, v2, ..., v...),... n For any intermediate node v i The model predicts node v based on its surrounding nodes. i Thus, node v is obtained. iThe embedding vector of each node is obtained. This process is repeated to obtain the embedding vectors of all nodes.

[0171] In S2208, the data stream is aggregated through the device dimension, and the behavioral statistics under the device sliding window are calculated to obtain the device behavior feature vector.

[0172] In some embodiments, the size of the device sliding window can be set manually for a time period according to actual needs, such as 1 minute, 10 minutes, 30 minutes, 1 hour, 2 hours, 1 day, etc., and this application does not limit it.

[0173] Specifically, the data stream is aggregated at the device level, and the behavioral statistics of the device are calculated under the sliding window (last day / last hour / last 10 minutes) to obtain the behavioral characteristics of the device.

[0174] In some embodiments, the behavioral statistics under the device sliding window include at least one of the following:

[0175] Number of requests, number of associated IP addresses, number of associated accounts, and number of associated phone numbers.

[0176] In some embodiments, the data stream is aggregated at the device dimension to calculate at least one of the following: the number of requests, the number of associated IP addresses, the number of associated accounts, and the number of associated mobile phone numbers under the device sliding window; and feature vector processing is performed on at least one of the following: the number of requests, the number of associated IP addresses, the number of associated accounts, and the number of associated mobile phone numbers under the device sliding window to obtain the aforementioned device behavior feature vector.

[0177] Figure 8 A schematic flowchart illustrating a method for training a device-level risk identification model according to an embodiment of this application is shown. Figure 9 A schematic structural diagram of the training of a device-dimensional risk identification model according to an embodiment of this application is shown. Figure 8 and Figure 9 Correspondingly, specifically, such as Figure 8 As shown, it can include S2-1 to S2-11 as follows. Details are as follows:

[0178] S2-1. Obtain the raw data stream.

[0179] Specifically, the aforementioned raw data streams can be generated in online activities such as registration, login, coupon redemption, group buying, helping to cut prices, placing product orders, video on demand, and commenting.

[0180] S2-2. Perform abnormal data filtering on the original data stream to obtain a new data stream, which includes at least IP information and device information.

[0181] Specifically, abnormal data in the original data stream can be filtered to obtain the aforementioned data stream. Abnormal data can include, for example, data with invalid IP addresses, invalid device numbers, invalid accounts, invalid content, etc. Of course, other types of invalid data are also possible; this application does not limit this.

[0182] The aforementioned IP information may be, for example, IP addresses. Specifically, IP information may include, for example, multiple IP addresses or multiple associated IP addresses, or multiple IP addresses or multiple associated IP addresses within a certain period of time; IP information may also include, for example, multiple IP addresses requesting the same service, or multiple IP addresses requesting the same service within a certain period of time.

[0183] The aforementioned device information may be, for example, a device number or a device identifier. Specifically, device information may include, for example, multiple device numbers or device identifiers, or multiple associated device numbers or device identifiers; device information may also include, for example, multiple device numbers or device identifiers within a certain period of time, or multiple associated device numbers or device identifiers within a certain period of time; device information may also include, for example, multiple device numbers or device identifiers requesting the same service, or multiple device numbers or device identifiers requesting the same service within a certain period of time.

[0184] In some embodiments, the data stream described above may also include, but is not limited to, account information and content information. Account information may be, for example, a user account, or information such as a mobile phone number or ID card number associated with the user account. Content information may be, for example, business-related information such as registration, login, coupon redemption, group buying, etc.

[0185] S2-3. Extract IP and device deduplication pairs from the data stream.

[0186] Specifically, the IP and device information of each request stream in the data stream is extracted and deduplicated to form IP-device relationship pairs. This ensures that each IP and each device is unique.

[0187] S2-4. Perform device-based graph processing on the deduplication of IP and device relationships to obtain a device relationship graph. In the device relationship graph, the nodes are device numbers. An edge is created between two devices with a common IP address, and the weight of the edge is the number of common IP addresses between the two devices.

[0188] S2-5. Perform node walking sampling on the equipment relationship graph to obtain equipment sequence samples.

[0189] S2-6. Calculate the embedding vector of each device in the device sequence sample to obtain the device relationship chain feature vector.

[0190] S2-7. Perform device-level data aggregation processing on the data stream.

[0191] Specifically, the data contained in the data stream is aggregated at the device level to calculate behavioral statistics under the device's sliding window.

[0192] S2-8. Calculate the behavior statistics of the device under the sliding window to obtain the device behavior feature vector.

[0193] In some embodiments, the size of the device sliding window can be set manually for a time period according to actual needs, such as 1 minute, 10 minutes, 30 minutes, 1 hour, 2 hours, 1 day, etc., and this application does not limit it.

[0194] Specifically, the data stream is aggregated at the device level, and the behavioral statistics under the device's sliding window are calculated to obtain the device's behavioral characteristics.

[0195] In some embodiments, the behavioral statistics under the device sliding window include at least one of the following:

[0196] Number of requests, number of associated IP addresses, number of associated accounts, and number of associated phone numbers.

[0197] S2-9. The device relationship chain feature vector and the device behavior feature vector are fused using a regularization algorithm to obtain a fused feature vector.

[0198] In some embodiments, the device relationship chain feature vector and the device behavior feature vector can be concatenated or connected to obtain a fused feature vector.

[0199] Assume the feature vector of the device relationship chain is (a1, a2, ..., a n The feature vector of the device behavior chain is (b1, b2, ..., b). m The feature vector fusion process can specifically include: concatenating the device relationship chain feature vector with the device behavior chain feature vector to obtain (a1, a2, ..., a...). n ,b1,b2,…,b m The concatenated feature vectors are then regularized to the same range using a regularization algorithm, resulting in (x1, x2, ..., x...). m+n ).

[0200] That is, the feature vector of the device relationship chain (a1, a2, ..., a...) n ) and device behavior feature vectors (b1, b2, ..., b m The fusion yields a fused feature vector of (x1, x2, ..., x). m+n ).

[0201] It should be noted that the device relationship chain feature vector and the device behavior feature vector can also be fused in other ways, and this application does not limit this.

[0202] S2-10. Input the fused feature vectors into the supervised learning model, clustering model, and anomaly detection model respectively for model training.

[0203] That is, at least one of the risk identification models mentioned above includes a supervised learning model, a clustering model, and an anomaly detection model. Specifically, the supervised learning model uses a random forest model, the clustering model uses a Gaussian mixture model, and the anomaly detection model uses an isolated forest model.

[0204] The labeled data used by the supervised learning model is obtained from a risk database accumulated in the business. The risk database may come from devices exhibiting abnormal behavior within the business itself, or it may come from the device's own profile tags, such as data center equipment and open port information.

[0205] Specifically, a risk database could be a blacklist / profile database of devices that exhibit abnormal behavior within the business itself.

[0206] If any of the supervised learning model, clustering model, or anomaly detection model reaches a first threshold, the data stream is judged to be risky, meaning it may be related to illicit activities.

[0207] S2-11. Verify the effectiveness of training the supervised learning model, clustering model, and anomaly detection model based on the equipment risk database accumulated in the business.

[0208] In some embodiments, a risk database accumulated in the business is used to validate the model's effectiveness.

[0209] Specifically, at least one verification sample is selected from the risk database. The verification sample includes a fused feature vector obtained by fusing the device relationship chain feature vector and the device behavior feature vector, and its corresponding risk identification label. When the risk identification model (such as a supervised learning model, clustering model, and anomaly detection model) is fitted to the training sample, the verification sample is input into the risk identification model. Based on the risk identification result output by the risk identification model and the risk identification label corresponding to the verification sample, the optimal parameter is selected from multiple sets of convergence parameters obtained by the risk identification model during the training process.

[0210] The above text combined Figures 3 to 9 The method embodiments of this application are described in detail below, in conjunction with... Figure 10 The present application describes the device embodiments in detail. It should be understood that the device embodiments correspond to the method embodiments, and similar descriptions can be referred to the method embodiments.

[0211] Figure 10 A block diagram schematically illustrates an apparatus for detecting data stream anomalies according to an embodiment of this application. This apparatus for detecting data stream anomalies can be a software unit or a hardware unit, or a combination of both, integrated into a computer device. Figure 10 As shown, the data stream anomaly detection device 300 provided in this application embodiment may specifically include:

[0212] The acquisition module 310 is used to acquire a data stream, which includes at least IP information and device information.

[0213] Model 320 is defined to perform feature vector processing on the data stream to obtain target relationship chain feature vector and target behavior feature vector; wherein, the target relationship chain feature vector is an IP relationship chain feature vector and the target behavior feature vector is an IP behavior feature vector; or, the target relationship chain feature vector is a device relationship chain feature vector and the target behavior feature vector is a device behavior feature vector.

[0214] The fusion module 330 is used to fuse the target relationship chain feature vector and the target behavior feature vector to obtain a fused feature vector;

[0215] The input module 340 is used to input the fused feature vector into at least one pre-trained risk identification model and obtain the risk identification result of the data stream output by at least one risk identification model.

[0216] In one embodiment, if the target relationship chain feature vector is an IP relationship chain feature vector and the target behavior feature vector is an IP behavior feature vector; model 320 is specifically used for:

[0217] The data stream is processed using IP-based graphing to obtain an IP relationship graph. The nodes of the IP relationship graph are IP addresses. An edge is created between two IP addresses that have at least one common device, account, or content. The weight of the edge is the number of devices, accounts, or content that the two IP addresses have at least one common device, account, or content.

[0218] A random walk approach is used to perform multiple samplings on nodes in the IP relationship graph to obtain IP address sequence samples; the embedding vector of each IP address in the IP address sequence samples is calculated to obtain the IP relationship chain feature vector; and

[0219] The data stream is aggregated by IP dimension, and the behavioral statistics under the IP sliding window are calculated to obtain the IP behavioral feature vector.

[0220] In one embodiment, determining model 320 is specifically used for:

[0221] After randomly arranging the nodes on the IP relationship graph, a traversal path is generated with the i-th randomly selected IP address as the root node. Each traversal step moves randomly from the current node to an adjacent node. The traversal ends when the next adjacent node cannot be found or the path length reaches the upper limit.

[0222] After performing multiple walk samplings, IP address sequence samples are obtained.

[0223] In one embodiment, determining model 320 is specifically used for:

[0224] Aggregate data streams by IP dimension, and calculate at least one of the following under the IP sliding window: number of requests, number of associated devices, number of associated accounts, and number of associated phone numbers;

[0225] The IP behavior feature vector is obtained by performing feature vector processing on at least one of the following: the number of requests, the number of associated devices, the number of associated accounts, and the number of associated mobile phone numbers under the IP sliding window.

[0226] In one embodiment, if the target relationship chain feature vector is a device relationship chain feature vector, and the target behavior feature vector is a device behavior feature vector; model 320 is specifically used for:

[0227] The data stream is processed using device-based graphing to obtain a device relationship graph. The nodes in the device relationship graph are device identifiers. An edge is created between two devices that share at least one of the following common IP addresses, accounts, or content: the weight of the edge is the number of times the two devices share at least one of the following common IP addresses, accounts, or content.

[0228] A random walk approach is used to perform multiple samplings on nodes in the device relationship graph to obtain device sequence samples; the embedding vector of each device in the device address sequence samples is calculated to obtain the device relationship chain feature vector; and

[0229] The data stream is aggregated by device dimension, and the behavioral statistics under the device sliding window are calculated to obtain the device behavior feature vector.

[0230] In one embodiment, determining model 320 is specifically used for:

[0231] After randomly arranging the nodes on the device relationship graph, a walking path is generated with the randomly selected j-th device as the root node. Each step of the walking path moves randomly from the current node to the adjacent node. The walking path ends when the next adjacent node cannot be found or the path length reaches the upper limit.

[0232] After performing multiple walk samplings, device sequence samples were obtained.

[0233] In one embodiment, determining model 320 is specifically used for:

[0234] Aggregate the data stream by device dimension and calculate at least one of the following: number of requests, number of associated IP addresses, number of associated accounts, and number of associated phone numbers under the device sliding window;

[0235] The device behavior feature vector is obtained by performing feature vector processing on at least one of the following: the number of requests, the number of associated IP addresses, the number of associated accounts, and the number of associated mobile phone numbers under the device's sliding window.

[0236] In one embodiment, at least one risk identification model includes a supervised learning model, a clustering model, and an anomaly detection model, wherein the label data used by the supervised learning model is obtained from a risk database accumulated in the business.

[0237] In one embodiment, the training process of at least one risk identification model includes:

[0238] Obtain a training sample set. Each training sample in the training sample set includes a fused feature vector obtained by fusing the target relationship chain feature vector and the target behavior feature vector, and its corresponding risk identification label.

[0239] Train at least one risk identification model based on the training sample set.

[0240] In one embodiment, the acquisition module 310 is used to acquire a user instruction, which is used to indicate a first threshold.

[0241] Model 320 is further used to determine that a data stream is a risky flow data when the risk identification result output by one or more risk identification models in at least one risk identification model is greater than a first threshold.

[0242] In one embodiment, obtaining model 310 is specifically used for:

[0243] Obtain the raw data stream;

[0244] Abnormal data in the original data stream is filtered to obtain the new data stream.

[0245] The specific implementation of each module in the data stream anomaly detection device provided in this application embodiment can refer to the content of the above-described data stream anomaly detection method, and will not be repeated here.

[0246] The modules in the aforementioned data flow anomaly detection device can be implemented entirely or partially through software, hardware, or a combination thereof. These modules can be embedded in or independent of the processor in a computer device, or stored in the memory of a computer device in software form, so that the processor can invoke and execute the operations corresponding to each module.

[0247] Figure 11 A schematic diagram of the structure of a computer system implementing an electronic device according to an embodiment of this application is shown. It should be noted that... Figure 11 The computer system 400 of the electronic device shown is merely an example and should not impose any limitations on the functionality and scope of use of the embodiments of this application.

[0248] like Figure 11 As shown, the computer system 400 includes a Central Processing Unit (CPU) 401, which can perform various appropriate actions and processes based on programs stored in Read-Only Memory (ROM) 402 or programs loaded from storage section 408 into Random Access Memory (RAM) 403. The RAM 403 also stores various programs and data required for system operation. The CPU 401, ROM 402, and RAM 403 are interconnected via a bus 404. An Input / Output (I / O) interface 405 is also connected to the bus 404.

[0249] The following components are connected to I / O interface 405: an input section 406 including a keyboard, mouse, etc.; an output section 407 including a cathode ray tube (CRT), liquid crystal display (LCD), etc., and speakers, etc.; a storage section 408 including a hard disk, etc.; and a communication section 409 including a network interface card such as a local area network (LAN) card, modem, etc. The communication section 409 performs communication processing via a network such as the Internet. A drive 410 is also connected to I / O interface 405 as needed. A removable medium 411, such as a disk, optical disk, magneto-optical disk, semiconductor memory, etc., is installed on drive 410 as needed so that computer programs read from it can be installed into storage section 408 as needed.

[0250] Specifically, according to embodiments of this application, the processes described in the flowcharts above can be implemented as computer software programs. For example, embodiments of this application include a computer program product comprising a computer program carried on a computer-readable medium, the computer program containing program code for performing the methods shown in the flowcharts above. In such embodiments, the computer program can be downloaded and installed from a network via communication section 409, and / or installed from removable medium 411. When the computer program is executed by central processing unit (CPU) 401, it performs the various functions defined in the apparatus of this application.

[0251] In one embodiment, an electronic device is also provided, comprising:

[0252] Processor; and

[0253] Memory for storing the executable instructions of the processor;

[0254] The processor is configured to execute the steps in the above method embodiments by executing executable instructions.

[0255] In one embodiment, a computer device is also provided, including a memory and a processor, wherein the memory stores a computer program, and the processor executes the computer program to implement the steps in the above method embodiments.

[0256] In one embodiment, a computer-readable storage medium is provided storing a computer program that, when executed by a processor, implements the steps in the above method embodiments.

[0257] It should be noted that the computer-readable storage medium described in this application can be, for example,—but not limited to—an electrical, magnetic, optical, electromagnetic, infrared, or semiconductor system, apparatus, or device, or any combination thereof. More specific examples of computer-readable storage media may include, but are not limited to: electrical connections having one or more wires, portable computer disks, hard disks, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM), flash memory, optical fiber, portable compact disc read-only memory (CD-ROM), optical storage devices, disk storage devices, or any suitable combination thereof. In this application, the computer-readable storage medium can be any tangible medium containing or storing a program that can be used by or in conjunction with an instruction execution system, apparatus, or device. In this application, the computer-readable signal medium may include a data signal propagated in baseband or as part of a carrier wave, carrying computer-readable program code. Such propagated data signals may take various forms, including but not limited to electromagnetic signals, optical signals, or any suitable combination thereof. Computer-readable signal media can also be any computer-readable storage medium other than a computer-readable storage medium, which can send, propagate, or transmit a program for use by or in connection with an instruction execution system, apparatus, or device. The program code contained on the computer-readable medium can be transmitted using any suitable medium, including but not limited to: wireless, wire, optical fiber, radio frequency, etc., or any suitable combination thereof.

[0258] This embodiment is only used to illustrate this application. The selection of software and hardware platform architecture, development environment, development language, message acquisition source, etc. in this embodiment can be varied. Based on the technical solution of this application, any improvement or equivalent transformation made to a certain part according to the principle of this application should not be excluded from the protection scope of this application.

[0259] It should be noted that the terminology used in the embodiments of this application and the appended claims is for the purpose of describing specific embodiments only, and is not intended to limit the embodiments of this application.

[0260] Those skilled in the art will recognize that the units and algorithm steps of the various examples described in conjunction with the embodiments disclosed herein can be implemented in electronic hardware, or a combination of computer software and electronic hardware. Whether these functions are implemented in hardware or software depends on the specific application and design constraints of the technical solution. Those skilled in the art can use different methods to implement the described functions for each specific application, but such implementations should not be considered beyond the scope of the embodiments of this application.

[0261] If implemented as a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of this application embodiment, essentially, or the part that contributes to the prior art, or a part of the technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute all or part of the steps of the method described in the embodiments of this application. The aforementioned storage medium includes various media capable of storing program code, such as USB flash drives, portable hard drives, read-only memory, random access memory, magnetic disks, or optical disks.

[0262] Those skilled in the art will clearly understand that, for the sake of convenience and brevity, the specific working processes of the devices, apparatuses, and units described above can be referred to the corresponding processes in the foregoing method embodiments, and will not be repeated here.

[0263] In the several embodiments provided in this application, it should be understood that the disclosed electronic devices, apparatuses and methods can be implemented in other ways.

[0264] For example, the division of units, modules, or components in the device embodiments described above is merely a logical functional division. In actual implementation, there may be other division methods. For example, multiple units, modules, or components may be combined or integrated into another system, or some units, modules, or components may be ignored or not executed.

[0265] For example, the units / modules / components described above as separate / display components may or may not be physically separate; that is, they may be located in one place or distributed across multiple network units. Some or all of the units / modules / components can be selected to achieve the objectives of the embodiments of this application, depending on actual needs.

[0266] Finally, it should be noted that the mutual coupling or direct coupling or communication connection shown or discussed above can be an indirect coupling or communication connection through some interface, device or unit, and can be electrical, mechanical or other forms.

[0267] The above description is merely a specific implementation of the embodiments of this application, but the protection scope of the embodiments of this application is not limited thereto. Any variations or substitutions that can be easily conceived by those skilled in the art within the technical scope disclosed in the embodiments of this application should be included within the protection scope of the embodiments of this application. Therefore, the protection scope of the embodiments of this application should be determined by the protection scope of the claims.

Claims

1. A method for detecting anomalies in a data stream, characterized in that, include: Acquire a data stream, wherein the data stream includes at least IP information and device information; The data stream is processed by feature vector processing to obtain a target relationship chain feature vector and a target behavior feature vector; wherein, the target relationship chain feature vector is an IP relationship chain feature vector and the target behavior feature vector is an IP behavior feature vector; or, the target relationship chain feature vector is a device relationship chain feature vector and the target behavior feature vector is a device behavior feature vector. The target relationship chain feature vector and the target behavior feature vector are fused to obtain a fused feature vector. The fused feature vector is input into at least one pre-trained risk identification model to obtain the risk identification result of the data stream output by the at least one risk identification model; If the target relationship chain feature vector is an IP relationship chain feature vector, and the target behavior feature vector is an IP behavior feature vector; the step of performing feature vector processing on the data stream to obtain the target relationship chain feature vector and the target behavior feature vector includes: performing IP-based graph processing on the data stream to obtain an IP relationship graph, wherein the nodes of the IP relationship graph are IP addresses, and an edge is created between two IP addresses that have at least one common corresponding device, account, or content, with the weight of the edge being the number of at least one common corresponding device, account, or content between the two IP addresses; performing multiple samplings on the nodes in the IP relationship graph using a random walk approach to obtain IP address sequence samples; calculating the embedding vector of each IP address in the IP address sequence samples to obtain the IP relationship chain feature vector; and aggregating the data stream through the IP dimension to calculate behavioral statistics under the IP sliding window to obtain the IP behavior feature vector; If the target relationship chain feature vector is a device relationship chain feature vector, and the target behavior feature vector is a device behavior feature vector; the step of performing feature vector processing on the data stream to obtain the target relationship chain feature vector and the target behavior feature vector includes: performing device-based graph processing on the data stream to obtain a device relationship graph, wherein the nodes of the device relationship graph are device identifiers, and an edge is created between two devices that have at least one common IP address, account, or content, with the weight of the edge being the number of at least one common IP address, account, or content between the two devices; performing multiple samplings on the nodes in the device relationship graph using a random walk approach to obtain device sequence samples; calculating the embedding vector of each device in the device sequence samples to obtain the device relationship chain feature vector; and aggregating the data stream through the device dimension to calculate the behavioral statistics under the device sliding window to obtain the device behavior feature vector.

2. The method according to claim 1, characterized in that, The method of performing multiple samplings on nodes in the IP relationship graph using a random walk approach to obtain IP address sequence samples includes: After randomly arranging the nodes on the IP relationship graph, the first node is randomly selected. i Each IP address is used to generate a traversal path for the root node. Each step of the traversal moves randomly from the current node to an adjacent node. The traversal ends when no next adjacent node can be found or the path length reaches the upper limit. After performing multiple walk samplings, the IP address sequence sample is obtained.

3. The method according to claim 1, characterized in that, The step of aggregating the data stream by IP dimension and calculating behavioral statistics under the IP sliding window to obtain the IP behavioral feature vector includes: The data stream is aggregated by IP dimension, and at least one of the following is calculated under the IP sliding window: number of requests, number of associated devices, number of associated accounts, and number of associated phone numbers. The IP behavior feature vector is obtained by performing feature vector processing on at least one of the following: the number of requests, the number of associated devices, the number of associated accounts, and the number of associated mobile phone numbers under the IP sliding window.

4. The method according to claim 1, characterized in that, The method of performing multiple samplings on nodes in the device relationship graph using a random walk approach to obtain device sequence samples includes: After randomly arranging the nodes on the device relationship graph, the first node is randomly selected. j Each device generates a walking path for the root node. Each step of the walk involves randomly moving from the current node to an adjacent node. The walk ends when the next adjacent node cannot be found or the path length reaches the upper limit. After performing multiple walk-sampling operations, the device sequence samples are obtained.

5. The method according to claim 1, characterized in that, The step of aggregating the data stream by device dimension, calculating behavioral statistics under the device sliding window, and obtaining the device behavior feature vector includes: The data stream is aggregated at the device level, and at least one of the following is calculated: the number of requests under the device sliding window, the number of associated IP addresses, the number of associated accounts, and the number of associated mobile phone numbers. The device behavior feature vector is obtained by performing feature vector processing on at least one of the following: the number of requests, the number of associated IP addresses, the number of associated accounts, and the number of associated mobile phone numbers under the device's sliding window.

6. The method according to claim 1, characterized in that, The at least one risk identification model includes a supervised learning model, a clustering model, and an anomaly detection model, wherein the label data used by the supervised learning model is obtained from a risk database accumulated in the business.

7. The method according to claim 1, characterized in that, The training process of the at least one risk identification model includes: Obtain a training sample set, wherein each training sample in the training sample set includes a fused feature vector obtained by fusing the target relationship chain feature vector and the target behavior feature vector, and its corresponding risk identification label; The at least one risk identification model is trained based on the training sample set.

8. The method according to claim 1, characterized in that, The method further includes: Obtain a user instruction, the user instruction being used to indicate a first threshold; If the risk identification result output by one or more of the at least one risk identification model is greater than the first threshold, the data stream is determined to be a data stream with risk.

9. The method according to claim 1, characterized in that, The acquisition of the data stream includes: Obtain the raw data stream; Abnormal data in the original data stream is filtered to obtain the data stream.

10. A device for detecting data stream anomalies, characterized in that, include: The acquisition module is used to acquire data streams, which include at least IP information and device information; A model is determined for performing feature vector processing on the data stream to obtain a target relationship chain feature vector and a target behavior feature vector; wherein, the target relationship chain feature vector is an IP relationship chain feature vector and the target behavior feature vector is an IP behavior feature vector; or, the target relationship chain feature vector is a device relationship chain feature vector and the target behavior feature vector is a device behavior feature vector. The fusion module is used to fuse the target relationship chain feature vector and the target behavior feature vector to obtain a fused feature vector; An input module is used to input the fused feature vector into at least one pre-trained risk identification model; The acquisition module is also used to acquire the risk identification result of the data stream output by the at least one risk identification model; If the target relationship chain feature vector is an IP relationship chain feature vector, and the target behavior feature vector is an IP behavior feature vector; the determining model is used to: perform IP-based graph processing on the data stream to obtain an IP relationship graph, wherein the nodes of the IP relationship graph are IP addresses, and an edge is created between two IP addresses that have at least one common corresponding device, account, or content, with the weight of the edge being the number of at least one common corresponding device, account, or content between the two IP addresses; perform multiple samplings on the nodes in the IP relationship graph using a random walk approach to obtain IP address sequence samples; calculate the embedding vector of each IP address in the IP address sequence samples to obtain the IP relationship chain feature vector; and aggregate the data stream through the IP dimension to calculate the behavioral statistics under the IP sliding window to obtain the IP behavior feature vector; If the target relationship chain feature vector is a device relationship chain feature vector, and the target behavior feature vector is a device behavior feature vector; the determining model is used to: perform device-based graph processing on the data stream to obtain a device relationship graph, wherein the nodes of the device relationship graph are device identifiers, and an edge is created between two devices that have at least one common IP address, account, or content, with the weight of the edge being the number of at least one common IP address, account, or content between the two devices; perform multiple samplings on the nodes in the device relationship graph using a random walk approach to obtain device sequence samples; calculate the embedding vector of each device in the device sequence samples to obtain the device relationship chain feature vector; and aggregate the data stream through the device dimension, calculate the behavioral statistics under the device sliding window, and obtain the device behavior feature vector.

11. An electronic device, characterized in that, include: processor; as well as Memory for storing the executable instructions of the processor; The processor is configured to perform the method of any one of claims 1 to 9 by executing the executable instructions.

12. A computer-readable storage medium storing a computer program, characterized in that, When the computer program is executed by a processor, it implements the steps of the method according to any one of claims 1 to 9.

Images