A data processing method, device, apparatus, and storage medium

Abstract

The application discloses a data processing method and device, equipment and storage medium, relates to the technical field of deep packet inspection and data analysis, and comprises the following steps: defining a streaming label rule and a streaming index calculation rule based on an initialized anti-fraud identification model; filtering original data collected by using a deep packet inspection technology, and performing standardization processing on the filtered data; performing real-time index calculation on the standardized data according to the streaming index calculation rule; forwarding the standardized data to a risk calculation and analysis node, so that the risk calculation and analysis node performs real-time risk calculation by using index results corresponding to the standardized data to obtain risk information corresponding to user behaviors existing gambling and fraud risks, and outputs the risk information. The technical scheme of the application realizes second-level early warning of the user behaviors existing gambling and fraud risks, and improves the timeliness and accuracy of gambling and fraud data processing.

Description

Technical Field

[0001] This invention relates to the fields of deep packet inspection technology and data analysis technology, and particularly to a data processing method, apparatus, device and storage medium. Background Technology

[0002] Currently, DPI (Deep Packet Inspection) adds application protocol identification, packet content inspection, and deep decoding to traditional IP (Internet Protocol) packet inspection technology, resulting in massive amounts of network data, i.e., DPI data. At the operator level, DPI data is mainly generated by user internet browsing behavior and business app usage. Based on its traffic generation method, it can be divided into traffic generated when communicating based on HTTP and 4G technologies, traffic generated when communicating based on HTTPS and 4G technologies, and traffic generated when communicating based on HTTP and 5G technologies. Traffic generated during communication using HTTPS and 5G technologies can be analyzed using massive amounts of DPI data to capture user browsing patterns and usage preferences. An anti-gambling and anti-fraud information database can be built based on abnormal access records. However, the massive nature of DPI data leads to large computational loads and long processing times. The anti-gambling and anti-fraud information database built using batch data processing combined with list collision analysis requires T+1 level alerts, resulting in low alert timeliness, poor list quality, and limited accuracy. This increases costs in terms of manpower and time, failing to meet the high-timeliness requirements of gambling and fraud data processing and alerting scenarios. Summary of the Invention

[0003] In view of this, the purpose of this invention is to provide a data processing method, apparatus, device, and storage medium that can improve the timeliness and accuracy of data processing related to gambling and fraud. The specific solution is as follows:

[0004] In a first aspect, the present invention discloses a data processing method, comprising:

[0005] Define streaming label rules and streaming index calculation rules based on the initialized anti-fraud identification model;

[0006] The raw data collected using deep packet inspection technology is filtered according to a preset whitelist rule to obtain filtered data, and the filtered data is then standardized to obtain corresponding standardized data.

[0007] The standardized data is calculated in real time according to the streaming index calculation rules to obtain the corresponding index results.

[0008] The standardized data is forwarded to the risk calculation and analysis node, so that the risk calculation and analysis node can use the indicator results corresponding to the standardized data to perform real-time risk calculation to obtain risk information corresponding to user behaviors that have gambling or fraud risks, and output the risk information.

[0009] Optionally, the step of filtering the raw data collected using deep packet inspection technology according to a preset whitelist rule to obtain filtered data includes:

[0010] A Bloom filter is constructed using a pre-defined list of normal URLs, and the Bloom filter is used to filter the raw data collected using deep packet inspection technology to obtain filtered data.

[0011] Optionally, the step of using the Bloom filter to filter the raw data acquired using deep packet inspection technology to obtain filtered data includes:

[0012] The Bloom filter is used to determine whether the domain name carried in the raw data collected by deep packet inspection technology meets the preset filtering conditions; wherein, the raw data is data containing user number field, access time field, access domain name field, access Uniform Resource Identifier field, server IP address field and user access base station information field;

[0013] If the domain name carried by the original data meets the preset filtering conditions, then the original data is filtered.

[0014] If the domain name carried by the original data does not meet the preset filtering conditions, the original data is retained to obtain the filtered data.

[0015] Optionally, the standardization process for the filtered data to obtain corresponding standardized data includes:

[0016] The filtered data is categorized based on the type of the domain name it carries, resulting in data of the bank type, data of the APK type, and other suspicious types.

[0017] According to the streaming labeling rules, the other suspicious data types are supplemented with labels to obtain the tagged other suspicious data types.

[0018] The data of the bank type, the data of the APK type, and the data of other suspicious types after being tagged are merged to obtain the merged data;

[0019] Standardized data is obtained by performing field completion processing on the merged data using the target information; wherein, the target information includes the IP address location information of the server, the user access base station information, and the current timestamp information.

[0020] Optionally, before defining the streaming label rules and streaming indicator calculation rules based on the initialized anti-fraud identification model, the method further includes:

[0021] Offline data is processed for clue mining and feature extraction to obtain corresponding extracted features; the offline data includes user information, base station information, case information, and access records;

[0022] The extracted features are used to initialize the pre-constructed anti-fraud identification model to obtain the initialized anti-fraud identification model.

[0023] Optionally, the step of performing clue mining and feature extraction processing on offline data to obtain the corresponding extracted features includes:

[0024] Natural language processing technology is used to extract the case occurrence period and related links from the case information, and the corresponding IP address, domain name and Uniform Resource Identifier information are determined based on the case occurrence period and related links;

[0025] The IP address, the domain name, and the Uniform Resource Identifier information are used to summarize rules to construct corresponding early warning indicators;

[0026] The user's access path and access frequency are determined from the access records, and corresponding risk characteristics are extracted based on the access path and access frequency.

[0027] The user information and the base station information are used to perform rule summarization to determine user characteristics and base station location characteristics; the user characteristics include user age, user gender, and the device used by the user.

[0028] Accordingly, the initialization of the pre-constructed anti-fraud identification model using the extracted features to obtain the initialized anti-fraud identification model includes:

[0029] The pre-built anti-fraud identification model is initialized using the warning indicators, risk features, user features, and base station location features to obtain the initialized anti-fraud identification model.

[0030] Optionally, the data processing method further includes:

[0031] Data analysis is performed using the risk information and standardized data stored in the preset database system to periodically update the streaming label rules and streaming indicator calculation rules defined based on the anti-fraud identification model, or to add new anti-fraud identification models.

[0032] In a second aspect, the present invention discloses a data processing apparatus, comprising:

[0033] The rule definition module is used to define streaming label rules and streaming indicator calculation rules based on the initialized anti-fraud identification model;

[0034] The data filtering module is used to filter the collected raw data according to preset whitelist rules to obtain filtered data.

[0035] The data standardization processing module is used to standardize the filtered data to obtain corresponding standardized data.

[0036] The data indicator calculation module is used to perform real-time indicator calculation on the standardized data according to the streaming indicator calculation rules to obtain the corresponding indicator results.

[0037] The data risk calculation module is used to forward the standardized data to the risk calculation and analysis node, so that the risk calculation and analysis node can use the indicator results corresponding to the standardized data to perform real-time risk calculation to obtain risk information corresponding to user behaviors that have gambling or fraud risks, and output the risk information.

[0038] Thirdly, the present invention discloses an electronic device, comprising:

[0039] Memory, used to store computer programs;

[0040] A processor is configured to execute the computer program to implement the steps of the aforementioned disclosed data processing method.

[0041] Fourthly, the present invention discloses a computer-readable storage medium for storing a computer program; wherein, when the computer program is executed by a processor, it implements the steps of the aforementioned disclosed data processing method.

[0042] As can be seen, this invention provides a data processing method, including: defining streaming label rules and streaming indicator calculation rules based on an initialized anti-fraud identification model; filtering the raw data collected using deep packet inspection technology according to preset whitelist rules to obtain filtered data, and standardizing the filtered data to obtain corresponding standardized data; performing real-time indicator calculation on the standardized data according to the streaming indicator calculation rules to obtain corresponding indicator results; forwarding the standardized data to a risk calculation and analysis node, so that the risk calculation and analysis node can use the indicator results corresponding to the standardized data to perform real-time risk calculation to obtain risk information corresponding to user behavior with gambling or fraud risks, and outputting the risk information. Therefore, this invention filters the raw data collected using deep packet inspection technology, then standardizes the filtered data to obtain standardized data, and then performs indicator calculation and risk calculation on the standardized data, achieving second-level early warning of user behavior with gambling or fraud risks, and improving the timeliness and accuracy of gambling and fraud data processing. Attached Figure Description

[0043] To more clearly illustrate the technical solutions in the embodiments of the present invention or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are only embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on the provided drawings without creative effort.

[0044] Figure 1 This is a flowchart of a data processing method disclosed in this invention;

[0045] Figure 2 This is a schematic diagram illustrating a supplementary data tag disclosed in this invention;

[0046] Figure 3 This is a schematic diagram of real-time data indicator calculation disclosed in this invention;

[0047] Figure 4 This is a schematic diagram of the structure of a data processing device disclosed in this invention;

[0048] Figure 5 This is a structural diagram of an electronic device disclosed in this invention. Detailed Implementation

[0049] The technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0050] Currently, due to the massive nature of DPI data, computation is intensive and time-consuming. Anti-gambling and anti-fraud information databases built using batch data processing combined with list collision analysis suffer from low alert timeliness, poor list quality, and limited alert accuracy, leading to increased costs in manpower and time. This makes them unsuitable for high-timeliness-required scenarios involving gambling and fraud data processing and alerting. Therefore, this invention provides a data processing solution that improves the timeliness and accuracy of gambling and fraud data processing.

[0051] This invention discloses a data processing method, see [link to relevant documentation]. Figure 1 As shown, the method includes:

[0052] Step S11: Define streaming label rules and streaming index calculation rules based on the initialized anti-fraud identification model.

[0053] Understandably, streaming labeling rules and streaming metric calculation rules are defined based on the initialized anti-fraud identification model for subsequent data labeling and metric calculation.

[0054] In this embodiment, before defining the streaming label rules and streaming indicator calculation rules based on the initialized anti-fraud identification model, the process may further include: performing clue mining and feature extraction processing on offline data to obtain corresponding extracted features; the offline data includes user information, base station information, case information, and access records; and using the extracted features to initialize the pre-built anti-fraud identification model to obtain the initialized anti-fraud identification model. It is understood that clue mining and feature extraction are performed on offline data to initialize the anti-fraud identification model using the extracted features. Specifically, natural language processing technology is used to extract the case occurrence period and related links from the case information, and the corresponding IP addresses, domain names, and Uniform Resource Identifiers (URIs) are determined based on the case occurrence period and related links. Rule-based induction is performed on the IP addresses, domain names, and URIs to construct corresponding early warning indicators. The user's access path and access frequency are determined from the access records, and corresponding risk characteristics are extracted based on the access path and access frequency. Rule-based induction is performed on the user information and base station information to determine user characteristics and base station location characteristics. The user characteristics include user age, user gender, and the device used by the user. In other words, the relevant links and the timeline of the case are extracted from the case information. Then, user operations are simulated based on the relevant links and the timeline to obtain all associated URI (Uniform Resource Identifier) ​​information, IP addresses, and domain name data. User access paths and frequencies are obtained from access records. User age, gender, terminal device used by the user, and base station location characteristics are summarized from user information and base station information. Then, feature extraction technology is used to summarize rules for IP addresses, domain names, and URI information to construct early warning indicators. Corresponding risk features are extracted from the access paths and frequencies. Finally, the pre-constructed anti-fraud identification model is initialized using the early warning indicators, risk features, user features, and base station location features to obtain the initialized anti-fraud identification model.

[0055] It is understandable that data clue mining and feature extraction are performed based on offline data, and then the extracted features are used to initialize the anti-fraud identification model. Based on the initialized model, streaming labeling rules and streaming indicator calculation rules are defined for subsequent data labeling and indicator calculation.

[0056] Step S12: Filter the raw data collected using deep packet inspection technology according to the preset whitelist rules to obtain filtered data, and then standardize the filtered data to obtain corresponding standardized data.

[0057] In this embodiment, the raw DPI data collected using deep packet inspection technology is filtered and standardized in real time. Specifically, the raw DPI data collected using deep packet inspection technology is filtered according to a preset whitelist rule to obtain filtered DPI data, and the filtered DPI data is then standardized to obtain corresponding standardized DPI data.

[0058] In this embodiment, the step of filtering the raw data collected using deep packet inspection technology according to a preset whitelist rule to obtain filtered data may specifically include: constructing a Bloom filter using a preset list of normal URLs, and using the Bloom filter to filter the raw data collected using deep packet inspection technology to obtain filtered data. That is, the Bloom filter is used to determine whether the domain name carried by the raw data collected using deep packet inspection technology meets the preset filtering conditions. The raw data includes a user number field, an access time field, an access domain name field, an access Uniform Resource Identifier field, a server IP address field, and a user access base station information field. If the domain name carried by the raw data meets the preset filtering conditions, the raw data is filtered; if the domain name carried by the raw data does not meet the preset filtering conditions, the raw data is retained to obtain filtered data. Understandably, the collected raw DPI data undergoes preprocessing, specifically by constructing a Bloom filter using a list of normal URLs. This pre-built Bloom filter filters the DPI data traffic to remove DPI data traffic from normal URL accesses. Furthermore, outlier handling and time formatting can be applied to the filtered data. The preprocessed DPI data structure is shown in Table 1.

[0059] Table 1

[0060] User number (encrypted) vWDs256zqcndPG1xMiK6HA== Access time 2023-03-20 15:36:00 Access Domain qqq.123.xyz Access URI http: / / qqq.123.xyz / 56app.apk server IP address 112.13.42.211 User access to base station information 22580_83503493

[0061] The original DPI data refers to data generated by user internet browsing behavior and business APP usage, and the original DPI data must contain data content in six fields: user number, access time, access domain name, access URI, server IP address, and user access base station information.

[0062] For example, the Bloom filter is used to determine whether the domain name carried in the collected raw DPI data is a normal website address and whether the domain name is a bank domain name; if the domain name carried in the raw DPI data is a normal website address and the domain name is a bank domain name, then the raw DPI data is filtered; if the domain name carried in the raw DPI data is not a normal website address or the domain name is not a bank domain name, then the raw DPI data that may indicate user behavior involving gambling or fraud is retained, resulting in filtered DPI data.

[0063] In this embodiment, the standardization process for the filtered data to obtain standardized data may specifically include: dividing the filtered data into bank-type data, APK-type data, and other suspicious-type data based on the domain name type carried by the filtered data; supplementing the other suspicious-type data with tags according to the streaming labeling rules to obtain tagged data of other suspicious-type data; merging the bank-type data, the APK-type data, and the tagged data of other suspicious-type data to obtain merged data; and using target information to complete the fields of the merged data to obtain standardized data. The target information includes the server's IP address location information, the user's access base station information, and the current timestamp information. In other words, the filtered DPI data is divided into bank-type, APK-type, and other suspicious-type DPI data based on the domain name type. For the other suspicious-type DPI data, tags are added according to the streaming labeling rules to supplement traffic tags. For example, see [link to relevant documentation]. Figure 2 As shown, a global method is used to calculate domain name tags, and other suspicious types of DPI data are tagged to supplement the tag information of other suspicious types of DPI data. The structure of the DPI data after supplementing the tags is shown in Table 2:

[0064] Table 2

[0065]

[0066]

[0067] In the JSON, 1 indicates that the tag is included, and 0 indicates that the tag is not included.

[0068] Understandably, filtered DPI traffic is categorized into bank-type, APK-type, and other suspicious types. For bank-type and APK-type DPI traffic, which are considered general risk access traffic, no tagging is required; only data completion is needed before subsequent indicator calculations. However, for other suspicious types of DPI traffic, data tagging is performed before data completion to supplement their tag information. The tagging process uses the domain characteristics of other suspicious traffic types and streaming tagging rules to continuously supplement domain characteristic tag fields for subsequent risk rule judgment.

[0069] In this embodiment, the data of the bank type, the data of the APK type, and the data of other suspicious types after tagging are merged. The merged DPI data is then padded with fields. For example, the IP location query service is called to supplement the IP location information. Then, the city code, town code, and district code are parsed from the user's access base station information, and the city code, town code, and district code are supplemented. The current timestamp information is also supplemented. The standardized DPI data structure after field completion is shown in Table 3.

[0070] Table 3

[0071]

[0072]

[0073] Step S13: Perform real-time index calculation on the standardized data according to the streaming index calculation rules to obtain the corresponding index results.

[0074] In this embodiment, after filtering, tagging, and field completion of the original DPI data to obtain standardized DPI data, the standardized DPI data is used to perform real-time indicator calculation according to the streaming indicator calculation rules to obtain the corresponding indicator results. That is, the tag information carried by the standardized DPI data is determined, and the corresponding indicator results are calculated in real time according to the tag information and the streaming indicator calculation rules.

[0075] It should be noted that for standardized DPI data, on the one hand, real-time index calculation is performed according to the streaming index calculation rules defined based on the initialized anti-fraud identification model, such as... Figure 2 As shown, on the other hand, it can be written into HDFS (Hadoop Distributed File System) for subsequent offline data analysis.

[0076] Step S14: Forward the standardized data to the risk calculation and analysis node, so that the risk calculation and analysis node can use the indicator results corresponding to the standardized data to perform real-time risk calculation to obtain risk information corresponding to user behaviors with gambling or fraud risks, and output the risk information.

[0077] In this embodiment, standardized data is forwarded to the risk calculation and analysis node in the form of events via a high-performance gateway. This allows the risk calculation node to load the pre-calculated multi-dimensional streaming indicator results corresponding to the standardized data. Based on these results, risk calculation and judgment are performed, and the calculated risk information is then persistently stored in the database and output with second-level risk warnings. The risk information can also be output via message middleware, including the user's geographical location information from their base station visit, for user location tracking. This allows for precise location tracking of individuals at risk of gambling or fraud, and then pushes the risk information to the corresponding city's public security department. This achieves highly accurate and timely risk warnings, enabling faster human intervention before financial losses occur, shortening the overall anti-gambling and anti-fraud response time, effectively preventing telecommunications network fraud, and protecting the property safety of the public. The data structure of the risk information is shown in Table 4 below.

[0078] Table 4

[0079]

[0080] The warning type includes multiple warning messages, and the number after the type field indicates the risk level. For example, 1, 2, and 3 can represent low, medium, and high risk levels, respectively.

[0081] In this embodiment, the risk information and standardized data stored in a preset database system are used for data analysis to periodically update the streaming label rules and streaming indicator calculation rules defined based on the anti-fraud identification model, or to add new anti-fraud identification models. That is, the stored risk information and standardized DPI data are periodically analyzed offline to update model rules or add new models, achieving iterative anti-fraud identification model development. For example, data analysis is performed using a T+1 scheduled execution of Spark tasks, and the iterated model is used to add or optimize indicator calculation methods to continuously improve the accuracy of anti-gambling and anti-fraud identification. For example, by extracting the characteristics of users who have been warned, the extracted user characteristics are used to capture the user's access records, and further secondary rule summarization is performed on the domain name, IP address, and URI information of the access records to update existing model rules or add new models.

[0082] As can be seen, in this embodiment of the invention, the raw data collected using deep packet inspection technology is filtered, and then the filtered data is standardized to obtain standardized data. Then, by performing indicator calculation and risk calculation on the standardized data, a second-level early warning of user behavior involving gambling or fraud risks is achieved, which can improve the timeliness and accuracy of gambling and fraud data processing.

[0083] Accordingly, embodiments of the present invention also disclose a data processing apparatus, see below. Figure 4 As shown, the device includes:

[0084] Rule definition module 11 is used to define streaming label rules and streaming index calculation rules based on the initialized anti-fraud identification model;

[0085] The data filtering module 12 is used to filter the collected raw data according to the preset whitelist rules to obtain filtered data.

[0086] Data standardization processing module 13 is used to standardize the filtered data to obtain corresponding standardized data;

[0087] Data indicator calculation module 14 is used to perform real-time indicator calculation on the standardized data according to the streaming indicator calculation rules to obtain the corresponding indicator results;

[0088] The data risk calculation module 15 is used to forward the standardized data to the risk calculation and analysis node, so that the risk calculation and analysis node can use the indicator results corresponding to the standardized data to perform real-time risk calculation to obtain risk information corresponding to user behaviors that have gambling or fraud risks, and output the risk information.

[0089] As can be seen from the above, in this embodiment of the invention, the raw data collected using deep packet inspection technology is filtered, and then the filtered data is standardized to obtain standardized data. Then, by performing indicator calculation and risk calculation on the standardized data, a second-level early warning of user behavior involving gambling or fraud risks is achieved, and the timeliness and accuracy of gambling and fraud data processing can be improved.

[0090] In some specific embodiments, the data filtering module 12 may specifically include:

[0091] The filter building submodule is used to build Bloom filters using a pre-defined list of normal URLs;

[0092] The filtering submodule is used to filter the raw data collected using deep packet inspection technology using the Bloom filter to obtain filtered data.

[0093] In some specific embodiments, the filtering submodule may specifically include:

[0094] The condition judgment unit is used to determine whether the domain name carried in the raw data collected by deep packet inspection technology meets the preset filtering conditions using the Bloom filter; wherein, the raw data is data containing user number field, access time field, access domain name field, access Uniform Resource Identifier field, server IP address field and user access base station information field;

[0095] A data filtering unit is used to filter the original data if the domain name carried by the original data meets the preset filtering conditions.

[0096] The data retention unit is used to retain the original data and obtain filtered data if the domain name carried by the original data does not meet the preset filtering conditions.

[0097] In some specific embodiments, the data standardization processing module 13 may specifically include:

[0098] The data segmentation submodule is used to segment the filtered data according to the type of the domain name carried by the filtered data to obtain data of bank type, APK type, and other suspicious types of data;

[0099] The tag supplementation submodule is used to supplement the tags of the other suspicious types of data according to the streaming tagging rules to obtain the other suspicious types of data after tagging.

[0100] The merge processing submodule is used to merge the bank-type data, the APK-type data, and the other suspicious-type data after tagging to obtain merged data.

[0101] The field completion submodule is used to complete the fields of the merged data using target information to obtain standardized data; wherein, the target information includes the IP address location information of the server, the user access base station information, and the current timestamp information.

[0102] In some specific embodiments, the data processing apparatus may further include:

[0103] The offline data processing module is used to perform clue mining and feature extraction on offline data to obtain corresponding extracted features; the offline data includes user information, base station information, case information, and access records;

[0104] The model initialization module is used to initialize the pre-built anti-fraud identification model using the extracted features to obtain the initialized anti-fraud identification model.

[0105] In some specific embodiments, the offline data processing module may specifically include:

[0106] The first information extraction submodule is used to extract the case occurrence period and the links involved in the case from the case information using natural language processing technology, and to determine the corresponding IP address, domain name and Uniform Resource Identifier information based on the case occurrence period and the links involved in the case.

[0107] The first information summarization submodule is used to summarize the IP address, the domain name and the Uniform Resource Identifier information according to rules to construct corresponding early warning indicators;

[0108] The second information extraction submodule is used to determine the user's access path and access frequency from the access records, and extract corresponding risk features based on the access path and access frequency.

[0109] The second information summarization submodule is used to perform rule summarization on the user information and the base station information to determine user characteristics and base station location characteristics; the user characteristics include user age, user gender, and the device used by the user.

[0110] In some specific embodiments, the model initialization module may specifically include:

[0111] The model initialization submodule is used to initialize the pre-built anti-fraud identification model using the warning indicators, the risk features, the user features, and the base station location features to obtain the initialized anti-fraud identification model.

[0112] In some specific embodiments, the data processing apparatus may further include:

[0113] The data analysis module is used to perform data analysis using the risk information and standardized data stored in the preset database system to periodically update the streaming label rules and streaming indicator calculation rules defined based on the anti-fraud identification model or to add new anti-fraud identification models.

[0114] Furthermore, embodiments of the present invention also provide an electronic device. Figure 5 This is a structural diagram of an electronic device 20 according to an exemplary embodiment. The content of the diagram should not be construed as limiting the scope of the invention.

[0115] Figure 5 This is a schematic diagram of the structure of an electronic device 20 provided in an embodiment of the present invention. Specifically, the electronic device 20 may include: at least one processor 21, at least one memory 22, a power supply 23, a communication interface 24, an input / output interface 25, and a communication bus 26. The memory 22 stores a computer program, which is loaded and executed by the processor 21 to implement the relevant steps in the data processing method disclosed in any of the foregoing embodiments. Furthermore, the electronic device 20 in this embodiment may specifically be an electronic computer.

[0116] In this embodiment, the power supply 23 is used to provide operating voltage for each hardware device on the electronic device 20; the communication interface 24 can create a data transmission channel between the electronic device 20 and external devices, and the communication protocol it follows can be any communication protocol applicable to the technical solution of this invention, and is not specifically limited here; the input / output interface 25 is used to acquire external input data or output data to the outside world, and its specific interface type can be selected according to specific application needs, and is not specifically limited here.

[0117] In addition, the memory 22, as a carrier for resource storage, can be a read-only memory, random access memory, disk or optical disk, etc. The resources stored thereon can include operating system 221, computer program 222, etc., and the storage method can be temporary storage or permanent storage.

[0118] The operating system 221 is used to manage and control the various hardware devices on the electronic device 20 and the computer program 222, which may be Windows Server, Netware, Unix, Linux, etc. In addition to including computer programs capable of performing the data processing methods executed by the electronic device 20 as disclosed in any of the foregoing embodiments, the computer program 222 may further include computer programs capable of performing other specific tasks.

[0119] Furthermore, embodiments of the present invention also disclose a computer-readable storage medium storing a computer program, which, when loaded and executed by a processor, implements the data processing method steps disclosed in any of the foregoing embodiments.

[0120] The various embodiments in this specification are described in a progressive manner, with each embodiment focusing on its differences from other embodiments. Similar or identical parts between embodiments can be referred to interchangeably. For the apparatus disclosed in the embodiments, since it corresponds to the method disclosed in the embodiments, the description is relatively simple; relevant parts can be referred to in the method section.

[0121] Finally, it should be noted that in this document, relational terms such as "first" and "second" are used only to distinguish one entity or operation from another, and do not necessarily require or imply any such actual relationship or order between these entities or operations. Furthermore, the terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such a process, method, article, or apparatus. Without further limitations, an element defined by the phrase "comprising one..." does not exclude the presence of other identical elements in the process, method, article, or apparatus that includes said element.

[0122] The above provides a detailed description of the data processing method, apparatus, device, and storage medium provided by the present invention. Specific examples have been used to illustrate the principles and implementation methods of the present invention. The description of the above embodiments is only for the purpose of helping to understand the method and core ideas of the present invention. At the same time, for those skilled in the art, there will be changes in the specific implementation methods and application scope based on the ideas of the present invention. Therefore, the content of this specification should not be construed as a limitation of the present invention.

Claims

1. A data processing method, characterized in that, include: Define streaming label rules and streaming index calculation rules based on the initialized anti-fraud identification model; The raw data collected using deep packet inspection technology is filtered according to a preset whitelist rule to obtain filtered data, and the filtered data is then standardized to obtain corresponding standardized data. The standardized data is calculated in real time according to the streaming index calculation rules to obtain the corresponding index results. The standardized data is forwarded to the risk calculation and analysis node, so that the risk calculation and analysis node can use the indicator results corresponding to the standardized data to perform real-time risk calculation to obtain risk information corresponding to user behaviors that have the risk of gambling or fraud, and output the risk information. The standardization process for the filtered data to obtain the corresponding standardized data includes: The filtered data is categorized based on the type of the domain name it carries, resulting in data of the bank type, data of the APK type, and other suspicious types. According to the streaming labeling rules, the other suspicious data types are supplemented with labels to obtain the tagged other suspicious data types. The data of the bank type, the data of the APK type, and the data of other suspicious types after tagging are merged to obtain the merged data; Standardized data is obtained by performing field completion processing on the merged data using the target information; wherein, the target information includes the IP address location information of the server, the user access base station information, and the current timestamp information.

2. The data processing method according to claim 1, characterized in that, The process of filtering the raw data collected using deep packet inspection technology according to preset whitelist rules to obtain filtered data includes: A Bloom filter is constructed using a pre-defined list of normal URLs, and the Bloom filter is used to filter the raw data collected using deep packet inspection technology to obtain filtered data.

3. The data processing method according to claim 2, characterized in that, The process of using the Bloom filter to filter the raw data acquired using deep packet inspection technology to obtain filtered data includes: The Bloom filter is used to determine whether the domain name carried in the raw data collected by deep packet inspection technology meets the preset filtering conditions; wherein, the raw data is data containing user number field, access time field, access domain name field, access Uniform Resource Identifier field, server IP address field and user access base station information field; If the domain name carried by the original data meets the preset filtering conditions, then the original data is filtered. If the domain name carried by the original data does not meet the preset filtering conditions, the original data is retained to obtain the filtered data.

4. The data processing method according to claim 1, characterized in that, Before defining the streaming label rules and streaming metric calculation rules based on the initialized anti-fraud identification model, the following is also included: Offline data is processed for clue mining and feature extraction to obtain corresponding extracted features; the offline data includes user information, base station information, case information, and access records; The extracted features are used to initialize the pre-constructed anti-fraud identification model to obtain the initialized anti-fraud identification model.

5. The data processing method according to claim 4, characterized in that, The process of performing clue mining and feature extraction on offline data to obtain the corresponding extracted features includes: Natural language processing technology is used to extract the case occurrence period and related links from the case information, and the corresponding IP address, domain name and Uniform Resource Identifier information are determined based on the case occurrence period and related links; The IP address, the domain name, and the Uniform Resource Identifier information are used to summarize rules to construct corresponding early warning indicators; The user's access path and access frequency are determined from the access records, and corresponding risk characteristics are extracted based on the access path and access frequency. The user information and the base station information are used to perform rule summarization to determine user characteristics and base station location characteristics; the user characteristics include user age, user gender, and the device used by the user. Accordingly, the initialization of the pre-constructed anti-fraud identification model using the extracted features to obtain the initialized anti-fraud identification model includes: The pre-built anti-fraud identification model is initialized using the warning indicators, risk features, user features, and base station location features to obtain the initialized anti-fraud identification model.

6. The data processing method according to any one of claims 1 to 5, characterized in that, Also includes: Data analysis is performed using the risk information and standardized data stored in the preset database system to periodically update the streaming label rules and streaming indicator calculation rules defined based on the anti-fraud identification model, or to add new anti-fraud identification models.

7. A data processing apparatus, characterized in that, include: The rule definition module is used to define streaming label rules and streaming indicator calculation rules based on the initialized anti-fraud identification model; The data filtering module is used to filter the collected raw data according to preset whitelist rules to obtain filtered data. The data standardization processing module is used to standardize the filtered data to obtain corresponding standardized data. The data indicator calculation module is used to perform real-time indicator calculation on the standardized data according to the streaming indicator calculation rules to obtain the corresponding indicator results. The data risk calculation module is used to forward the standardized data to the risk calculation and analysis node, so that the risk calculation and analysis node can use the indicator results corresponding to the standardized data to perform real-time risk calculation to obtain risk information corresponding to user behaviors that have gambling or fraud risks, and output the risk information. The data standardization processing module includes: The data segmentation submodule is used to segment the filtered data according to the type of the domain name carried by the filtered data to obtain data of bank type, APK type, and other suspicious types of data; The tag supplementation submodule is used to supplement the tags of the other suspicious types of data according to the streaming tagging rules to obtain the other suspicious types of data after tagging. The merge processing submodule is used to merge the bank-type data, the APK-type data, and the other suspicious-type data after tagging to obtain merged data. The field completion submodule is used to complete the fields of the merged data using target information to obtain standardized data; wherein, the target information includes the IP address location information of the server, the user access base station information, and the current timestamp information.

8. An electronic device, characterized in that, include: Memory, used to store computer programs; A processor for executing the computer program to implement the steps of the data processing method as described in any one of claims 1 to 6.

9. A computer-readable storage medium, characterized in that, Used to store a computer program; wherein, when the computer program is executed by a processor, it implements the steps of the data processing method as described in any one of claims 1 to 6.

Images