Abnormal user detection method and device, computer equipment and storage medium
By introducing LASSO algorithm and data preprocessing steps into the linear regression model, the multicollinearity problem of traditional models in high-dimensional feature data is solved, and the generalization ability and accuracy of the model are improved.
Patent Information
- Application Number
- CN202411187883.1
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2024-08-27
- Publication Date
- 2024-11-29
AI Technical Summary
Traditional linear regression models are difficult to effectively deal with multicollinearity problems when processing high-dimensional feature data, resulting in a decrease in model stability and prediction accuracy, and may overfit, affecting generalization ability.
The linear regression model is optimized by using the LASSO algorithm. By obtaining historical business data corresponding to the target business scenario for encoding, cleaning, redundancy detection and eliminating, an initial linear regression model is constructed, and user data is obtained in the system database for abnormalities. predict.
It effectively reduces the complexity of the model, avoids overfitting, improves the generalization ability of the model on unknown data, realizes automatic selection and dimensionality reduction of features, and significantly improves the accuracy of the model.
Smart Images

Figure CN119046736A_ABST
Abstract
Description
Technical Field
[0001] The present application relates to the field of artificial intelligence technology, and in particular to an abnormal user detection method, device, computer equipment and storage medium. Background Art
[0002] With the rapid development of big data technology and the sharp increase in the amount of enterprise data, traditional linear regression models are facing more and more challenges. Especially when dealing with business scenarios with high-dimensional features, the problem of multicollinearity in the data is particularly prominent, which directly affects the stability and prediction accuracy of the model. In addition, high-dimensional data may contain many unimportant or redundant features, which not only increase the complexity of the model, but also may lead to overfitting, affecting the generalization ability of the model on unknown data.
[0003] Therefore, how to automatically select the most influential features from a large number of features and effectively deal with the problem of multicollinearity becomes the key to improving the generalization ability of the model. Summary of the invention
[0004] The purpose of the embodiments of the present application is to propose an abnormal user detection method, device, computer equipment and storage medium to solve the problem that the traditional linear regression model cannot effectively handle the multicollinearity problem and improve the generalization ability of the model.
[0005] In order to solve the above technical problems, the embodiment of the present application provides an abnormal user detection method, which adopts the following technical solution:
[0006] Obtain historical business data corresponding to the target business scenario;
[0007] Performing encoding operation on the historical service data to obtain a service feature matrix;
[0008] Constructing an initial linear regression model, inputting the business feature matrix into the initial linear regression model for model training operation, and optimizing the model training operation according to the LASSO algorithm to obtain a target linear regression model;
[0009] Receiving an anomaly detection request sent by a user terminal, wherein the anomaly detection request includes at least a to-be-detected user identifier corresponding to the target service scenario;
[0010] Calling a system database, and acquiring user service data corresponding to the user identifier to be detected in the system database;
[0011] Performing encoding operation on the user service data to obtain a user feature matrix;
[0012] The user feature matrix is input into the target linear regression model to perform an abnormality prediction operation to obtain an abnormality prediction result.
[0013] Furthermore, after the step of acquiring the historical service data corresponding to the target service scenario and before the step of encoding the historical service data to obtain the service feature matrix, the following steps are also included:
[0014] Performing a data cleaning operation on the historical business data to obtain cleaned business data;
[0015] Performing data integration operation on the cleaning service data to obtain an integrated data table;
[0016] Performing a redundancy detection operation on the integrated data table to obtain a redundancy detection result;
[0017] Performing a data elimination operation on the historical business data according to the redundant detection result to obtain pre-processed business data;
[0018] The step of performing encoding operation on the historical service data to obtain a service feature matrix specifically comprises the following steps:
[0019] An encoding operation is performed on the preprocessed service data to obtain the service feature matrix.
[0020] Furthermore, the objective function of the model training operation is expressed as:
[0021]
[0022] Among them, X ij represents the jth eigenvector of the i-th sample, y i represents the target variable of the i-th sample, β j represents the intercept term, β represents the coefficient vector, and λ represents the weight parameter of the L1 regularization term.
[0023] Furthermore, after the step of acquiring the historical business data corresponding to the target business scenario, the following steps are also included:
[0024] Identify highly relevant data corresponding to the target business scenario in the historical business data according to statistical analysis and visual data analysis;
[0025] Acquire feedback information of the model training operation, and confirm key business features as inputs of the target linear regression model based on the feedback information and the highly correlated data.
[0026] In order to solve the above technical problems, the embodiment of the present application further provides an abnormal user detection device, which adopts the following technical solution:
[0027] A historical data acquisition module is used to acquire historical business data corresponding to the target business scenario;
[0028] A first encoding module, used for encoding the historical service data to obtain a service feature matrix;
[0029] A model training module is used to construct an initial linear regression model, input the business feature matrix into the initial linear regression model to perform a model training operation, and optimize the model training operation according to the LASSO algorithm to obtain a target linear regression model;
[0030] A request acquisition module, configured to receive an anomaly detection request sent by a user terminal, wherein the anomaly detection request at least includes a to-be-detected user identifier corresponding to the target business scenario;
[0031] A user data acquisition module, used to call a system database and acquire user service data corresponding to the user identifier to be detected in the system database;
[0032] A second encoding module is used to perform an encoding operation on the user service data to obtain a user feature matrix;
[0033] The abnormality prediction module is used to input the user feature matrix into the target linear regression model to perform an abnormality prediction operation and obtain an abnormality prediction result.
[0034] Furthermore, the device includes: a data cleaning module, a data integration module, a redundancy detection module and a data elimination module, and the first encoding module includes: a first encoding submodule, wherein:
[0035] A data cleaning module, used to perform a data cleaning operation on the historical business data to obtain cleaned business data;
[0036] A data integration module, used to perform data integration operations on the cleaning business data to obtain an integrated data table;
[0037] A redundancy detection module, used for performing a redundancy detection operation on the integrated data table to obtain a redundancy detection result;
[0038] A data elimination module, used for performing a data elimination operation on the historical business data according to the redundant detection result to obtain pre-processed business data;
[0039] The first encoding submodule is used to perform encoding operation on the pre-processed service data to obtain the service feature matrix.
[0040] In order to solve the above technical problems, the embodiment of the present application further provides a computer device, which adopts the following technical solution:
[0041] The method comprises a memory and a processor, wherein the memory stores computer-readable instructions, and the processor implements the steps of the abnormal user detection method as described above when executing the computer-readable instructions.
[0042] In order to solve the above technical problems, the embodiment of the present application further provides a computer-readable storage medium, which adopts the following technical solution:
[0043] The computer-readable storage medium stores computer-readable instructions, and when the computer-readable instructions are executed by the processor, the steps of the abnormal user detection method described above are implemented.
[0044] The present application provides an abnormal user detection method, comprising: acquiring historical business data corresponding to a target business scenario; encoding the historical business data to obtain a business feature matrix; constructing an initial linear regression model, inputting the business feature matrix into the initial linear regression model for model training, and optimizing the model training operation according to a LASSO algorithm; when the model training operation satisfies a convergence condition, obtaining a target linear regression model; receiving an abnormal detection request sent by a user terminal, wherein the abnormal detection request at least includes a user identifier to be detected corresponding to the target business scenario; calling a system database, and obtaining user business data corresponding to the user identifier to be detected in the system database; encoding the user business data to obtain a user feature matrix; inputting the user feature matrix into the target linear regression model for abnormal prediction operation to obtain an abnormal prediction result; and outputting the abnormal prediction result to the user terminal. Compared with the prior art, the present application effectively reduces the complexity of the model, avoids the overfitting problem, and significantly improves the generalization ability of the model on unknown data. In addition, through the introduction of LASSO, automatic feature selection and dimensionality reduction are achieved, which effectively solves the multicollinearity problem and greatly improves the accuracy of the model. BRIEF DESCRIPTION OF THE DRAWINGS
[0045] In order to more clearly illustrate the scheme in the present application, a brief introduction is given below to the drawings required for use in the description of the embodiments of the present application. Obviously, the drawings described below are some embodiments of the present application. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying any creative work.
[0046] Figure 1 is an exemplary system architecture diagram to which the present application may be applied;
[0047] Figure 2 is a flowchart of an implementation of the abnormal user detection method provided in Example 1 of the present application;
[0048] Figure 3 is a structural diagram of an abnormal user detection device provided in Embodiment 2 of the present application;
[0049] Figure 4 It is a structural diagram of an embodiment of a computer device according to the present application. DETAILED DESCRIPTION
[0050] Unless otherwise defined, all technical and scientific terms used herein have the same meaning as those commonly understood by technicians in the technical field of the present application; the terms used in the specification of the application herein are only for the purpose of describing specific embodiments and are not intended to limit the present application; the terms "including" and "having" and any variations thereof in the specification and claims of the present application and the above-mentioned drawings are intended to cover non-exclusive inclusions. The terms "first", "second", etc. in the specification and claims of the present application or the above-mentioned drawings are used to distinguish different objects, not to describe a specific order.
[0051] Reference to "embodiments" herein means that a particular feature, structure, or characteristic described in conjunction with the embodiments may be included in at least one embodiment of the present application. The appearance of the phrase in various locations in the specification does not necessarily refer to the same embodiment, nor is it an independent or alternative embodiment that is mutually exclusive with other embodiments. It is explicitly and implicitly understood by those skilled in the art that the embodiments described herein may be combined with other embodiments.
[0052] In order to enable those skilled in the art to better understand the solution of the present application, the technical solution in the embodiments of the present application will be clearly and completely described below in conjunction with the accompanying drawings.
[0053] like Figure 1 As shown, the system architecture 100 may include terminal devices 101, 102, 103, a network 104 and a server 105. The network 104 is used to provide a medium for communication links between the terminal devices 101, 102, 103 and the server 105. The network 104 may include various connection types, such as wired, wireless communication links or optical fiber cables, etc.
[0054] Users can use terminal devices 101, 102, 103 to interact with server 105 through network 104 to receive or send messages, etc. Various communication client applications can be installed on terminal devices 101, 102, 103, such as web browser applications, shopping applications, search applications, instant messaging tools, email clients, social platform software, etc.
[0055] Terminal devices 101, 102, 103 can be various electronic devices with display screens and supporting web browsing, including but not limited to smart phones, tablet computers, e-book readers, MP3 players (Moving Picture Experts Group Audio Layer III), MP4 (Moving Picture Experts Group Audio Layer IV) players, laptop computers, desktop computers, etc.
[0056] The server 105 may be a server that provides various services, such as a background server that provides support for web pages displayed on the terminal devices 101 , 102 , and 103 .
[0057] It should be noted that the abnormal user detection method provided in the embodiment of the present application is generally executed by a server / terminal device, and accordingly, the abnormal user detection device is generally set in the server / terminal device.
[0058] It should be understood that Figure 1 The number of terminal devices, networks and servers in the embodiment is only for illustration. Any number of terminal devices, networks and servers may be provided according to implementation requirements.
[0059] Embodiment 1
[0060] Continue to refer Figure 2 , shows a flow chart of an embodiment of an abnormal user detection method according to the present application. The abnormal user detection method comprises: step S201, step S202, step S203, step S204, step S205, step S206, step S207, step S208 and step S209.
[0061] In step S201, historical business data corresponding to the target business scenario is obtained.
[0062] In an embodiment of the present application, in order to improve the prediction accuracy and applicability of the model, historical business data related to the business scenario is first queried from the enterprise database. As an example, the historical business data can be vehicle attribute data such as vehicle model, age, mileage, etc., driver attribute data such as driving experience, past claims record, violation record, etc., as well as accident attribute data such as accident type, accident severity, weather and road conditions, etc. It should be understood that the examples of historical business data here are only for the convenience of understanding and are not used to limit this application.
[0063] In step S202, the historical service data is encoded to obtain a service feature matrix.
[0064] In the embodiment of the present application, the acquired historical business data is encoded. As an example, for example, the vehicle model is converted into a numerical identifier through One-Hot Encoding, the vehicle age and mileage are directly input as numerical values, and the driver's driving experience and the number of past claims records are also processed according to the actual numerical values; accident types such as "collision" and "scratches" are converted into numerical values through label encoding, and the severity of the accident is converted into numerical weights according to the accident report rating; weather and road condition information, such as "sunny" and "rainy", are converted into numerical data through One-Hot Encoding; these encoded data are combined with claim data (such as claim amount, claim time interval) to form a complete business feature matrix.
[0065] In step S203, an initial linear regression model is constructed, the business feature matrix is input into the initial linear regression model for model training operation, and the model training operation is optimized according to the LASSO algorithm to obtain a target linear regression model.
[0066] In an embodiment of the present application, a linear regression model is used to perform preliminary training on the business feature matrix, and the degree of fit of the model to the data is observed. Specifically, the performance of the model can be preliminarily evaluated by calculating the R2 value and mean square error (MSE) of the model.
[0067] In the embodiment of the present application, the LASSO regression method is used to optimize the model parameters. This method can reduce the number of features by penalizing the coefficient with the smallest absolute value, thereby controlling overfitting and ensuring the generalization ability of the model.
[0068] In some optional implementations of the embodiments of the present application, the optimized model can be trained again, and a cross-validation method (such as K-fold cross-validation) can be used to ensure the robustness of the model.
[0069] In an embodiment of the present application, the number of iterations and performance changes are monitored during the model training process. For example, if the improvement of the model does not exceed an error threshold of 0.1 after five consecutive iterations, the model can be considered to be stable.
[0070] In an embodiment of the present application, the model is combined with additional fraud risk attributes (such as high-risk area identification, suspicious repair shop records, and suspicious witness information) and further trained to improve the ability to identify potential fraudulent behavior.
[0071] In an embodiment of the present application, the linear regression model obtained by the above method can be effectively applied in the actual claims review process to help identify and predict potential fraud risks in claims cases, thereby reducing the financial losses of insurance companies.
[0072] In step S204, an anomaly detection request sent by a user terminal is received, wherein the anomaly detection request at least includes a to-be-detected user identifier corresponding to the target service scenario.
[0073] In step S205, the system database is called to obtain user service data corresponding to the user identification to be detected in the system database.
[0074] In step S206, the user service data is encoded to obtain a user feature matrix.
[0075] In step S207, the user feature matrix is input into the target linear regression model to perform an abnormality prediction operation to obtain an abnormality prediction result.
[0076] In an embodiment of the present application, an abnormal user detection method is provided, including: obtaining historical business data corresponding to a target business scenario; encoding the historical business data to obtain a business feature matrix; constructing an initial linear regression model, inputting the business feature matrix into the initial linear regression model for model training, and optimizing the model training operation according to the LASSO algorithm to obtain a target linear regression model; receiving an abnormal detection request sent by a user terminal, wherein the abnormal detection request at least includes a user identifier to be detected corresponding to the target business scenario; calling a system database, obtaining user business data corresponding to the user identifier to be detected in the system database; encoding the user business data to obtain a user feature matrix; inputting the user feature matrix into the target linear regression model for abnormal prediction operation to obtain an abnormal prediction result. Compared with the prior art, the present application effectively reduces the complexity of the model, avoids the overfitting problem, and significantly improves the generalization ability of the model on unknown data. In addition, through the introduction of LASSO, automatic feature selection and dimensionality reduction are realized, which effectively solves the multicollinearity problem and greatly improves the accuracy of the model.
[0077] In some optional implementations of the embodiments of the present application, after step S201 and before step S202, the following steps may also be included:
[0078] Perform data cleaning operations on historical business data to obtain cleaned business data;
[0079] Perform data integration operations on the cleaning business data to obtain an integrated data table;
[0080] Performing a redundancy detection operation on the integrated data table to obtain a redundancy detection result;
[0081] Perform data elimination operation on historical business data according to the redundant detection result to obtain pre-processed business data;
[0082] Step S202 specifically includes the following steps:
[0083] The preprocessed business data is encoded to obtain a business feature matrix.
[0084] In the embodiment of the present application, the data is first cleaned by using the Pandas library in the Python programming language to remove noise data such as meaningless special symbols and outliers. Then, the cleaned business data is integrated to combine the data from multiple data sources into a coherent data table, and data redundancy detection is performed based on the data table to obtain redundant data information.
[0085] In the embodiment of the present application, the redundancy detection may be implemented as follows:
[0086] 1) Analyze the relationship between data tables:
[0087] ① Check how each table is connected and whether there are redundant key values and data columns;
[0088] ② Eliminate duplicate data by deleting these columns, modifying the table structure, or merging tables;
[0089] 2) Retrieve duplicate records:
[0090] ① Use SQL statements to query all duplicate data items in the table and mark invalid records;
[0091] ② After marking, you can delete them manually or write SQL commands to automatically delete them;
[0092] 3. Database design analysis:
[0093] ① Use tools such as ER diagram (entity-relationship diagram) and data dictionary to analyze database design and find the existence of redundant data;
[0094] 4. Database query analysis:
[0095] ① By querying the database, analyzing the relationship between data tables, and identifying whether there is redundant data;
[0096] 5. Database statistical analysis:
[0097] ① Count the data in the statistical table, analyze the data distribution, and identify whether there is redundant data;
[0098] ②If a data value appears multiple times in multiple data tables, there may be redundant data;
[0099] 6. Database index analysis:
[0100] ① By analyzing the indexes in the database, identify whether there are duplicate indexes and redundant indexes. Duplicate indexes and redundant indexes will cause database performance to degrade and increase the complexity of data management;
[0101] 7. Use deduplication technology:
[0102] In spreadsheet software such as Excel, you can use the "conditional formatting" function to mark duplicate rows, and then delete redundant data based on the marked content;
[0103] In the database, you can directly use deduplication commands similar to the distinct function.
[0104] It should be understood that the examples of redundant detection operations here are only for ease of understanding and are not intended to limit the present application.
[0105] In some optional implementations of the embodiments of the present application, the objective function of the model training operation is expressed as:
[0106]
[0107] Among them, X ij represents the jth eigenvector of the i-th sample, y i represents the target variable of the i-th sample, β j represents the intercept term, β represents the coefficient vector, and λ represents the weight parameter of the L1 regularization term.
[0108] In some optional implementations of the embodiments of the present application, step S203 specifically includes the following steps:
[0109] Determine the optimal weight parameters of the objective function based on the feature cross-validation method;
[0110] The coefficient vector β is updated according to the coordinate descent method until the difference between the current value of the objective function in the current iteration and the objective function value obtained in the previous iteration meets the preset threshold, and the final coefficient vector β is obtained.
[0111] In the present application embodiment, in order to determine the best regularization parameter in the LASSO regression model, we need to select a set of candidate parameters, such as 1, 1, 10 and 100, which will be used to adjust the complexity of the model to prevent overfitting or underfitting. Next, the 10-fold cross validation method is used to evaluate the impact of these regularization parameters on model performance. By dividing the data set into 10 parts, 9 parts are used for training each time, and the remaining 1 part is used for testing, so that the model can be fully verified on different subsets. In each verification, the mean square error (MSE) and mean absolute error (MAE) of the model on the test set are calculated. These two indicators help us understand the accuracy and stability of model prediction. By comparing the MSE and MAE under different regularization parameters, we can find the optimal regularization parameter that minimizes the MSE. For example, if it is found that the MSE of the model reaches the minimum when the regularization parameter is 1, then we will select 1 as the best parameter. After finding the best regularization parameter, the LASSO model is retrained using the entire data set to ensure that the model fully learns all data features. After the model training is completed, we can obtain the coefficient vector and intercept term of the model. Each element in the coefficient vector represents the weight of the corresponding feature in the prediction, and the intercept term is the basic output value of the model. Through the above steps, we can build a parameter-optimized LASSO regression model, which not only considers preventing overfitting, but also optimizes the model's ability to interpret data, making full preparations for future prediction tasks.
[0112] In the embodiments of the present application, it is possible to ensure that the model provides high-quality prediction results in practical applications, which helps to make more accurate decisions in various business scenarios.
[0113] In the embodiment of the present application, the specific implementation method of using the coordinate descent method to optimize LASSO can be:
[0114] 1. Initialization: Randomly give the initial values of β0 and β (basically 0);
[0115] 2. Loop convergence:
[0116] For each feature j, other coefficients are fixed and only β is optimized:
[0117] Update β0 first: Update the intercept term by calculating the mean of the residuals.
[0118] Residual: The estimated model risk fraud value and the actual model risk fraud value (100 if there is risk fraud, 0 if there is no risk fraud);
[0119] Intercept term: constant of linear model: basic target value when no feature influence is considered;
[0120] Then update βj : The soft threshold function is used here to update (soft_thresholding function package in python) to find out the impact of each feature on the predicted risk fraud value.
[0121] Among them, the soft threshold function is expressed as:
[0122] β j =sign(β j )·max(0,|β j |-λ)
[0123] In the embodiment of the present application, the soft threshold function is mainly used to make unimportant feature coefficients tend to zero. Where λ is a regularization parameter, sign is a sign function (returns the sign of a real number, if x>0, returns 1, x=0, returns 0, x<0, returns -1).
[0124] In the embodiment of the present application, because m is relatively large and there are requirements for timeliness and resources in the existing business scenarios, not all features can be used, and the required coefficients will tend to be compressed to zero to obtain a sparser model, so it is necessary to select a larger λ.
[0125] In some optional implementations of the embodiments of the present application, after step S201, the following steps are further included:
[0126] Use statistical analysis and visual data analysis to identify highly relevant data corresponding to the target business scenario in historical business data;
[0127] Get feedback from model training operations and use the feedback and highly relevant data to identify key business features that serve as input to the target linear regression model.
[0128] In the embodiment of the present application, according to the business scenario and goal, expert knowledge and historical data analysis are used to obtain a preliminary feature set closely related to the business, and a preliminary feature list is obtained, which contains various factors that may affect the prediction results. By statistically analyzing the preliminary feature list, using methods such as information gain and Gini impurity, the discrimination of each feature is determined, and key features that can effectively distinguish different categories or prediction targets are obtained. The key features obtained are then subjected to time series analysis and variance analysis, and stability test methods such as ADF test are used to obtain features with relatively stable statistical characteristics. These features are not very sensitive to data fluctuations and are suitable for long-term use. Through discussions with business experts and literature review, the business explanation of the features is obtained, and the relationship between these features and business goals is tested using models with strong interpretability such as logistic regression to obtain features with high interpretability. Correlation analysis tools such as Pearson correlation coefficient and Spearman rank correlation coefficient are used to analyze the correlation between features, determine which features have strong correlations, and obtain a feature set after removing redundant features. Use data quality assessment tools to check the data integrity and accuracy of each feature, obtain indicators such as the missing value ratio and outlier ratio in the feature, judge the data quality of the feature, and ensure that the feature data used is of high quality without too many missing values or outliers.
[0129] In the embodiment of the present application, the above steps can be used to obtain a feature set that meets business needs, has high discrimination, stability, interpretability, reasonable correlation, and good data quality, which can be used for subsequent model training and business decision analysis.
[0130] The embodiments of the present application can acquire and process relevant data based on artificial intelligence technology. Among them, artificial intelligence (AI) is the theory, method, technology and application system that uses digital computers or machines controlled by digital computers to simulate, extend and expand human intelligence, perceive the environment, acquire knowledge and use knowledge to obtain the best results.
[0131] AI basic technologies generally include sensors, dedicated AI chips, cloud computing, distributed storage, big data processing technology, operation / interaction systems, mechatronics, etc. AI software technologies mainly include computer vision technology, robotics technology, biometrics technology, speech processing technology, natural language processing technology, and machine learning / deep learning.
[0132] Those skilled in the art can understand that all or part of the processes in the above-mentioned embodiments can be implemented by instructing the relevant hardware through computer-readable instructions, and the computer-readable instructions can be stored in a computer-readable storage medium. When the program is executed, it can include the processes of the embodiments of the above-mentioned methods. Among them, the aforementioned storage medium can be a non-volatile storage medium such as a disk, an optical disk, a read-only memory (ROM), or a random access memory (RAM).
[0133] It should be understood that, although the steps in the flowchart of the accompanying drawings are displayed in sequence as indicated by the arrows, these steps are not necessarily executed in sequence in the order indicated by the arrows. Unless otherwise specified herein, there is no strict order restriction on the execution of these steps, and they can be executed in other orders. Moreover, at least a part of the steps in the flowchart of the accompanying drawings may include multiple sub-steps or multiple stages, and these sub-steps or stages are not necessarily executed at the same time, but can be executed at different times, and their execution order is not necessarily sequential, but can be executed in turn or alternately with other steps or at least a part of the sub-steps or stages of other steps.
[0134] Embodiment 2
[0135] Further references Figure 3 , as a response to the above Figure 2 The present application provides an embodiment of an abnormal user detection device. Figure 2 Corresponding to the method embodiment shown, the device can be specifically applied to various electronic devices.
[0136] like Figure 3 As shown, the abnormal user detection device 200 of the embodiment of the present application includes:
[0137] A historical data acquisition module 210 is used to acquire historical business data corresponding to a target business scenario;
[0138] The first encoding module 220 is used to perform encoding operation on the historical service data to obtain a service feature matrix;
[0139] The model training module 230 is used to construct an initial linear regression model, input the business feature matrix into the initial linear regression model for model training, and optimize the model training operation according to the LASSO algorithm to obtain a target linear regression model;
[0140] The request acquisition module 240 is used to receive an anomaly detection request sent by a user terminal, wherein the anomaly detection request at least includes a user identifier to be detected corresponding to a target business scenario;
[0141] The user data acquisition module 250 is used to call the system database and obtain the user service data corresponding to the user identification to be detected in the system database;
[0142] The second encoding module 260 is used to perform encoding operation on the user service data to obtain a user feature matrix;
[0143] The abnormality prediction module 270 is used to input the user feature matrix into the target linear regression model to perform an abnormality prediction operation and obtain an abnormality prediction result.
[0144] In an embodiment of the present application, an abnormal user detection device 200 is provided, including: a historical data acquisition module 210, used to acquire historical business data corresponding to a target business scenario; a first encoding module 220, used to encode the historical business data to obtain a business feature matrix; a model training module 230, used to construct an initial linear regression model, input the business feature matrix into the initial linear regression model for model training, and optimize the model training operation according to the LASSO algorithm; obtain a target linear regression model; a request acquisition module 240, used to receive an abnormal detection request sent by a user terminal, wherein the abnormal detection request at least includes a user identifier to be detected corresponding to the target business scenario; a user data acquisition module 250, used to call a system database, and obtain user business data corresponding to the user identifier to be detected in the system database; a second encoding module 260, used to encode the user business data to obtain a user feature matrix; an abnormal prediction module 270, used to input the user feature matrix into the target linear regression model for abnormal prediction operation, and obtain an abnormal prediction result. Compared with the prior art, the present application effectively reduces the complexity of the model, avoids the overfitting problem, and significantly improves the generalization ability of the model on unknown data. In addition, through the introduction of LASSO, automatic feature selection and dimensionality reduction are achieved, which effectively solves the multicollinearity problem and greatly improves the accuracy of the model.
[0145] In some optional implementations of the embodiments of the present application, the abnormal user detection device 200 further includes: a data cleaning module, a data integration module, a redundancy detection module and a data elimination module, and the first encoding module includes: a first encoding submodule, wherein:
[0146] A data cleaning module is used to perform data cleaning operations on historical business data to obtain cleaned business data;
[0147] The data integration module is used to perform data integration operations on the cleaning business data to obtain an integrated data table;
[0148] A redundancy detection module is used to perform a redundancy detection operation on the integrated data table to obtain a redundancy detection result;
[0149] A data elimination module is used to perform data elimination operations on historical business data according to the redundant detection results to obtain pre-processed business data;
[0150] The first encoding submodule is used to perform encoding operation on the pre-processed service data to obtain a service feature matrix.
[0151] To solve the above technical problems, the present application also provides a computer device. Figure 4 , Figure 4 This is a basic structural block diagram of a computer device according to an embodiment of the present application.
[0152] The computer device 300 includes a memory 310, a processor 320, and a network interface 330 that are interconnected and communicated through a system bus. It should be noted that the figure only shows a computer device 300 having components 310-330, but it should be understood that it is not required to implement all the components shown, and more or fewer components can be implemented instead. Among them, those skilled in the art can understand that the computer device here is a device that can automatically perform numerical calculations and / or information processing according to pre-set or stored instructions, and its hardware includes but is not limited to microprocessors, application specific integrated circuits (Application Specific Integrated Circuit, ASIC), programmable gate arrays (Field-Programmable Gate Array, FPGA), digital processors (Digital Signal Processor, DSP), embedded devices, etc.
[0153] The computer device may be a computing device such as a desktop computer, a notebook, a PDA, a cloud server, etc. The computer device may interact with a user through a keyboard, a mouse, a remote controller, a touch pad, or a voice control device.
[0154] The memory 310 includes at least one type of readable storage medium, and the readable storage medium includes flash memory, hard disk, multimedia card, card-type memory (for example, SD or DX memory, etc.), random access memory (RAM), static random access memory (SRAM), read-only memory (ROM), electrically erasable programmable read-only memory (EEPROM), programmable read-only memory (PROM), magnetic memory, magnetic disk, optical disk, etc. In some embodiments, the memory 310 can be an internal storage unit of the computer device 300, such as a hard disk or memory of the computer device 300. In other embodiments, the memory 310 can also be an external storage device of the computer device 300, such as a plug-in hard disk equipped on the computer device 300, a smart memory card (Smart Media Card, SMC), a secure digital (Secure Digital, SD) card, a flash card, etc. Of course, the memory 310 can also include both the internal storage unit of the computer device 300 and its external storage device. In the embodiment of the present application, the memory 310 is generally used to store the operating system and various application software installed on the computer device 300, such as computer-readable instructions of the abnormal user detection method, etc. In addition, the memory 310 can also be used to temporarily store various data that have been output or are to be output.
[0155] The processor 320 may be a central processing unit (CPU), a controller, a microcontroller, a microprocessor, or other data processing chip in some embodiments. The processor 320 is generally used to control the overall operation of the computer device 300. In the embodiment of the present application, the processor 320 is used to run the computer-readable instructions or process data stored in the memory 310, such as computer-readable instructions for running the abnormal user detection method.
[0156] The network interface 330 may include a wireless network interface or a wired network interface. The network interface 330 is generally used to establish a communication connection between the computer device 300 and other electronic devices.
[0157] The computer device provided by the present application effectively reduces the complexity of the model, avoids the overfitting problem, and significantly improves the generalization ability of the model on unknown data. In addition, through the introduction of LASSO, automatic feature selection and dimensionality reduction are achieved, which effectively solves the multicollinearity problem and greatly improves the accuracy of the model.
[0158] The present application also provides another implementation, namely, providing a computer-readable storage medium, wherein the computer-readable storage medium stores computer-readable instructions, and the computer-readable instructions can be executed by at least one processor to enable the at least one processor to perform the steps of the abnormal user detection method as described above.
[0159] The computer-readable storage medium provided in the present application effectively reduces the complexity of the model, avoids the overfitting problem, and significantly improves the generalization ability of the model on unknown data. In addition, through the introduction of LASSO, automatic feature selection and dimensionality reduction are achieved, which effectively solves the multicollinearity problem and greatly improves the accuracy of the model.
[0160] Through the description of the above implementation methods, those skilled in the art can clearly understand that the above-mentioned embodiment methods can be implemented by means of software plus a necessary general hardware platform, and of course by hardware, but in many cases the former is a better implementation method. Based on such an understanding, the technical solution of the present application, or the part that contributes to the prior art, can be embodied in the form of a software product, which is stored in a storage medium (such as ROM / RAM, magnetic disk, optical disk), and includes a number of instructions for a terminal device (which can be a mobile phone, computer, server, air conditioner, or network device, etc.) to execute the methods described in each embodiment of the present application.
[0161] Obviously, the embodiments described above are only some embodiments of the present application, rather than all embodiments. The preferred embodiments of the present application are given in the accompanying drawings, but they do not limit the patent scope of the present application. The present application can be implemented in many different forms. On the contrary, the purpose of providing these embodiments is to make the understanding of the disclosure of the present application more thorough and comprehensive. Although the present application is described in detail with reference to the aforementioned embodiments, for those skilled in the art, it is still possible to modify the technical solutions recorded in the aforementioned specific implementation methods, or to perform equivalent replacement of some of the technical features therein. Any equivalent structure made using the contents of the specification and drawings of this application, directly or indirectly used in other related technical fields, is similarly within the scope of patent protection of this application.
Claims
1. A method for detecting abnormal users, characterized in that: The steps include: Obtain historical business data corresponding to the target business scenario; Performing encoding operation on the historical service data to obtain a service feature matrix; Constructing an initial linear regression model, inputting the business feature matrix into the initial linear regression model for model training operation, and optimizing the model training operation according to the LASSO algorithm to obtain a target linear regression model; Receiving an anomaly detection request sent by a user terminal, wherein the anomaly detection request includes at least a to-be-detected user identifier corresponding to the target service scenario; Calling a system database, and acquiring user service data corresponding to the user identifier to be detected in the system database; Performing encoding operation on the user service data to obtain a user feature matrix; The user feature matrix is input into the target linear regression model to perform an abnormality prediction operation to obtain an abnormality prediction result.
2. The abnormal user detection method according to claim 1, characterized in that: After the step of acquiring the historical service data corresponding to the target service scenario and before the step of encoding the historical service data to obtain the service feature matrix, the following steps are also included: Performing a data cleaning operation on the historical business data to obtain cleaned business data; Performing data integration operation on the cleaning service data to obtain an integrated data table; Performing a redundancy detection operation on the integrated data table to obtain a redundancy detection result; Performing a data elimination operation on the historical business data according to the redundant detection result to obtain pre-processed business data; The step of performing encoding operation on the historical service data to obtain a service feature matrix specifically comprises the following steps: An encoding operation is performed on the preprocessed service data to obtain the service feature matrix.
3. The abnormal user detection method according to claim 1, characterized in that: The objective function of the model training operation is expressed as: Among them, X ij represents the jth eigenvector of the i-th sample, y i represents the target variable of the i-th sample, β j represents the intercept term, β represents the coefficient vector, and λ represents the weight parameter of the L1 regularization term.
4. The abnormal user detection method according to claim 3, characterized in that: The step of constructing an initial linear regression model, inputting the business feature matrix into the initial linear regression model for model training, and optimizing the model training operation according to the LASSO algorithm specifically includes the following steps: Determining the optimal weight parameters of the objective function according to a feature cross-validation method; The coefficient vector β is updated according to the coordinate descent method until the difference between the current value of the objective function in the current iteration number and the objective function value obtained in the previous iteration meets a preset threshold, thereby obtaining the final coefficient vector β.
5. The abnormal user detection method according to claim 4, characterized in that: The step of updating the coefficient vector β according to the coordinate descent method specifically includes the following steps: The coefficient vector β is updated according to a soft threshold function, wherein the soft threshold function is expressed as: b j =sign(β j )·max(0,|β j |-l) Among them, λ represents the regularization parameter and sign represents the sign function.
6. The abnormal user detection method according to claim 1, characterized in that: After the step of acquiring the historical business data corresponding to the target business scenario, the following steps are also included: Identify highly relevant data corresponding to the target business scenario in the historical business data according to statistical analysis and visual data analysis; Acquire feedback information of the model training operation, and confirm key business features as inputs of the target linear regression model based on the feedback information and the highly correlated data.
7. An abnormal user detection device, characterized in that: include: A historical data acquisition module is used to acquire historical business data corresponding to the target business scenario; A first encoding module, used for encoding the historical service data to obtain a service feature matrix; A model training module is used to construct an initial linear regression model, input the business feature matrix into the initial linear regression model to perform a model training operation, and optimize the model training operation according to the LASSO algorithm to obtain a target linear regression model; A request acquisition module, configured to receive an anomaly detection request sent by a user terminal, wherein the anomaly detection request at least includes a to-be-detected user identifier corresponding to the target business scenario; A user data acquisition module, used to call a system database and acquire user service data corresponding to the user identifier to be detected in the system database; A second encoding module is used to perform an encoding operation on the user service data to obtain a user feature matrix; The abnormality prediction module is used to input the user feature matrix into the target linear regression model to perform an abnormality prediction operation and obtain an abnormality prediction result.
8. The abnormal user detection device according to claim 7, characterized in that: The device comprises: a data cleaning module, a data integration module, a redundancy detection module and a data elimination module, and the first encoding module comprises: a first encoding submodule, wherein: A data cleaning module, used to perform a data cleaning operation on the historical business data to obtain cleaned business data; A data integration module, used to perform data integration operations on the cleaning business data to obtain an integrated data table; A redundancy detection module, used for performing a redundancy detection operation on the integrated data table to obtain a redundancy detection result; A data elimination module, used for performing a data elimination operation on the historical business data according to the redundant detection result to obtain pre-processed business data; The first encoding submodule is used to perform encoding operation on the pre-processed service data to obtain the service feature matrix.
9. A computer device comprising a memory and a processor, characterized in that: The memory stores computer-readable instructions, and when the processor executes the computer-readable instructions, the steps of the abnormal user detection method according to any one of claims 1 to 6 are implemented.
10. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores computer-readable instructions, and when the computer-readable instructions are executed by a processor, the steps of the abnormal user detection method according to any one of claims 1 to 6 are implemented.
