User data query method, system and equipment for HBase and medium

Abstract

The invention provides a user data query method, system and device for HBase and a medium, and the method specifically comprises the steps: creating a corresponding row key for each user in the HBase, and creating a specific column for the corresponding row key according to behavior characteristic data of a plurality of users, the specific column being used for storing the behavior characteristic data of the corresponding user; when each row key is created, creating a reference column in which a preset value is stored for each row key; when a request for querying any row key is obtained, a query result is determined according to the reference column, and the query result comprises the user existence and behavior feature data, the user existence and no behavior feature data and the user absence. According to the method and the device, the accurate judgment on the HBase query result is realized, the query efficiency is improved, the resource allocation is optimized, and the overall service processing capability and the user experience are improved.

Description

Technical Field

[0001] The present invention relates to the field of big data technology, and in particular to a user data query method, system, device and medium for HBase. Background Art

[0002] In the field of big data processing and analysis, HBase, as an open source non-relational distributed database (NoSQL), is widely used to store large-scale data due to its high consistency and high performance. HBase supports random real-time read and write access to data tables, making data query and processing efficient and convenient. However, in actual business applications, especially in scenarios such as user portrait analysis and behavior feature tracking, HBase's query mechanism has certain limitations.

[0003] Traditional HBase query methods rely on row keys to locate specific data rows, and access specific cell values ​​through column names. However, when querying a column value, if the queried row key does not exist, or the row key exists but the column has no value, HBase will return a 404 status code. This unified error feedback mechanism has caused trouble at the business level, because it is impossible to accurately distinguish whether the row key itself does not exist, or the row key exists but the corresponding column value is empty.

[0004] In services such as user portrait analysis, it is crucial to accurately determine whether a user exists and whether the user has behavioral feature data. If the user does not exist, it may mean that a new user registration or data completion is required; if the user exists but has no behavioral feature data, it may mean that the user is a potential inactive user and a corresponding activation strategy needs to be adopted. However, due to the limitations of HBase's query mechanism, these important business judgments become difficult.

[0005] In addition, with the continuous growth of data volume and the increasing complexity of business needs, HBase performance monitoring and resource management are also facing challenges. How to monitor the operating status of HBase in real time, discover and handle exceptions in a timely manner, and how to dynamically adjust resource allocation according to business needs and traffic forecasts are all issues that need to be solved urgently. Summary of the invention

[0006] The purpose of the present invention is to provide a user data query method, system, device and medium for HBase, which can achieve accurate judgment of HBase query results, improve query efficiency, optimize resource allocation, and enhance overall business processing capabilities and user experience, so as to solve at least one of the above-mentioned prior art problems.

[0007] In a first aspect, the present invention provides a user data query method for HBase, the method specifically comprising:

[0008] Create a corresponding row key for each user in HBase, and create specific columns for the corresponding row keys according to the behavioral characteristic data of several users, wherein the specific columns are used to store the behavioral characteristic data of the corresponding users;

[0009] When each row key is created, a reference column is created for each row key that stores a preset value;

[0010] When a request to query any row key is obtained, a query result is determined according to the reference column, and the query result includes the user existing and having behavior feature data, the user existing but without behavior feature data, and the user not existing.

[0011] In a second aspect, the present invention provides a user data query system for HBase, the system specifically comprising:

[0012] The first query module is used to create a corresponding row key for each user in HBase, and to create specific columns for the corresponding row keys according to the behavior characteristic data of several users, wherein the specific columns are used to store the behavior characteristic data of the corresponding users;

[0013] A second query module is used to create a reference column storing a preset value for each row key when each row key is created;

[0014] The third query module is used to determine the query result according to the reference column when a request for querying any row key is obtained, and the query result includes the user existing and having behavior feature data, the user existing but without behavior feature data, and the user not existing.

[0015] In a third aspect, the present invention provides a computer device, comprising: a memory and a processor and a computer program stored in the memory, and when the computer program is executed on the processor, a user data query method for HBase as described in any one of the above methods is implemented.

[0016] In a fourth aspect, the present invention provides a computer-readable storage medium having a computer program stored thereon, wherein when the computer program is executed by a processor, the method for querying user data for HBase as described in any one of the above methods is implemented.

[0017] Compared with the prior art, the present invention has at least one of the following technical effects:

[0018] 1. The present invention realizes accurate judgment of HBase query results, improves query efficiency, optimizes resource allocation, and enhances overall business processing capabilities and user experience.

[0019] 2. The present invention introduces a reference column mechanism, which makes it possible to distinguish whether a user exists and whether the user has behavioral feature data during query, solving the problem that traditional HBase queries cannot clearly distinguish between the non-existence of row keys and empty column values.

[0020] 3. The present invention utilizes hash functions and Bloom filters to improve the efficiency of row key queries and reduce unnecessary disk I / O operations. Through the fast judgment of the Bloom filter, it can quickly determine whether the row key exists in HBase, further improving the query performance.

[0021] 4. The present invention sets cache according to the query frequency priority of row keys, improves the response speed of frequent row key queries, reduces query latency, reduces the access pressure of HBase through cache optimization, and improves the overall performance of the system.

[0022] 5. The present invention uses machine learning algorithms to analyze user access data, realizes the determination of dynamic permission levels, and improves the security and flexibility of the system.

[0023] 6. The present invention obtains and analyzes the historical traffic records of each query area on HBase, trains and generates a traffic prediction model, and dynamically adjusts resource allocation according to the prediction results of the traffic prediction model, thereby achieving rational resource utilization and load balancing, and improving the throughput and stability of the system.

[0024] 7. The present invention sets HBase abnormality monitoring indicators to monitor the real-time status of HBase in real time, can promptly discover and handle abnormal situations, and uses machine learning algorithms to analyze real-time status data, thereby improving the accuracy and timeliness of abnormality detection and reducing the risk of system failure.

[0025] 8. The present invention divides users into inactive users and active users according to the query results, which provides a basis for subsequent user portrait analysis and marketing strategy formulation. It generates a user presence status classifier through decision tree algorithm training, realizes the classification of active users' activity levels, and provides strong support for refined operations. BRIEF DESCRIPTION OF THE DRAWINGS

[0026] In order to more clearly illustrate the technical solutions in the embodiments of the present application, the drawings required for use in the embodiments will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present application. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative work.

[0027] Figure 1 It is a flowchart of a user data query method for HBase provided by the first embodiment of the present invention;

[0028] Figure 2 It is a flowchart of a user data query method for HBase provided by the second embodiment of the present invention;

[0029] Figure 3 It is a structural diagram of a user data query system for HBase provided by an embodiment of the present invention;

[0030] Figure 4 It is a structural schematic diagram of a computer device provided by an embodiment of the present invention. DETAILED DESCRIPTION

[0031] In the following description, specific details such as specific system structures, technologies, etc. are provided for the purpose of illustration rather than limitation, so as to provide a thorough understanding of the embodiments of the present application. However, it should be clear to those skilled in the art that the present application may also be implemented in other embodiments without these specific details. In other cases, detailed descriptions of well-known systems, devices, circuits, and methods are omitted to prevent unnecessary details from obstructing the description of the present application.

[0032] It should be understood that when used in the present specification and the appended claims, the term "comprising" indicates the presence of described features, wholes, steps, operations, elements and / or components, but does not exclude the presence or addition of one or more other features, wholes, steps, operations, elements, components and / or combinations thereof.

[0033] It should also be understood that the term “and / or” used in the specification and appended claims refers to any and all possible combinations of one or more of the associated listed items, and includes these combinations.

[0034] As used in the specification and appended claims of this application, the term "if" can be interpreted as "when" or "uponce" or "in response to determining" or "in response to detecting", depending on the context. Similarly, the phrase "if it is determined" or "if [described condition or event] is detected" can be interpreted as meaning "uponce it is determined" or "in response to determining" or "uponce [described condition or event] is detected" or "in response to detecting [described condition or event]", depending on the context.

[0035] In addition, in the description of the present application specification and the appended claims, the terms "first", "second", "third", etc. are only used to distinguish the descriptions and cannot be understood as indicating or implying relative importance.

[0036] References to "one embodiment" or "some embodiments" etc. described in the specification of this application mean that one or more embodiments of the present application include specific features, structures or characteristics described in conjunction with the embodiment. Therefore, the statements "in one embodiment", "in some embodiments", "in some other embodiments", "in some other embodiments", etc. that appear in different places in this specification do not necessarily refer to the same embodiment, but mean "one or more but not all embodiments", unless otherwise specifically emphasized in other ways. The terms "including", "comprising", "having" and their variations all mean "including but not limited to", unless otherwise specifically emphasized in other ways.

[0037] In the embodiment of the present application, the execution subject of the process includes a terminal device, which includes but is not limited to: a server, a computer, a smart phone, a tablet computer, and other devices capable of executing the method disclosed in the present application. Figure 1 A schematic flow chart of a method for querying user data in HBase disclosed in the first embodiment of the present invention is shown, and the details are as follows:

[0038] S101, creating a corresponding row key for each user in HBase, and creating specific columns for the corresponding row keys according to the behavior characteristic data of several users, wherein the specific columns are used to store the behavior characteristic data of the corresponding users.

[0039] In this embodiment, HBase Shell or HBase Java API is used to create an HBase table. The user ID (such as UUID) can be used as the row key in the HBase table to ensure global uniqueness. Based on the user's behavioral characteristic data, specific columns are created for the corresponding row keys, and the user's behavioral characteristic data such as the number of clicks, browsing time, and purchase amount are stored in the specific columns.

[0040] This embodiment uses the distributed architecture and flexible data model of HBase to create a corresponding row key for each user, and creates specific columns based on the user's behavioral feature data to store this data. This design not only improves the efficiency of data storage and access, but also has a flexible data model and real-time data analysis capabilities, which can meet the storage and analysis needs of user behavioral feature data in big data scenarios.

[0041] S102, when each row key is created, create a reference column storing a preset value for each row key;

[0042] When a request to query any row key is obtained, a query result is determined according to the reference column, and the query result includes the user existing and having behavior feature data, the user existing but without behavior feature data, and the user not existing.

[0043] In this embodiment, in HBase, whenever a new user (ie, a new row key) is created, the system will simultaneously create a reference column for the row key and set a preset value for it, such as using the registration time as the reference column and the specific time data as the preset value.

[0044] If the traditional query method is used, when a 404 status code is queried, the analysis system cannot clearly know whether it is because the user record (rowkey) itself does not exist, or the user does exist but has no specific behavior record (i.e., the value of the target column is empty). In this embodiment, when a request to query any row key is received, if the system receives a 404 status code, it means that the user record does not exist. If the query returns the registration time but no other behavior data, it means that the user exists but has no behavior record. If the query returns the registration time and other behavior data, it means that the user exists and has behavior records.

[0045] In this embodiment, by first checking the value of the reference column, the user's existence status can be quickly determined, avoiding unnecessary full table scans and deep queries, thereby improving query efficiency. For non-existent users, the reference column can quickly return results, avoiding invalid disk I / O operations and waste of computing resources. The value of the reference column can be used as a checkpoint for data consistency to ensure the accuracy and consistency of data during the query process.

[0046] As a brief description of the application of the above steps S101 to S102, Figure 2 As shown, first, the application client requests the hbase service to carry the reference column to query the specified rowkey target column. If hbase returns a 404 status code, it means that the rowkey record does not exist. Because if the rowkey exists, the reference column must have a value, and at least the reference column data will be returned instead of returning a 404 status code; if hbase returns the data of the reference column and the target column, it means that the rowkey record exists and the target column has a value; if hbase only returns the reference column data, it means that the rowkey record exists but the target column has no value. This method can also optimize the query efficiency of the system because it avoids unnecessary secondary queries. In addition, it can maintain data consistency while enhancing the system's ability to handle abnormal situations, thereby improving user experience and the quality of business decisions.

[0047] In some embodiments, the method further comprises:

[0048] A hash function is used to calculate the query hash value of each row key, and the corresponding index position in the bit array of the Bloom filter is determined according to the query hash value corresponding to each row key, and an identification value is added to each index position;

[0049] When a request to query any row key is obtained, the query result is determined according to whether there is an identification value at the index position corresponding to the current row key in the bit array of the Bloom filter.

[0050] In this embodiment, for a given row key set, a preset hash function is used to calculate the query hash value of each row key, and at least one corresponding index position is determined in the pre-built Bloom filter bit array according to the query hash value, and a preset identification value is added to each index position. When a request to query any row key is received, the target row key in the request is obtained, the target hash value of the target row key is calculated using the preset hash function, and at least one corresponding target index position is determined in the Bloom filter bit array according to the target hash value. It is judged whether the preset identification value exists in the target index position. If it exists, it is determined that the target row key exists in the original row key set, and the target row key is added to the candidate row key set; if it does not exist, it is determined that the target row key does not exist in the original row key set, and the target row key is added to the non-candidate row key set. For the candidate row key set, the complete data corresponding to each candidate row key in the original database is obtained as a candidate query result; for the non-candidate row key set, it is determined that the data corresponding to each non-candidate row key does not exist, and a null value is used as a non-candidate query result. The candidate query results are merged with the non-candidate query results to obtain complete query result data. According to the requirements of the original query request, the query result data is sorted, filtered, and other processes are performed to generate the final target query result. A machine learning algorithm is used to perform cluster analysis on the original row key set to obtain at least one row key subset. For each row key subset, a corresponding Bloom filter subset is constructed, and each Bloom filter subset is combined into a complete Bloom filter set for subsequent query request processing. A decision tree algorithm is used to analyze the characteristics of the query request. According to the number of requested row keys, query frequency, and other characteristics, the parameters of the Bloom filter, including the bit array size, the number of hash functions, etc., are dynamically adjusted to balance the query performance and storage space overhead, thereby improving the overall efficiency of the system.

[0051] Exemplarily, first, multiple hash functions such as MD5, SHA-1, etc. are selected to perform hash operations on each row key to obtain multiple hash values. Then, multiple corresponding index positions in the bit array of the Bloom filter are determined according to the hash value. For example, a 1 billion-bit bit array can be obtained by taking the hash value modulo 1 billion. Then, the binary bits of these index positions are set to 1 as an identifier of the existence of the row key. When querying a row key, the same hash function is used to calculate its hash value, and check whether the corresponding bit array index is 1. If so, the row key is considered to exist, otherwise it is considered not to exist. Although there is a certain misjudgment rate, the misjudgment rate can be controlled very low by reasonably setting the number of hash functions and the size of the bit array. For example, with 10 hash functions and a 1 billion-bit array, the misjudgment rate is less than one in ten thousand, and the actual storage cost is only about 1GB, which is much lower than the cost of storing all row keys. In summary, the Bloom filter uses extremely small space costs and extremely fast judgment speeds to achieve efficient query of specific row keys in massive data.

[0052] In some embodiments, the method further comprises:

[0053] Obtain row key query records in HBase, and determine the query frequency priority of each row key according to the row key query records;

[0054] A cache space is set, and query results corresponding to a plurality of row keys are pre-stored in the cache space according to the query frequency priority.

[0055] In this embodiment, query records of all row keys in HBase are obtained, and the query frequency of each row key is counted. The query priority of each row key is calculated according to the query frequency. The priority can be expressed by a numerical value, and the larger the numerical value, the higher the priority. A fixed-size cache space is set to store the row key query results. According to the query priority of the row key, the query results corresponding to the row keys with high query frequency are pre-loaded into the cache space. When a query request arrives, first determine whether the requested row key exists in the cache. If so, the result is returned directly from the cache without accessing HBase. If the cache misses, query the record from HBase, add the query result to the cache, and update the query frequency and priority of the row key. Clean the cache space regularly, remove the cache data corresponding to the row keys with lower query frequency, and make room for new high-frequency query row keys. At the same time, update the query priority of the row key according to the latest query frequency.

[0056] Exemplarily, when obtaining row key query records in HBase, the system will count the number of queries for each row key in the last hour, and calculate the query frequency priority based on the number of queries. For example, the priority of a row key with a query count greater than 100 is 1, the priority of a row key with a query count between 50 and 100 is 2, and the priority of a row key with a query count less than 50 is 3. The system will pre-set a 100MB cache space, and store the query results corresponding to the row key in the cache in order from high to low priority until the cache space occupancy reaches 80%. When there is a new query request, the system first checks whether the query result for the row key already exists in the cache. If so, the cache content is directly returned; if not, it is queried in HBase and the query result is added to the cache. When the cache space occupancy reaches 90%, the system will automatically delete the cache content with the lowest priority until the cache space occupancy drops below 80%. In this way, the system can efficiently respond to high-frequency query requests and improve overall query performance.

[0057] In some embodiments, the method further comprises:

[0058] Obtain user access data of each HBase client, wherein the user access data includes user operation frequency, user access path, user operation type, user current location, and user device type;

[0059] Extracting features from the user access data to obtain a user feature vector, wherein the user feature vector includes a statistical value of user operation frequency, a frequent item set of the user access path, a classification label of the user operation type, the latitude and longitude of the user's current location or an area code, and a classification label of the user device type;

[0060] The user feature vector is analyzed using a machine learning algorithm to determine the dynamic permission level of each HBase client in each query area on HBase.

[0061] In this embodiment, the user access data of the HBase client is obtained through log collection or point-of-use technology, including information such as operation frequency, access path, operation type, current location and device type. The obtained user access data is preprocessed, invalid or abnormal data is cleaned, and the data is converted into a format suitable for feature extraction. For different types of user access data, corresponding feature extraction methods are adopted. For the operation frequency, the number and frequency statistics of various operations are calculated; for the access path, the common access path pattern is extracted using the frequent item set mining algorithm; for the operation type and device type, classification coding processing is performed; for the current location, the longitude and latitude coordinates or area codes are extracted. The various features extracted are combined into a user feature vector, and each user corresponds to a feature vector, which contains operation frequency statistics, access path frequent item sets, operation type classification labels, location longitude and latitude or area codes, and device type classification labels. Select a suitable machine learning algorithm, such as a decision tree, support vector machine or neural network, to train and classify the user feature vector. According to business needs, different permission levels are set, such as read-write permissions, read-only permissions, and no permissions. After the machine learning model is trained, it is deployed to the production environment. When an HBase client initiates a query request, the user access data of the client is obtained, the feature vector is extracted, and input into the trained machine learning model for prediction to obtain the dynamic permission level of the client in the query area. According to the predicted permission level, the query request of the HBase client is controlled and processed accordingly. If the permission is allowed, the query is executed normally; if the permission is insufficient, the query request is rejected or partial results are returned. At the same time, the query is logged for subsequent model optimization and adjustment.

[0062] Exemplarily, the system will obtain user access data of each HBase client in real time, including information such as the user's operation frequency, access path, operation type, current location, and device type on HBase. For example, the system records that a user has performed 50 query operations on HBase in the past hour, mainly accessing the two paths " / user / profile" and " / order / history", and the operation type is mainly SELECT. The user is currently located in xx District, xx City, and uses an Android phone. The system will extract features from these user access data and generate corresponding user feature vectors. For example, the user's operation frequency statistics are 50, the access path frequent item set is {" / user / profile", " / order / history"}, the operation type label is SELECT, the location longitude and latitude are (xxx, xxxx), and the device type label is Android. Then, the system uses machine learning algorithms such as SVM to train and classify user feature vectors, and dynamically determine the user's permission level in different query areas of HBase based on factors such as user behavior patterns, location characteristics, and device attributes. For example, for clients that frequently query user sensitive data, the system may set their query permissions on the " / user / profile" path to read-only, while for clients that occasionally query non-sensitive data, the system grants them read and write permissions. Through this dynamic permission control method based on user access characteristics, HBase can more finely manage the data access behavior of different clients, improving system security while taking into account ease of use.

[0063] In some embodiments, the method further comprises:

[0064] Get the historical traffic records of each query area on HBase;

[0065] Using a machine learning algorithm to analyze the historical process records and the user access data, and train and generate a traffic prediction model;

[0066] According to the prediction result of the traffic prediction model, the resource allocation of each query area is dynamically adjusted.

[0067] In this embodiment, the historical traffic data of each query area stored in HBase is obtained, and the historical traffic data and user access data are input into the machine learning model for training to obtain a traffic prediction model; according to the traffic prediction model, the traffic situation of each query area in the future period is predicted, and the prediction result is used as the basis for dynamic resource allocation; if the predicted traffic is higher than the threshold, it is judged as a high-traffic area, and more computing resources and storage resources are allocated to the area from the resource pool; if the predicted traffic is lower than the threshold, it is judged as a low-traffic area, and part of the computing resources and storage resources occupied by the area are recovered and put back into the resource pool; by real-time monitoring of the actual traffic situation of each query area, the actual traffic data is compared with the predicted traffic data, and the prediction deviation is calculated; if the prediction deviation exceeds the set threshold, the model retraining is triggered, and the latest historical traffic data and user access data are used to update the traffic prediction model to improve the prediction accuracy; the above process is repeated continuously, and resource allocation is dynamically adjusted according to the traffic prediction, while ensuring the service quality of the high-traffic area, the overall resource utilization is improved.

[0068] Exemplarily, first, the system will obtain the traffic records of each query area in the past period of time from HBase, including the number of queries in each time period, query response time and other indicators. Then, these historical traffic records are used together with the current user access data (such as user operation frequency, access path, etc.) as training data, and machine learning algorithms (such as LSTM neural networks) are used to analyze and model them to generate a traffic prediction model. The model can predict the traffic change trend of each query area in the future period of time based on the user's access pattern and historical traffic conditions. For example, through analysis, it is found that a query area has a high traffic volume between 9 and 10 am every day, with an average of 1,000 queries per minute, and the users are mainly from mobile devices. The model will predict that the traffic volume of this area in this time period will also remain at a high level in the future.

[0069] In some embodiments, the method further comprises:

[0070] Set HBase abnormal monitoring indicators, which include response time indicators of HBase operation types, read and write speed indicators of each row key and its corresponding first specific column and second specific column;

[0071] Use performance monitoring tools to monitor the real-time status of HBase and obtain real-time status data;

[0072] Based on the HBase abnormal monitoring index, the real-time status data is analyzed using a machine learning algorithm to obtain an abnormal analysis result, and it is determined whether to trigger an alarm process according to the abnormal analysis result.

[0073] In this embodiment, according to the preset HBase abnormal monitoring indicators, a performance monitoring tool is used to obtain the operation response time, row key read and write speed and other status data of HBase in real time. The acquired HBase real-time status data is input into the machine learning model, and the data is analyzed for abnormality using algorithms such as support vector machine and random forest. According to the abnormal analysis results of the machine learning algorithm, it is determined whether the real-time status of HBase exceeds the preset normal threshold range. If the real-time status of HBase exceeds the normal threshold range, the alarm process is triggered and abnormal alarm information is generated. The abnormal alarm information is sent to the operation and maintenance personnel, and the relevant personnel are notified to handle it by SMS, email, etc. The operation and maintenance personnel diagnose and repair HBase according to the abnormal alarm information to ensure that HBase returns to normal operation. The abnormal state and processing results of HBase are recorded in the log for subsequent abnormal cause analysis and optimization and improvement.

[0074] Exemplarily, the HBase abnormal monitoring indicator settings include the response time indicator of the HBase operation type, and the read and write speed indicators of each row key and its corresponding first specific column and second specific column. For example, for the Get operation, the response time threshold is set to 50ms; for the Put operation, the write speed thresholds of the "info:age" column and the "info:gender" column of the row key "user001" are set to 1000 times / s and 800 times / s respectively. Use performance monitoring tools such as Ganglia to collect CPU utilization, memory usage, disk I / O and other indicator data of the HBase cluster in real time, once every 5 seconds. Then use the random forest algorithm to train the collected real-time status data to obtain an abnormality detection model. When the average response time of the Get operation is monitored to exceed 60ms, or the write speed of the "info:age" column of the "user001" row key is less than 800 times / s, the abnormality detection model is determined to be abnormal and the alarm process is triggered. During the alarm process, the system automatically sends alarm text messages and emails to the on-duty personnel, and attaches detailed information about the abnormal indicators, such as the time of the abnormality, the name of the abnormal indicator, the current value of the abnormal indicator, etc., to help the on-duty personnel quickly locate the problem. At the same time, the system automatically records the abnormal information in the log file for subsequent problem tracking and analysis.

[0075] In some embodiments, the method further comprises:

[0076] According to the query result, the user without behavior characteristic data and the user with behavior characteristic data are respectively determined as inactive users and active users;

[0077] The basic information, behavioral characteristics and preference labels of the active users are obtained, and a user presence status classifier is generated by training through a decision tree algorithm according to the basic information, the behavioral characteristics and the preference labels. The user presence status classifier is used to classify the activity level of the active users.

[0078] In this embodiment, according to the preset rules, the user's behavioral feature data is queried from the user database, and the query results are divided into two categories: active users and inactive users according to whether there is behavioral feature data. The basic information, behavioral characteristics, preference tags and other related attribute data of the active users are obtained, and the attribute data is preprocessed and converted into a data format acceptable to the decision tree algorithm. According to the attribute data of the active users, the decision tree algorithm is used for training to obtain a classifier model for judging the user's presence status. The attribute data of the active users is predicted and classified using the generated user presence status classifier to determine the activity level of each active user. If the user's activity level is lower than the preset threshold, the user is marked as a user with reduced activity and included in the list of key users. For users with reduced activity, the trend of changes in their behavioral characteristics is analyzed, the key factors leading to the decline in activity are identified, and targeted user activation strategies are formulated. The subsequent behavioral changes of users with reduced activity are continuously tracked, the effects of different activation strategies are evaluated through A / B testing and other methods, and the activation scheme is dynamically adjusted and optimized to improve user activity.

[0079] Exemplarily, according to the query results, users without behavioral feature data and users with behavioral feature data are respectively determined as inactive users and active users. For active users, their basic information, such as age, gender, region, etc., as well as behavioral characteristics, such as login frequency, access time, number of interactions, etc., and preference tags, such as favorite product categories, price ranges, etc., are obtained. The decision tree algorithm is used to train with basic information, behavioral characteristics and preference tags as feature variables and activity level as target variable. For example, active users are divided into three levels according to the average monthly consumption amount: 0-100 yuan, 100-500 yuan, and more than 500 yuan, corresponding to low, medium, and high activity, respectively. The optimal feature is selected by the information gain ratio, and the decision tree is recursively constructed until all samples belong to the same category or the feature is exhausted. The final generated decision tree can be used as a user presence state classifier to determine the activity level of new users. Through this classifier, differentiated operation strategies can be adopted for users with different activity levels to improve user retention and activity.

[0080] Reference Figure 3 An embodiment of the present invention provides a user data query system 3 for HBase, and the system 3 specifically includes:

[0081] The first query module 301 is used to create a corresponding row key for each user in HBase, and to create specific columns for the corresponding row keys according to the behavior characteristic data of several users, wherein the specific columns are used to store the behavior characteristic data of the corresponding users;

[0082] A second query module 302 is used to create a reference column storing a preset value for each row key when each row key is created;

[0083] The third query module 303 is used to determine the query result according to the reference column when a request for querying any row key is obtained, and the query result includes the user existing and having behavior feature data, the user existing but without behavior feature data, and the user not existing.

[0084] It is understandable that if Figure 1 The contents of the user data query method embodiment for HBase shown in the figure are applicable to the user data query system embodiment for HBase. The functions specifically implemented by the user data query system embodiment for HBase are the same as those in the figure. Figure 1 The user data query method embodiment for HBase shown in FIG. 1 is the same as that shown in FIG. 1 , and the beneficial effects achieved are the same as those of FIG. Figure 1 The beneficial effects achieved by the user data query method embodiment for HBase shown are also the same.

[0085] It should be noted that the information interaction, execution process and other contents between the above-mentioned systems are based on the same concept as the embodiment of the method of the present invention. Their specific functions and technical effects can be found in the method embodiment part and will not be repeated here.

[0086] Those skilled in the art can clearly understand that, for the convenience and simplicity of description, only the division of the above-mentioned functional units and modules is used as an example for illustration. In practical applications, the above-mentioned function allocation can be completed by different functional units and modules as needed, that is, the internal structure of the system can be divided into different functional units or modules to complete all or part of the functions described above. The functional units and modules in the embodiment can be integrated in a processing unit, or each unit can exist physically separately, or two or more units can be integrated in one unit. The above-mentioned integrated unit can be implemented in the form of hardware or in the form of software functional units. In addition, the specific names of the functional units and modules are only for the convenience of distinguishing each other, and are not used to limit the scope of protection of this application. The specific working process of the units and modules in the above-mentioned system can refer to the corresponding process in the aforementioned method embodiment, which will not be repeated here.

[0087] Reference Figure 4The embodiment of the present invention further provides a computer device 4, comprising: a memory 402 and a processor 401 and a computer program 403 stored in the memory 402. When the computer program 403 is executed on the processor 401, the user data query method for HBase described in any one of the above methods is implemented.

[0088] The computer device 4 may be a computing device such as a desktop computer, a notebook, a PDA, or a cloud server. The computer device 4 may include, but is not limited to, a processor 401 and a memory 402. Those skilled in the art will appreciate that Figure 4 It is only an example of the computer device 4 and does not constitute a limitation on the computer device 4. It may include more or fewer components than shown in the figure, or a combination of certain components, or different components. For example, it may also include input and output devices, network access devices, etc.

[0089] The processor 401 may be a central processing unit (CPU), or other general-purpose processors, digital signal processors (DSP), application-specific integrated circuits (ASIC), field-programmable gate arrays (FPGA) or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc. A general-purpose processor may be a microprocessor or any conventional processor, etc.

[0090] In some embodiments, the memory 402 may be an internal storage unit of the computer device 4, such as a hard disk or memory of the computer device 4. In other embodiments, the memory 402 may also be an external storage device of the computer device 4, such as a plug-in hard disk, a smart memory card (Smart Media Card, SMC), a secure digital (Secure Digital, SD) card, a flash card (Flash Card), etc. equipped on the computer device 4. Further, the memory 402 may also include both an internal storage unit and an external storage device of the computer device 4. The memory 402 is used to store an operating system, an application program, a boot loader (Boot Loader), data, and other programs, such as the program code of the computer program, etc. The memory 402 may also be used to temporarily store data that has been output or is to be output.

[0091] An embodiment of the present invention further provides a computer-readable storage medium on which a computer program is stored. When the computer program is executed by a processor, the user data query method for HBase as described in any one of the above methods is implemented.

[0092] In this embodiment, if the integrated unit is implemented in the form of a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the present application implements all or part of the processes in the above-mentioned embodiment method, which can be completed by instructing the relevant hardware through a computer program, and the computer program can be stored in a computer-readable storage medium. When the computer program is executed by the processor, the steps of the above-mentioned various method embodiments can be implemented. Among them, the computer program includes computer program code, and the computer program code can be in source code form, object code form, executable file or some intermediate form. The computer-readable medium can at least include: any entity or device that can carry the computer program code to the camera / terminal device, recording medium, computer memory, read-only memory (ROM, Read-Only Memory), random access memory (RAM, Random Access Memory), electric carrier signal, telecommunication signal and software distribution medium. For example, a USB flash drive, a mobile hard disk, a disk or an optical disk. In some jurisdictions, according to legislation and patent practice, computer-readable media cannot be electric carrier signals and telecommunication signals.

[0093] In the above embodiments, the description of each embodiment has its own emphasis. For parts that are not described or recorded in detail in a certain embodiment, reference can be made to the relevant descriptions of other embodiments.

[0094] Those of ordinary skill in the art will appreciate that the units and algorithm steps of each example described in conjunction with the embodiments disclosed herein can be implemented in electronic hardware, or a combination of computer software and electronic hardware. Whether these functions are performed in hardware or software depends on the specific application and design constraints of the technical solution. Professional and technical personnel can use different methods to implement the described functions for each specific application, but such implementation should not be considered to be beyond the scope of this application.

[0095] In the embodiments disclosed in the present application, it should be understood that the disclosed devices / terminal equipment and methods can be implemented in other ways. For example, the device / terminal equipment embodiments described above are only schematic. For example, the division of the modules or units is only a logical function division. There may be other division methods in actual implementation, such as multiple units or components can be combined or integrated into another system, or some features can be ignored or not executed. Another point is that the mutual coupling or direct coupling or communication connection shown or discussed can be through some interfaces, indirect coupling or communication connection of devices or units, which can be electrical, mechanical or other forms.

[0096] The units described as separate components may or may not be physically separated, and the components shown as units may or may not be physical units, that is, they may be located in one place or distributed on multiple network units. Some or all of the units may be selected according to actual needs to achieve the purpose of the solution of this embodiment.

Claims

1. A user data query method for HBase, characterized in that: The method specifically comprises: Create a corresponding row key for each user in HBase, and create specific columns for the corresponding row keys according to the behavioral characteristic data of several users, wherein the specific columns are used to store the behavioral characteristic data of the corresponding users; When each row key is created, a reference column is created for each row key that stores a preset value; When a request to query any row key is obtained, a query result is determined according to the reference column, and the query result includes the user existing and having behavior feature data, the user existing but without behavior feature data, and the user not existing.

2. The method according to claim 1, characterized in that The method further comprises: A hash function is used to calculate the query hash value of each row key, and the corresponding index position in the bit array of the Bloom filter is determined according to the query hash value corresponding to each row key, and an identification value is added to each index position; When a request to query any row key is obtained, the query result is determined according to whether there is an identification value at the index position corresponding to the current row key in the bit array of the Bloom filter.

3. The method according to claim 2, characterized in that The method further comprises: Obtain row key query records in HBase, and determine the query frequency priority of each row key according to the row key query records; A cache space is set, and query results corresponding to a plurality of row keys are pre-stored in the cache space according to the query frequency priority.

4. The method according to claim 3, characterized in that The method further comprises: Obtain user access data of each HBase client, wherein the user access data includes user operation frequency, user access path, user operation type, user current location, and user device type; Extracting features from the user access data to obtain a user feature vector, wherein the user feature vector includes a statistical value of user operation frequency, a frequent item set of the user access path, a classification label of the user operation type, the latitude and longitude of the user's current location or an area code, and a classification label of the user device type; The user feature vector is analyzed using a machine learning algorithm to determine the dynamic permission level of each HBase client in each query area on HBase.

5. The method according to claim 4, characterized in that The method further comprises: Get the historical traffic records of each query area on HBase; Using a machine learning algorithm to analyze the historical process records and the user access data, and train and generate a traffic prediction model; According to the prediction result of the traffic prediction model, the resource allocation of each query area is dynamically adjusted.

6. The method according to any one of claims 1 to 5, characterized in that The method further comprises: Set HBase abnormal monitoring indicators, which include response time indicators of HBase operation types, read and write speed indicators of each row key and its corresponding first specific column and second specific column; Use performance monitoring tools to monitor the real-time status of HBase and obtain real-time status data; Based on the HBase abnormal monitoring index, the real-time status data is analyzed using a machine learning algorithm to obtain an abnormal analysis result, and it is determined whether to trigger an alarm process according to the abnormal analysis result.

7. The method according to any one of claims 1 to 5, characterized in that The method further comprises: According to the query result, the user without behavior characteristic data and the user with behavior characteristic data are respectively determined as inactive users and active users; The basic information, behavioral characteristics and preference labels of the active users are obtained, and a user presence status classifier is generated by training through a decision tree algorithm according to the basic information, the behavioral characteristics and the preference labels. The user presence status classifier is used to classify the activity level of the active users.

8. A user data query system for HBase, characterized in that: The system specifically comprises: The first query module is used to create a corresponding row key for each user in HBase, and to create specific columns for the corresponding row keys according to the behavior characteristic data of several users, wherein the specific columns are used to store the behavior characteristic data of the corresponding users; A second query module is used to create a reference column storing a preset value for each row key when each row key is created; The third query module is used to determine the query result according to the reference column when a request for querying any row key is obtained, and the query result includes the user existing and having behavior feature data, the user existing but without behavior feature data, and the user not existing.

9. A computer device, characterized in that: include: A memory, a processor, and a computer program stored in the memory. When the computer program is executed on the processor, the user data query method for HBase as described in any one of claims 1 to 7 is implemented.

10. A computer-readable storage medium, characterized in that: A computer program is stored thereon, and when the computer program is executed by a processor, the user data query method for HBase as claimed in any one of claims 1 to 7 is implemented.

Images