A bitmap-based in-cabin status identification method, device and medium

Abstract

The application relates to the technical field of APP data extraction, in particular to a bitmap-based in-app state identification method, device and medium, which comprises the following steps: based on a task type, target APP behavior index sets are screened out; original user APP behavior data sets to be analyzed are divided according to user identifiers and distributed to each computing node in a preset distributed cluster for local pre-aggregation; aggregated data of user APP behavior is obtained by summarization; bitmap difference set operation is performed according to the aggregated data; APP in-app state user sets at the current moment are obtained by combining APP recent active data; a bitmap structure template is constructed to update the APP in-app state when new user APP behavior data sets are received; and the application significantly improves the data extraction efficiency of massive user APP behavior data processing, the cluster resource utilization rate and the accuracy of in-app state identification.

Description

Technical Field

[0001] This invention relates to the field of APP data extraction technology, and in particular to a bitmap-based method, device and medium for identifying the installation status. Background Technology

[0002] With the rapid development of the mobile internet, the number of app users has exploded, generating billions or even hundreds of billions of user behavior logs daily. How to quickly and accurately extract core data such as currently installed users and active users from this massive amount of user behavior data has become a pressing technical problem for app stores, advertising platforms, and data service providers.

[0003] Current technologies typically employ distributed databases for grouping and aggregation or offline batch computation when handling massive user datasets. However, when the user base reaches hundreds of millions, traditional methods require scanning all detailed data or performing numerous relational queries, resulting in poor data extraction efficiency and high computational resource consumption. While introducing bitmap technology can improve the performance of a single full computation, each incremental update requires recalculation, leading to significant redundant overhead and low computational efficiency. Furthermore, bitmap data can only represent a binary "present" or "absent" state, lacking an effective means to determine the presence or absence of apps in complex business scenarios such as uninstallation and reinstallation, resulting in poor accuracy in identifying the app's installation status. Summary of the Invention

[0004] To address the aforementioned technical issues, this invention provides a bitmap-based method, device, and medium for identifying the in-installation status. By pre-aggregating and summarizing user APP behavior data according to dynamically selected indicators and storing it in a bitmap data structure, this invention significantly improves the efficiency of processing massive amounts of user APP behavior data, the utilization rate of cluster resources, and the accuracy of identifying the in-installation status.

[0005] According to a first aspect of the present invention, a bitmap-based method for identifying the in-installation state is provided, comprising the following steps: S1. Based on the data extraction task type, a target APP behavior indicator set is selected from several preset APP behavior indicators. The target APP behavior indicator set includes at least APP installation data, APP uninstallation data, and APP recent active data. The data extraction task type is either a real-time data extraction task or a batch data extraction task.

[0006] S2, the original user APP behavior dataset to be analyzed is divided into data according to user identifiers and distributed to each computing node in the preset distributed cluster, so that each computing node performs local pre-aggregation according to the target APP behavior index set and summarizes the aggregated data of user APP behavior.

[0007] S3, convert the APP installation data and APP uninstallation data in the aggregated data into bitmap data structures respectively, and obtain the set of users in the APP installation status at the current moment by bitmap difference operation and combining it with the APP's most recently active data.

[0008] S4. Based on the current set of users in the APP installation status and the mapping relationship between user identifiers and bit indexes, generate a bitmap structure template and store it in a preset columnar database.

[0009] S5, upon receiving a new user APP behavior dataset, reads the bitmap structure template and calculates the bitmap change amount, and synchronously updates the corresponding APP installation status change results to the preset columnar database.

[0010] According to a second aspect of the present invention, a non-transitory computer-readable storage medium is provided, wherein at least one instruction or at least one program is stored therein, the at least one instruction or the at least one program being loaded and executed by a processor to implement the bitmap-based on-load state identification method described above.

[0011] According to a third aspect of the present invention, an electronic device is provided, including a processor and the aforementioned non-transitory computer-readable storage medium.

[0012] The present invention has at least the following beneficial effects: This invention provides a bitmap-based method for identifying the in-installation status. First, based on the data extraction task type, a target APP behavior indicator set is selected, allowing the indicator selection to be dynamically adjusted according to business scenarios, improving business adaptability. Then, the original user APP behavior dataset to be analyzed is partitioned according to user identifiers and distributed to various computing nodes in a pre-defined distributed cluster for local pre-aggregation, summarizing the aggregated user APP behavior data. This significantly reduces the amount of data entering the core computing stage, substantially improving the data extraction efficiency and cluster resource utilization of massive user APP behavior data processing. Finally, bitmap difference operations are performed on the aggregated data, combined with the APP's most recently active data. By acquiring the current set of users with the app's installed status, and leveraging the parallel computing advantages of bitmap operations, the computation time for hundreds of millions of users can be compressed from minutes to milliseconds. Furthermore, by combining active data with secondary judgments on users who have uninstalled and reinstalled the app, the accuracy of status recognition in complex uninstallation and reinstallation scenarios is significantly improved while maintaining extreme computational performance. Finally, a bitmap structure template is constructed to update the app's installed status upon receiving a new user app behavior dataset. By reusing pre-generated Bitmap templates, bit status change calculations are performed only for existing users, significantly improving the efficiency of incremental computation while ensuring high accuracy in installed status recognition. Attached Figure Description

[0013] To more clearly illustrate the technical solutions in the embodiments of the present invention, the accompanying drawings used in the description of the embodiments will be briefly introduced below. Obviously, the accompanying drawings described below are only some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0014] Figure 1 A flowchart of a bitmap-based in-installation status identification method provided in an embodiment of the present invention. Detailed Implementation

[0015] The technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0016] This invention provides a bitmap-based method for identifying the in-installation state, such as... Figure 1 As shown, the method includes the following steps: S1. Based on the data extraction task type, a target APP behavior indicator set is selected from several preset APP behavior indicators; the target APP behavior indicator set includes at least APP installation data, APP uninstallation data, and APP recent active data. Specifically, the data extraction task type is either a real-time data extraction task or a batch data extraction task. In specific implementation scenarios, real-time data extraction tasks should prioritize the simplicity of the metrics, while batch data extraction tasks should prioritize the completeness of the metrics. The weights of the metrics are set accordingly.

[0017] The above-mentioned approach, by dynamically selecting indicators based on the data extraction task type, resolves the contradiction between excessive redundancy in real-time queries and insufficient information in batch analysis for fixed dimensions. While ensuring calculation accuracy, it significantly reduces the range of data fields that need to be processed in subsequent steps, thereby improving the overall calculation efficiency.

[0018] Specifically, step S1 includes the following steps: S101, based on the information gain value of each preset APP behavior indicator obtained in advance, calculate the importance of each preset APP behavior indicator; wherein, the importance of any preset APP behavior indicator meets the following condition: Among them, W i To determine the importance of the i-th preset APP behavior metric, G i and G k B represents the information gain values ​​of the i-th and k-th preset APP behavior metrics, respectively. i and B k These are the preset weights of the i-th and k-th preset APP behavior metrics under the corresponding data extraction task type, respectively, where n is the total number of preset APP behavior metrics, and α and β are the first and second preset weight coefficients, respectively. In specific implementations, different metric weights are set according to different business scenarios. For example, for real-time data extraction tasks, the APP's recent active data is given a higher metric weight, while for batch data extraction tasks, the APP uninstallation data is given a higher metric weight.

[0019] It's important to note that information gain measures how much the uncertainty about the target variable—whether the app is currently installed—is reduced after adding a certain metric. A higher information gain indicates that the metric is more important for determining the target variable. For example, for the metric to be analyzed, "whether the app has been active in the last 7 days," the initial information entropy needs to be calculated first. If out of 1 million users, 300,000 are currently installing the app and 700,000 are not, then p(yes) = 0.3 and p(no) = 0.7. Based on the indicators to be analyzed, users are divided into active and inactive groups. The proportion of users in the installed and inactive states within each group is calculated. Then, the information entropy of the active and inactive groups is calculated separately, thereby determining the conditional entropy. The difference between the initial information entropy and the conditional entropy is used as the initial information gain. Then, the final information gain value is obtained by normalizing the information gain of each index.

[0020] S102, preset APP behavior indicators with an importance level greater than a preset threshold are selected as target APP behavior indicators. Those skilled in the art can set the preset threshold according to actual needs, which will not be elaborated here.

[0021] The above-mentioned method uses information gain value to quantify the ability of indicators to distinguish the installation status of an APP, and combines it with indicator weights applicable to the corresponding data extraction task type. This allows the indicator selection to be dynamically adjusted according to business scenarios while maintaining the objective quantification of data characteristics, thereby improving business adaptability.

[0022] S2 divides the original user APP behavior dataset to be analyzed into data according to user identifiers and distributes it to each computing node in the preset distributed cluster, so that each computing node can perform local pre-aggregation based on the target APP behavior index set and summarize to obtain aggregated data of user APP behavior; it can be understood that the original user APP behavior dataset includes APP behavior data of several users, such as metadata such as installation events, uninstallation events, and usage information.

[0023] Specifically, the user identifier is a unique identifier that represents a user, and it is represented by a hash value.

[0024] As described above, through data partitioning and local pre-aggregation operations, the massive detailed data in the original user APP behavior dataset is compressed into summary records that are statistically analyzed only according to the target APP behavior indicator set. This significantly reduces the amount of data entering the core computing stage, avoiding the network overhead caused by cross-node transmission of the original data, and providing a lightweight and standardized input for the subsequent construction of the bitmap structure template.

[0025] Furthermore, the step of dividing the original user APP behavior dataset to be analyzed according to user identifiers and distributing it to various computing nodes in a preset distributed cluster includes the following steps: S201: The hash value space corresponding to the user identifier is evenly divided into several logical partitions, and each logical partition is initially mapped to each computing node using a hash modulo method, resulting in several logical partitions corresponding to each computing node. This can be understood as: evenly dividing the hash value space 0-2^(32-1) into several consecutive partition intervals, with each interval corresponding to one logical partition. In specific implementations, the number of logical partitions is much greater than the number of computing nodes.

[0026] S202 collects the node load of each computing node in real time and obtains the real-time load normalized value for each computing node through normalization processing. For example, the node load can be the CPU utilization of the node or the current task queue length of the node.

[0027] S203: For any computing node, when the normalized value of the real-time load corresponding to the computing node exceeds a preset load threshold, the logical shards corresponding to the computing node are migrated to the computing node with the lowest current load according to a preset migration step size, so as to realize the allocation of the original user APP behavior dataset. Those skilled in the art can set the preset load threshold according to actual needs, which will not be elaborated here.

[0028] As described above, the hash modulo method for sharding ensures a uniform initial distribution of data. Combined with real-time node load awareness and dynamic migration of logical shards, the cluster can automatically balance the computing pressure of each computing node. Without interrupting service, it effectively solves the problems of data skew and node overload, and significantly improves the overall throughput efficiency and response stability of distributed aggregation processing.

[0029] Furthermore, step S2 also includes the following steps: S210, for any target APP behavior indicator in the target APP behavior indicator set, calculate the pre-aggregation result of each computing node and the summarization error of the original user APP behavior dataset on the target APP behavior indicator.

[0030] For example, if the total number of users with apps installed is 20,000, and the total number of users with apps installed is 21,000, then the aggregation error E = |2.1 - 2| / 2 × 100% = 5%.

[0031] S220, when the aggregation error exceeds a preset error threshold, the local pre-aggregation operation is re-executed on the corresponding computing node. Those skilled in the art can set the preset error threshold according to actual needs.

[0032] As described above, by adding an error verification step after pre-aggregation, it is ensured that the compressed aggregated data of each computing node is consistent with the original data in terms of the target APP behavior indicators, avoiding deviations in calculation results due to aggregation logic errors or data loss. Furthermore, when the error exceeds the standard, recalculation is automatically triggered, ensuring the reliability of data quality for subsequent status determination and change updates based on Bitmap data from the source.

[0033] S3, convert the APP installation data and APP uninstallation data in the aggregated data into bitmap data structures respectively, and obtain the set of users in the APP installation status at the current moment by bitmap difference operation and combining the APP's most recently active data; it can be understood that: when performing bitmap data conversion, each bitmap bit corresponds to a unique user identifier.

[0034] In one embodiment, before converting the app installation data and app uninstallation data in the aggregated data into bitmap data structures respectively, the method further includes: S310, identify outliers and outlier types in the aggregated data, and map the degree of anomaly corresponding to the outlier type of each outlier; the degree of anomaly is either mild or severe. For example, mild outlier types include null values ​​and format errors, while severe outlier types include data tampering and logical conflicts.

[0035] S320: For slightly anomalous data, mean-filling method is used for correction.

[0036] S330 removes severely abnormal data and records the operation log.

[0037] The above-mentioned method involves hierarchical identification and differentiated processing of abnormal data to eliminate data noise while retaining valid information. Mild anomalies are repaired by mean filling to avoid sample loss caused by simple rejection, while severe anomalies are directly rejected and traced to prevent erroneous data from affecting subsequent Bitmap calculations and APP installation status determination, thereby ensuring the accuracy of APP installation status calculation.

[0038] Specifically, step S3 includes the following steps: S301, convert the APP installation data and APP uninstallation data in the aggregated data into bitmap installation data set and bitmap uninstallation data set respectively, and obtain a number of APP users in the installed state without uninstallation records through the difference operation; it can be understood as: through the difference operation of bitmap, the calculation of APP installation data minus APP uninstallation data is realized, and the bitmap set of APPs in the confirmed installed state without uninstallation records is directly obtained.

[0039] S302, for users with installation and uninstallation records, if it is determined that the user's most recent active time for the app is later than the app's uninstallation time, then the user is determined to be a user with the app currently installed. Conversely, the app's activity level within the user's current time window is evaluated, and if the app's activity level exceeds a preset activity threshold, the user is determined to be a user with the app currently installed. For example, in scenarios where active events have not yet been reported due to delays, uninstallation events are reported repeatedly, or false alarms occur, the app's most recent active time may be earlier than the app's uninstallation time, but the user may still be in an installed state.

[0040] Preferably, the APP activity level within the current time window meets the following conditions: Where S is the APP activity level within the current time window, T0 is the APP's first installation time, T1 is the APP's most recent active time, ΔT is the total number of days in the current time window, F is the number of active days within the current time window, and η and λ are the third and fourth preset weight coefficients, respectively; it can be understood that: T1-T0 represents the total number of days between the user's first installation and the most recent active time.

[0041] As mentioned above, since bitmap difference operations cannot identify the app installation status of users who have both installation and uninstallation records, a method of comparing the most recent active time with the uninstallation time is introduced to achieve filtering. Furthermore, an app activity calculation method is adopted to achieve secondary filtering of users with app installation status by measuring activity persistence and activity frequency, which is conducive to identifying a more comprehensive range of users with app installation status.

[0042] S303: Combine the several APP installation status users obtained based on step S301 and the APP installation status users determined in step S302 to obtain the APP installation status user set at the current moment.

[0043] As mentioned above, since many scenarios only record the first installation event and not the reinstallation event, and the activity event is reported after reinstallation, the above method, by leveraging the parallel computing advantage of bitmap operations, can compress the time of the intersection, union, and difference operations for hundreds of millions of users from minutes to milliseconds, quickly identifying users who have never uninstalled the APP and are still installed. At the same time, it combines the activity data to make a secondary judgment on users who have reinstalled after uninstallation. While maintaining extreme computing performance, it significantly improves the accuracy of state recognition in complex uninstallation and reinstallation scenarios.

[0044] S4. Based on the current set of users with APP installation status and the mapping relationship between user identifier and bit index, generate a bitmap structure template and store it in a preset columnar database. This can be understood as: based on the mapping relationship between user identifier and bit index, the installation status of the corresponding APP can be queried and located. For example, when the bit is 1, it indicates that the APP is in the installation status, and when it is 0, it indicates that the APP is not in the installation status.

[0045] Specifically, the default columnar database is ClickHouse, which is suitable for analysis scenarios that require high throughput and low latency.

[0046] This step solidifies the calculated set of users in the app's installed state into a bitmap structure template with mapping relationships and persists it, avoiding the need to rebuild the mapping relationship of user indexes with each incremental update, and providing a directly reusable baseline structure for subsequent incremental calculations and updates.

[0047] S5, upon receiving a new user APP behavior dataset, reads the bitmap structure template and calculates the bitmap change amount, and synchronously updates the corresponding APP installation status change results to the preset columnar database.

[0048] Specifically, the S5 steps include the following: S501 reads new user app behavior datasets in real time using the Spark computing framework and queries whether each user identifier in the new user app behavior dataset exists in the bitmap structure template based on the bitmap structure template.

[0049] S502, for any user identifier, if the user identifier exists in the bitmap structure template, query and locate the corresponding bit index for the user identifier, and perform a state change of the bit index to generate a bitmap change amount; otherwise, temporarily store the user identifier and the corresponding user APP behavior data in the buffer.

[0050] S503 synchronizes the bitmap changes to the preset columnar database to update the APP installation status corresponding to the user identifier; at the same time, it triggers the update of the bitmap structure template based on the user APP behavior data in the buffer.

[0051] As described above, by reusing pre-generated Bitmap templates, bit state change calculations are performed only for existing users, reducing incremental update volume from hundreds of millions of calculations to thousands of change calculations, significantly improving computational efficiency and achieving second-level streaming state synchronization. At the same time, template updates are triggered through buffers, ensuring both the efficiency of incremental calculations and that new users are not missed, thus achieving an adaptive update closed loop. Furthermore, considering that bitmap calculations may put pressure on database ingestion, Spark calculations are used to migrate the computational pressure and ensure the normal operation of the database.

[0052] Embodiments of the present invention also provide a non-transitory computer-readable storage medium, which can be disposed in an electronic device to store at least one instruction or at least one program related to implementing a method in the method embodiments, wherein the at least one instruction or the at least one program is loaded and executed by the processor to implement the bitmap-based on-device state recognition method provided in the above embodiments.

[0053] Embodiments of the present invention also provide an electronic device, including a processor and the aforementioned non-transitory computer-readable storage medium.

[0054] While specific embodiments of the invention have been described in detail by way of example, those skilled in the art should understand that the examples are for illustrative purposes only and not intended to limit the scope of the invention. It should also be understood that various modifications can be made to the embodiments without departing from the scope and spirit of the invention. The scope of the invention is defined by the appended claims.

Claims

1. A bitmap-based method for identifying the in-installation state, characterized in that, The method includes the following steps: S1. Based on the data extraction task type, a target APP behavior indicator set is selected from several preset APP behavior indicators. The target APP behavior indicator set includes at least APP installation data, APP uninstallation data, and APP recent active data. The data extraction task type is either a real-time data extraction task or a batch data extraction task. S2, the original user APP behavior dataset to be analyzed is divided into data according to user identifiers and distributed to each computing node in the preset distributed cluster, so that each computing node performs local pre-aggregation according to the target APP behavior index set and summarizes the aggregated data of user APP behavior. S3, convert the APP installation data and APP uninstallation data in the aggregated data into bitmap data structures respectively, and obtain the set of users in the APP installation status at the current moment by bitmap difference operation and combining it with the APP's most recently active data; S4. Based on the current set of users in the APP installation status and the mapping relationship between user identifiers and bit indexes, generate a bitmap structure template and store it in a preset columnar database. S5, upon receiving a new user APP behavior dataset, reads the bitmap structure template and calculates the bitmap change amount, and synchronously updates the corresponding APP installation status change results to the preset columnar database.

2. The bitmap-based in-installation status identification method according to claim 1, characterized in that, Step S1 includes the following steps: S101, based on the information gain value of each preset APP behavior indicator obtained in advance, calculate the importance of each preset APP behavior indicator; wherein, the importance of any preset APP behavior indicator meets the following condition: Among them, W i To determine the importance of the i-th preset APP behavior metric, G i and G k B represents the information gain values ​​of the i-th and k-th preset APP behavior metrics, respectively. i and B k , respectively, are the preset indicator weights of the i-th and k-th preset APP behavior indicators under the corresponding data extraction task type, n is the total number of preset APP behavior indicators, and α and β are the first preset weight coefficient and the second preset weight coefficient, respectively. S102, filter the preset APP behavior indicators whose importance is greater than the preset threshold as target APP behavior indicators.

3. The bitmap-based in-installation state identification method according to claim 1, characterized in that, The step of dividing the original user APP behavior dataset to be analyzed according to user identifiers and distributing it to various computing nodes in a preset distributed cluster includes the following steps: S201, the hash value space corresponding to the user identifier is evenly divided into several logical partitions, and each logical partition is initially mapped to each computing node using the hash modulo method, so as to obtain several logical partitions corresponding to each computing node. S202 collects the node load of each computing node in real time and obtains the normalized value of the real-time load of each computing node through normalization processing. S203: For any computing node, when the normalized value of the real-time load corresponding to the computing node exceeds the preset load threshold, the logical shards corresponding to the computing node are migrated to the computing node with the lowest current load according to the preset migration step size, so as to realize the allocation of the original user APP behavior dataset.

4. The bitmap-based in-installation status identification method according to claim 1, characterized in that, Step S2 also includes the following steps: S210, For any target APP behavior indicator in the target APP behavior indicator set, calculate the pre-aggregation result of each computing node and the summarization error of the original user APP behavior dataset on the target APP behavior indicator. S220, when the aggregation error is greater than the preset error threshold, the local pre-aggregation operation is re-executed on the corresponding computing node.

5. The bitmap-based in-installation status identification method according to claim 1, characterized in that, Step S3 includes the following steps: S301, convert the APP installation data and APP uninstallation data in the aggregated data into bitmap installation data set and bitmap uninstallation data set respectively, and obtain several APP in-installation users with no uninstallation records through difference set operation; S302, for users with installation and uninstallation records, if it is determined that the user's most recent APP activity time is later than the APP uninstallation time, then the user is determined to be a user with the APP installed; otherwise, the user's APP activity level within the current time window is evaluated, and when the APP activity level is greater than the preset activity level threshold, the user is determined to be a user with the APP installed. S303: Combine the several APP installation status users obtained based on step S301 and the APP installation status users determined in step S302 to obtain the APP installation status user set at the current moment.

6. The bitmap-based in-installation state identification method according to claim 5, characterized in that, The app activity level within the current time window meets the following conditions: Where S is the APP activity level within the current time window, T0 is the APP's first installation time, T1 is the APP's most recent active time, ΔT is the total number of days in the current time window, F is the number of active days within the current time window, and η and λ are the third and fourth preset weight coefficients, respectively.

7. The bitmap-based in-installation state identification method according to claim 1, characterized in that, The S5 steps include the following: S501 reads new user app behavior datasets in real time through the Spark computing framework and queries whether each user identifier in the new user app behavior dataset exists in the bitmap structure template based on the bitmap structure template. S502, for any user identifier, if the user identifier exists in the bitmap structure template, query and locate the corresponding bit index for the user identifier, perform the state change of the bit index, and generate the bitmap change amount; Conversely, the user identifier and the corresponding user APP behavior data are temporarily stored in the buffer. S503 synchronizes the bitmap changes to the preset columnar database to update the APP installation status corresponding to the user identifier; at the same time, it triggers the update of the bitmap structure template based on the user APP behavior data in the buffer.

8. The bitmap-based in-installation state identification method according to claim 1, characterized in that, In step S3, before converting the APP installation data and APP uninstallation data in the aggregated data into bitmap data structures respectively, the following steps are also included: S310, Identify outliers and outlier types in the aggregated data, and map the degree of anomalousness corresponding to the outlier type of each outlier; the degree of anomalousness is one of mild anomaly and severe anomaly. S320: For slightly anomalous data, mean-filling method is used for correction; S330 removes severely abnormal data and records the operation log.

9. A non-transitory computer-readable storage medium, wherein the storage medium stores at least one instruction or at least one program segment, characterized in that, The at least one instruction or the at least one program segment is loaded and executed by the processor to implement the bitmap-based in-installation state identification method as described in any one of claims 1-8.

10. An electronic device, characterized in that, Includes a processor and the non-transitory computer-readable storage medium as described in claim 9.

Images