Data processing method, system, storage medium, device and program product

Abstract

The application discloses a data processing method, system, storage medium, device and program product. The method comprises the following steps: acquiring a filter bit mask and a user data set. Each bit in the filter bit mask corresponds to a user dimension. The user dimension corresponding to the bit in the preset filter state of the filter bit mask is a target filter dimension. The user data set comprises a plurality of user bit masks. The length of the user bit mask and the filter bit mask is the same, and the same-position bits correspond to the same user dimension. The bit in the preset label state of the user bit mask corresponds to a user label. Based on the bit state of the plurality of user bit masks and the filter bit mask, it is determined whether the corresponding user hits the target filter dimension to obtain a comparison result. In the case of determining that part of the comparison result indicates that the target filter dimension is hit, the corresponding user bit mask is subjected to first filtering processing. The above method can reduce query delay and greatly reduce data processing pressure in the scene of processing millions of users in batches.

Description

Technical Field

[0001] This application relates to the field of content delivery technology, specifically to a data processing method, system, storage medium, device, and program product. Background Technology

[0002] In content delivery scenarios within the internet entertainment industry, servers typically need to filter and screen massive numbers of users based on multiple dimensions of user profiles to control the reach of specific content. In related technologies, this multi-dimensional attribute data is usually generated and stored independently by different upstream business systems. The server often stores each dimension's tags in a distributed manner, such as storing each dimension as an independent data field or a separate data set. When filtering is required, the server needs to query the corresponding tag fields for each user's dimensions sequentially and perform multiple rounds of conditional checks or set member validations at the application layer.

[0003] However, under the aforementioned storage and judgment methods, as the user base expands, the number of network interactions between the server and the storage medium increases linearly with the number of users and dimensions, easily leading to strained network bandwidth resources and a surge in the computational load on the server processor. Simultaneously, distributing the storage of dimension tags consumes a significant amount of additional key-value metadata space, resulting in low memory utilization. Furthermore, when multi-source heterogeneous data undergoes full or incremental synchronization at different frequencies, frequent overwrite and format conversion operations are not only time-consuming but also prone to data inconsistency and concurrent write conflicts, thus significantly impacting the system's storage efficiency, query performance, and data synchronization stability. Summary of the Invention

[0004] This application provides a data processing method, system, storage medium, device, and program product that can complete the joint filtering and judgment of multiple dimensions through a single integer bit operation, reducing the latency of a single query and significantly reducing the data processing pressure in batch processing of millions of users.

[0005] On one hand, embodiments of this application provide a data processing method, the method comprising: obtaining a filter bitmask and a user dataset, wherein each bit in the filter bitmask corresponds to a user dimension, the user dimension corresponding to a bit in the filter bitmask that is in a preset filtering state is a target filtering dimension, the user dataset includes multiple user bitmasks, the user bitmasks and the filter bitmasks have the same length, and the same bits in the same position correspond to the same user dimension, and bits in the user bitmask that are in a preset label state correspond to a user label; determining whether a corresponding user matches the target filtering dimension based on the bit states of the multiple user bitmasks and the filter bitmask respectively, and obtaining a comparison result; and when it is determined that some of the comparison result identifiers match the target filtering dimension, performing a first filtering process on the corresponding user bitmask.

[0006] On the other hand, embodiments of this application provide a data processing system, which includes at least one server. The server is configured to: acquire a filter bitmask and a user dataset, wherein each bit in the filter bitmask corresponds to a user dimension, and the user dimension corresponding to the bit in the filter bitmask that is in a preset filtering state is a target filtering dimension; the user dataset includes multiple user bitmasks, the user bitmasks and the filter bitmasks have the same length, and the bits in the same position correspond to the same user dimension; and the bits in the user bitmask that are in a preset label state correspond to a user label; determine whether the corresponding user matches the target filtering dimension based on the bit states of the multiple user bitmasks and the filter bitmask, and obtain a comparison result; and if it is determined that some of the comparison result identifiers match the target filtering dimension, perform a first filtering process on the corresponding user bitmask.

[0007] On the other hand, embodiments of this application provide a computer-readable storage medium storing a computer program adapted for loading by a processor to execute the data processing method as described in any of the above embodiments.

[0008] On the other hand, embodiments of this application provide a computer device, the computer device including a processor and a memory, the memory storing a computer program, the processor executing the data processing method as described in any of the above embodiments by calling the computer program stored in the memory.

[0009] On the other hand, embodiments of this application provide a computer program product, including computer instructions, which, when executed by a processor, implement the data processing method as described in any of the above embodiments.

[0010] This application embodiment determines whether a user is matched in a specific dimension by performing a bitwise AND operation between two bitmasks. Compared with querying user data item by item across multiple fields, it can achieve the effect of joint filtering and judgment of multiple dimensions with O(1) time complexity, which can greatly reduce query latency. In the scenario of batch processing of millions of users, it is beneficial to significantly reduce data processing pressure. Attached Figure Description

[0011] To more clearly illustrate the technical solutions in the embodiments of this application, the accompanying drawings used in the description of the embodiments will be briefly introduced below. Obviously, the accompanying drawings described below are only some embodiments of this application. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0012] Figure 1 This is a schematic diagram of a data processing system provided in an embodiment of this application.

[0013] Figure 2 This is a flowchart illustrating the data processing method provided in an embodiment of this application.

[0014] Figure 3 This is a schematic diagram of the structure of the data processing system provided in the embodiments of this application.

[0015] Figure 4 A schematic diagram of the structure of a computer device provided in an embodiment of this application. Detailed Implementation

[0016] The technical solutions of the embodiments of this application will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of this application, and not all embodiments. Based on the embodiments of this application, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of this application.

[0017] This application provides a data processing method, system, storage medium, device, and program product. Specifically, the data processing method of this application can be executed by a computer device, which can be a terminal or a server. The terminal can be a smartphone, tablet, laptop, smart TV, wearable smart device, smart vehicle terminal, etc. The terminal can also include a client, which can be a game client, browser client, instant messaging client, or mini-program, etc. The server can be an independent physical server, a server cluster composed of multiple physical servers, or a distributed system. It can also be a cloud server that provides basic cloud computing services such as cloud services, cloud databases, cloud computing, cloud functions, cloud storage, network services, cloud communication, middleware services, domain name services, security services, content delivery networks (CDN), and big data and artificial intelligence platforms.

[0018] For example, when this data processing method is run on a terminal device, the terminal device may include a display screen and a processor. The display screen is used to present game visuals and receive commands generated by the player interacting with the game visuals. The game visuals may include a portion of a virtual game scene, which is a virtual world where virtual characters move. The processor is used to store the game application, run the game, generate game visuals, respond to commands, and control the display of the game visuals on the display screen. When the player interacts with the game visuals through the display screen, the game visuals can control the local content of the terminal device in response to the received operation commands. The terminal device can provide the graphical user interface to the player in various ways, such as rendering the display on the terminal device's screen or presenting the graphical user interface through holographic projection.

[0019] For example, when this data processing method runs on a server, it can be implemented and executed based on a cloud data processing system. A cloud data processing system refers to a game-based approach using cloud computing. This system includes servers and client devices. The core of the game application and the core of the game screen presentation are separate; the storage and execution of the data processing method are completed on the server. The game screen presentation is completed on the client, which is primarily used for receiving and sending game data and displaying the game screen. For example, the client can be a display device with data transmission capabilities located close to the player, such as a mobile terminal, television, computer, PDA, personal digital assistant, or head-mounted display device. However, the terminal device for processing game data is the server in the cloud. During gameplay, the player operates the client to send commands to the server. The server controls the game operation according to the commands, encodes and compresses game screen data, returns it to the client via the network, and finally, the client decodes and outputs the game screen.

[0020] It should be noted that, in this embodiment, the entity executing the data processing method can be a terminal device or a server. The terminal device can be a local terminal device or a client device in the aforementioned cloud gaming. This embodiment does not limit the type of the entity executing the method.

[0021] It is understood that in the specific implementation of this application, user object data, context data and other related data are involved. When the embodiments of this application are applied to specific products or technologies, user permission or consent is required, and the collection, use and processing of related data must comply with the relevant laws, regulations and standards of the relevant countries and regions.

[0022] For example, in conjunction with the above description, Figure 1 This application illustrates a data processing system 1000 for implementing a data processing method, as provided in an embodiment of this application. The data processing system 1000 may include at least one terminal 1001, at least one server 1002, at least one database 1003, and a network. The user-held terminal 1001 can connect to different servers via the network. The terminal is any device with computing hardware capable of supporting and executing software applications corresponding to the game.

[0023] In the aforementioned data processing system 1000, terminal 1001 is used to install and run game applications. In some cases, the game application may not need to be pre-installed on terminal 1001, and players can directly access the game through a browser or other client. Players log in to the game application using their registered game account to control the virtual character corresponding to that account and participate in the game. When a player logs in to the game application, terminal 1001 sends a login request to server 1002. Server 1002 verifies the game account used by the player and determines the game mechanics corresponding to the game account based on the login request. If the verification is successful, a login success notification is returned to terminal 1001. During the player's participation in the game through the game application, terminal 1001 and server 1002 exchange data. Terminal 1001 sends various information to server 1002. Server 1002 determines the display data for terminal 1001 based on the stored game mechanics and the received information, and sends the display data back to terminal 1001 so that terminal 1001 can display the display data sent by server 1002 to the player.

[0024] In possible application scenarios, different terminals 1001 may be served by different servers 1002. Therefore, in order to distinguish the servers 1002 corresponding to different game terminals 1001, the embodiments of this application will use the terms "first" and "second" to describe them. In fact, the servers 1002 corresponding to different game terminals 1001 can be the same server 1002. Therefore, without distinguishing between "first" and "second", it can be understood that the terminals 1001 corresponding to virtual characters in the same game scene are served by the same server 1002.

[0025] Furthermore, when the data processing system 1000 includes multiple terminals, multiple servers, and multiple networks, different terminals can connect to each other through different networks and servers. The network can be a wireless network or a wired network; for example, wireless networks include wireless local area networks (WLAN), local area networks (LAN), cellular networks, 2G networks, 3G networks, 4G networks, 5G networks, etc. Additionally, different terminals can also connect to other terminals or servers using their own Bluetooth networks or hotspot networks. Furthermore, the system 100 can include multiple databases coupled to different servers, and can continuously store game-related information in the databases while different users are playing multiplayer games online.

[0026] It should be noted that in this embodiment, multiple terminal devices are running the same virtual game. Therefore, data interaction between the multiple terminal devices can be achieved through the virtual game's server. Thus, sending data from terminal device 1 to terminal device 2 can be understood as: terminal device 1 sends data to the virtual game's server, and the server sends the data to terminal device 2. Receiving data from terminal device 2 can be understood as: terminal device 1 receives data sent by the virtual game's server, which is the data sent by terminal device 2 to the server. Alternatively, there may be no game server, and terminal device 1 directly sends game data to terminal device 2.

[0027] It should be noted that, Figure 1 The schematic diagram of the data processing system shown is merely an example. The data processing system 1000 described in this application embodiment is intended to more clearly illustrate the technical solutions of this application embodiment and does not constitute a limitation on the technical solutions provided in this application embodiment. As those skilled in the art will know, with the evolution of data processing systems and the emergence of new business scenarios, the technical solutions provided in this application embodiment are also applicable to similar technical problems.

[0028] It should be noted that the triggering operations mentioned in the subsequent detailed description of the data processing method provided in the embodiments of this application can all be regarded as triggering operations performed by the player through a finger or by controlling a medium such as a mouse, keyboard, or stylus. The specific medium used can be determined according to the type of computer device. For example, when the computer device is a touchscreen device such as a mobile phone, tablet, or game console, the player can operate on the touchscreen using any suitable object or accessory such as a finger or stylus. When the terminal device is a non-touchscreen terminal device such as a desktop computer or laptop, the player can operate using external devices such as a mouse or keyboard.

[0029] The technical solution of this application will be described in detail below through specific embodiments. It should be noted that the following specific embodiments can be combined with each other, and the same or similar concepts or processes may not be described again in some embodiments.

[0030] In this embodiment of the application, a graphical user interface is provided through a terminal device. The graphical user interface includes at least a portion of the virtual scene and at least one virtual character.

[0031] The aforementioned virtual scene can be a game scene, which can be understood as a simulation of the real world within a game, a semi-simulated / semi-fictional virtual environment, or a purely fictional virtual environment. A game scene can be any of the following: two-dimensional, 2.5-dimensional, or three-dimensional virtual scenes. A virtual scene typically includes multiple scene elements, which are the various elements required to construct the virtual scene. For example, these may include, but are not limited to, at least one of the following: virtual character elements, virtual item elements, virtual building elements, virtual terrain elements, and virtual vegetation elements. Virtual terrain elements may include, but are not limited to, natural landforms such as land, ocean, lakes, and rivers. A virtual scene is a scenario where players control virtual characters to complete game logic.

[0032] As can be understood, a virtual character is a game character controlled by the player in a game. The player manipulates this virtual character to perform various game activities within the game environment, such as picking up items, engaging in combat, exploring, or solving puzzles. This virtual character can represent the player's image, and each virtual character can be implemented using a 3D or 2D virtual model; this embodiment does not specifically limit this. Virtual characters include, but are not limited to, at least one of the following: virtual human, virtual animal, and virtual machine.

[0033] Please refer to Figure 2 One data processing method of this application may include: 03: Obtain the filter bitmask and user dataset. Each bit in the filter bitmask corresponds to a user dimension. The user dimension corresponding to the bit in the filter bitmask that is in the preset filter state is the target filter dimension. The user dataset includes multiple user bitmasks. The user bitmask and the filter bitmask have the same length, and the bits in the same position correspond to the same user dimension. The bit in the user bitmask that is in the preset label state corresponds to a user label. 04: Based on the bit states of multiple user bitmasks and filter bitmasks respectively, determine whether the corresponding user matches the target filter dimension and obtain the comparison result; 05: If it is determined that some of the comparison results match the target filtering dimension, the corresponding user bitmask is subjected to the first filtering process.

[0034] The data processing method of this application embodiment can be implemented by the data processing system 100 of this application embodiment. Specifically, please refer to... Figure 3 This application discloses a data processing system 100, which may include at least one server 110. The server 110 can be used to: acquire a filter bitmask and a user dataset, wherein each bit in the filter bitmask corresponds to a user dimension, and the user dimension corresponding to a bit in a preset filtering state in the filter bitmask is the target filtering dimension; the user dataset includes multiple user bitmasks, the user bitmasks and filter bitmasks have the same length, and corresponding bits in the same position correspond to the same user dimension; bits in a preset label state in the user bitmask correspond to a user label; based on the bit states of the multiple user bitmasks and the filter bitmask, determine whether the corresponding user matches the target filtering dimension, and obtain a comparison result; if it is determined that some comparison results match the target filtering dimension, perform a first filtering process on the corresponding user bitmask.

[0035] The above data processing method and data processing system 100 determine whether a user is matched in a specific dimension by performing a bitwise AND operation between two bitmasks. Compared with querying user data item by item across multiple fields, it can achieve the effect of joint filtering and judgment of multiple dimensions with O(1) time complexity, which can greatly reduce query latency. In the scenario of batch processing of millions of users, it is beneficial to significantly reduce data processing pressure.

[0036] Specifically, in some implementations, the filter bitmask and user bitmask are multi-bit binary strings, where each bit corresponds to a user dimension. A value of 0 in the filter bitmask indicates that the user dimension corresponding to that bit is not currently filtered; a value of 1 in the filter bitmask indicates that the user is in a preset filtering state, or that the user dimension corresponding to that bit is currently filtered. A value of 0 in the user bitmask indicates that the corresponding user does not currently possess the characteristics of the user dimension corresponding to that bit; a value of 1 in the user bitmask indicates that the user is in a preset label state, or that the corresponding user currently possesses the characteristics of the user dimension corresponding to that bit.

[0037] In one implementation, the filter bitmask is 010111100, and one of the user bitmasks is 010010000. By performing a bitwise AND operation on the filter bitmask and the user bitmask, the resulting comparison bitmask is 010010000. This means that the corresponding user has features in the user dimension corresponding to the first bit and the user dimension corresponding to the fourth bit (with the leftmost bit of the bitmask as the zeroth bit). In other words, the target filter dimensions corresponding to the first and fourth bits are matched, and the user bitmask matches the filter, thus performing the first filtering process on the user bitmask.

[0038] Based on the above, each bit in the bitmask corresponds to a user dimension. When performing the bitwise AND operation, all user dimensions can be filtered and judged at the same time, with a time complexity of O(1). Moreover, as long as the comparison bitmask is determined to be a non-zero mask, it means that the corresponding user has features in the user dimension that needs to be filtered. Compared with the solution of querying users one by one according to all user dimensions to obtain all query results, the query latency can be greatly reduced, which is beneficial to significantly reduce the data processing pressure in the scenario of batch processing of millions of users.

[0039] Additionally, please combine Figure 3 ,exist Figure 3 In this system, there are multiple servers 110, and these servers 110 can communicate with each other. Specifically, files containing user datasets can be distributed across different servers 110 using a distributed caching method. In this way, a complete set of data can be distributed from one server 110 to other servers 110, and multiple data segments can be sent from different servers 110 to the same server 110 to form a complete set of data.

[0040] In some implementations, the user dimension may include at least one of the following: an identity attribute dimension, a behavioral risk control dimension, and a probability assessment dimension. The identity attribute dimension is used to represent the user's basic identity information, the behavioral risk control dimension is used to represent the user's historical behavioral tag information, and the probability assessment dimension is used to represent the probability level information obtained based on user behavior predictions.

[0041] In some implementations, identity attribute dimensions may include high-risk account dimensions, real-name authentication dimensions, age exceeding dimensions, important person dimensions, and facial recognition dimensions. Behavioral risk control dimensions may include refund behavior dimensions.

[0042] Specifically, in some implementations, the bitmask has 9 bits. From the leftmost bit to the rightmost bit of the bitmask, the corresponding user dimensions are high-risk account dimension, real-name authentication dimension, age limit exceeded dimension, important person dimension, refund behavior dimension, low probability assessment dimension, medium probability assessment dimension, high probability assessment dimension, and facial recognition dimension.

[0043] Taking minors as an example, the high-risk account dimension corresponds to whether the user is marked as a high-risk minor account in the risk control system; the real-name authentication dimension corresponds to whether the user has been confirmed as a minor through real-name authentication; the age limit dimension corresponds to whether the user exceeds the age limit; the important person dimension corresponds to whether the user has a minor mark in the VIP (important person) system; the refund behavior dimension corresponds to whether the user has refund behavior and is suspected of being a minor; the low probability assessment dimension corresponds to the probability that the user is a minor according to the machine learning model is low; the medium probability assessment dimension corresponds to the probability that the user is a minor according to the machine learning model is medium; the high probability assessment dimension corresponds to the probability that the user is a minor according to the machine learning model is high; and the facial recognition dimension corresponds to the user being determined to be a minor through facial recognition.

[0044] In some implementations, the data processing method may include: The acquired raw user data is subjected to probability prediction processing to determine the target probability assessment dimension. The user dimension includes multiple probability assessment dimensions, and the target probability assessment dimension is one of the multiple probability assessment dimensions. Each probability assessment dimension has a corresponding probability prediction degree. Set the bits in the user bitmask corresponding to the target probability evaluation dimension, and clear the bits corresponding to other probability evaluation dimensions.

[0045] The data processing method of this application embodiment can be implemented by the data processing system 100 of this application embodiment. Specifically, please refer to... Figure 3Server 110 can be used to: perform probability prediction processing on the acquired raw user data, determine the target probability evaluation dimension, the user dimension includes multiple probability evaluation dimensions, the target probability evaluation dimension is one of the multiple probability evaluation dimensions, each probability evaluation dimension has a corresponding probability prediction degree; set the bit in the user bitmask corresponding to the target probability evaluation dimension, and clear the bits corresponding to other probability evaluation dimensions.

[0046] In this way, the mutual exclusion constraint effect of different user dimensions in the bitmask can be achieved.

[0047] Specifically, taking the user dimension, which includes low-probability assessment, medium-probability assessment, and high-probability assessment dimensions, as an example, the corresponding user bitmask is a three-digit string. The rightmost bit corresponds to the low-probability assessment dimension, the middle bit to the medium-probability assessment dimension, and the leftmost bit to the high-probability assessment dimension. After obtaining the raw user data, probability prediction can be performed on the user based on the raw user data to determine the likelihood that the user is a minor.

[0048] In the probability prediction process, a corresponding predicted value is obtained. The predicted value is a number from 0 to 1. Based on the specific interval of the predicted value, a corresponding probability evaluation dimension can be determined. In one implementation, the interval corresponding to the low probability evaluation dimension is greater than or equal to 0.6 and less than 0.75, the interval corresponding to the medium probability evaluation dimension is greater than or equal to 0.75 and less than 0.9, and the interval corresponding to the high probability evaluation dimension is greater than or equal to 0.9. Based on the interval of the predicted value, the user dimension corresponding to the interval can be determined as the target probability evaluation dimension.

[0049] Accordingly, if the target probability evaluation dimension is a low probability evaluation dimension, the corresponding user bitmask is 001; if the target probability evaluation dimension is a medium probability evaluation dimension, the corresponding user bitmask is 010; if the target probability evaluation dimension is a high probability evaluation dimension, the corresponding user bitmask is 100; if the predicted value is less than 0.6, the target probability evaluation dimension is empty, and the corresponding user bitmask is 000.

[0050] Building upon the above, when some user dimensions are mutually exclusive, the aforementioned mutually exclusive encoding can eliminate semantic ambiguity between these user dimensions—it would be logically contradictory to simultaneously hit both "high probability" and "medium probability." Adopting a strategy of taking the highest value level ensures the determinism and interpretability of the encoding.

[0051] In addition, in some cases, the original user data may come from different sources, or users may have different terminal devices that may provide different original user data. In this case, probabilistic prediction processing can be performed on the original user data from different sources to obtain corresponding predicted values, and then the maximum value among these predicted values ​​can be taken as the final predicted value.

[0052] In some implementations, the data processing method may include: Acquire tag data for multiple users, each user has multiple tag data, and each tag data corresponds to a user dimension; Multiple tag data corresponding to each user are merged to generate a corresponding user bitmask; Multiple user bitmasks are stored to generate a user information storage file, which includes the user dataset.

[0053] The data processing method of this application embodiment can be implemented by the data processing system 100 of this application embodiment. Specifically, please refer to... Figure 3 Server 110 can be used to: acquire tag data of multiple users, each user having multiple tag data, each tag data corresponding to a user dimension; merge the multiple tag data corresponding to each user to generate a corresponding user bitmask; store the multiple user bitmasks to generate a user information storage file, the user information storage file including the user dataset.

[0054] In this way, user datasets can be generated.

[0055] Specifically, please combine Figure 1 When a user uses a terminal, their relevant information can be recorded within the terminal to form corresponding tag data. This tag data can then be transmitted to server 110, allowing the acquisition of tag data for multiple users. Since these tag data are structurally independent, they are merged. Specifically, according to the user dimension corresponding to each tag data, the tag data is merged based on the number of bits in the bitmask, resulting in a single user bitmask stored together. These tag data can then be correlated through the user bitmask. Furthermore, each user can be associated with a corresponding user bitmask, allowing multiple user bitmasks to be stored to generate a user information storage file containing the user dataset. When subsequent filtering and judgment are needed, the user dataset can be retrieved by calling the user information storage file.

[0056] In some implementations, before merging the multiple tag data corresponding to each user, the process may include: Perform data quality verification on tag data for multiple users; After verifying that the tag data of multiple users has passed the validation, it is determined that the multiple tag data corresponding to each user will be merged.

[0057] The data processing method of this application embodiment can be implemented by the data processing system 100 of this application embodiment. Specifically, please refer to... Figure 3 Server 110 can be used to: perform data quality verification on tag data of multiple users; and after determining that the tag data of multiple users has passed the verification, determine to merge the multiple tag data corresponding to each user.

[0058] This avoids the possibility of data loss in the obtained user bitmask.

[0059] Specifically, please combine Figure 1 In practical applications, a user's tag data may come from different terminals, and the upload times of each tag data point to the server 110 may also be inconsistent, easily leading to occasional data loss. Therefore, by performing data quality verification on the tag data of multiple users, it can be ensured that data from each user dimension data source from different upstream systems has been completely written before user bitmasking encoding.

[0060] In some implementations, data quality verification of tag data for multiple users may include: Detect whether the number of tag data for multiple users under the same user dimension is greater than or equal to the corresponding data volume threshold. The data volume threshold is determined by the user dimension and the data source type of the tag data. If the detection results for all user dimensions are positive, then multiple tag data are determined to have passed the verification.

[0061] The data processing method of this application embodiment can be implemented by the data processing system 100 of this application embodiment. Specifically, please refer to... Figure 3 Server 110 can be used to: detect whether the number of tag data for multiple users under the same user dimension is greater than or equal to the corresponding data volume threshold, the data volume threshold being determined by the user dimension and the data source type of the tag data; if the detection results for all user dimensions are yes, determine that the multiple tag data passes the verification.

[0062] Specifically, in one implementation, the data source type for the tag data can include a first type of data source and a second type of data source. User dimensions can include real-name authentication, age exceeding limits, probability assessment, probability prediction, refund behavior, refund application, historical important figures, important figures, and facial recognition. The first type of data source can be a mobile game data source, and the second type of data source can be a PC game data source.

[0063] Once the tag data is acquired, it can be partitioned and stored according to daily frequency. In other words, the storage space can be divided into multiple partitions, each corresponding to all tag data acquired within a given day. The tag data used for merging can be used to detect all tag data within a partition on a given day under different user dimensions, and then threshold detection can be applied to the data volume of these tag data according to the user dimension.

[0064] Based on the above, the data volume thresholds for each user dimension are shown in Table 1 below:

[0065] Table 1 The data volume threshold of -1 indicates that the check of this dimension in the corresponding data source type is skipped. This is because the label data of the corresponding user dimension only produces data in the corresponding data source type, and other data sources do not need to be checked.

[0066] In some implementations, the data processing method may include: If the detection results for some user dimensions are negative, backtracking is performed on the tag data of multiple users according to the corresponding user dimensions.

[0067] The data processing method of this application embodiment can be implemented by the data processing system 100 of this application embodiment. Specifically, please refer to... Figure 3 Server 110 can be used to: perform backtracking processing on the tag data of multiple users according to the corresponding user dimensions when the detection results corresponding to some user dimensions are negative.

[0068] This avoids contamination of invalid data due to data source failure.

[0069] Specifically, in one implementation, if the number of tag data under the same user dimension is less than the corresponding data volume threshold, there may be cases where the data source did not upload tag data on that day. In this case, the tag data under the user dimension is backtracked to the data of the previous day. Alternatively, the tag data under the same user dimension is obtained from the tag data stored in the partition of the previous day to generate the user bitmask.

[0070] Furthermore, since the backtracking is performed on the user-level tag data, it does not affect the acquisition and merging of tag data under other user dimensions, and it also reduces the amount of data required for backtracking processing. In some cases, the maximum time limit for backtracking processing is 30 days, meaning it can backtrack to data from 30 days ago. This allows it to determine if the data source has a long-term failure and cannot upload valid data, thus causing the merging task to fail directly and trigger an alarm.

[0071] In some implementations, after a backtracking process is performed, a polling wait interval can be set during which the next data quality check will not be performed. This provides sufficient time for the upstream system to delay writing data and avoids misjudgment due to scheduling timing differences.

[0072] In some implementations, obtaining tag data from multiple users may include: Retrieve all primary keys for each user. Each primary key contains multiple tag data for a corresponding user. Different primary keys for the same user store their corresponding multiple tag data independently. The multiple tag data corresponding to each user are merged to generate a corresponding user bitmask, which may include: Based on the multiple tag data corresponding to each primary key, perform bitwise or fusion processing to generate a user bitmask for that user.

[0073] The data processing method of this application embodiment can be implemented by the data processing system 100 of this application embodiment. Specifically, please refer to... Figure 3 Server 110 can be used to: obtain all primary keys of each user, each primary key having multiple tag data corresponding to a user, and storing the corresponding multiple tag data independently between different primary keys of the same user; perform bitwise or fusion processing on the multiple tag data corresponding to all primary keys to generate a user bitmask corresponding to a user.

[0074] This helps ensure the integrity of the data used for subsequent filtering and judgment.

[0075] Specifically, a user's primary key corresponds to different upstream systems for tag data. It's understandable that in practical applications, users can use the application through different terminals (upstream systems), which fragments the user's complete application behavior across different terminals, resulting in certain deficiencies in all tag data provided by any single upstream system.

[0076] Based on the above, by obtaining multiple tag data on all primary keys, the tag data corresponding to different primary keys are merged to eliminate the segmentation of complete user behavior by the upstream system. This allows the final user bitmask to reflect the complete user behavior as much as possible, so that in subsequent filtering decisions, it can be accurately determined whether a user will be filtered.

[0077] In such an implementation, taking the user dimension including the real-name authentication dimension as an example, the bit pattern is determined based on the multiple tag data corresponding to each primary key, which can be reflected by the following code: "bit[1] = 1IFFaccount_realname == "1"ORurs_realname == "1""; Where bit[1] represents the first bit of the real-name authentication dimension in the user bitmask, account_realname represents the tag data of the real-name authentication dimension in the mobile game, and urs_realname represents the tag data of the real-name authentication dimension in the PC game.

[0078] Additionally, in some implementations, the user dimension may include a key figure dimension, and the tag data may include the following fields: account_vip (indicating whether the user is a key figure on the mobile game), account_vip_history (indicating whether the user was once a key figure on the mobile game), urs_vip (indicating whether the user was a key figure on the PC game), and urs_vip_history (indicating whether the user was once a key figure on the PC game). Wherein, if any of the above fields has a value of 1, the bit in the user bitmask corresponding to the key figure dimension will be set to 1.

[0079] Furthermore, in some implementations, the user dimension may include a refund behavior dimension, and the tag data may include the following fields: account_refund (indicating whether the user has had a refund behavior on the mobile game), account_refund_submit (indicating whether the user has submitted a refund request on the mobile game), urs_refund (indicating whether the user has had a refund behavior on the PC game), and urs_refund_submit (indicating whether the user has submitted a refund request on the PC game). Wherein, if any of the above fields has a value of 1, the bit in the user bitmask corresponding to the refund behavior dimension will be set to 1.

[0080] In some implementations, merging multiple tag data corresponding to each user may include: Retrieve the account primary key mapping table and the passport primary key mapping table. The account primary key mapping table includes multiple tag data stored in the account primary key and the corresponding passport primary key identifier. The passport primary key mapping table includes multiple tag data stored in the passport primary key and the corresponding account primary key identifier. Each user has at least one primary key, which includes the account primary key and the passport primary key. Different primary keys of the same user independently store the corresponding multiple tag data. Merge the account primary key mapping table and the passport primary key mapping table to obtain a full table of primary key mapping relationships; Generate a user bitmask based on the tag data in the full table of primary key mapping relationships.

[0081] The data processing method of this application embodiment can be implemented by the data processing system 100 of this application embodiment. Specifically, please refer to... Figure 3 Server 110 can be used to: obtain the account primary key mapping table and the passport primary key mapping table. The account primary key mapping table includes multiple tag data stored in the account primary key and the corresponding passport primary key identifier. The passport primary key mapping table includes multiple tag data stored in the passport primary key and the corresponding account primary key identifier. Each user has at least one primary key, which includes the account primary key and the passport primary key. Different primary keys of the same user independently store multiple tag data. The server can merge the account primary key mapping table and the passport primary key mapping table to obtain a full table of primary key mapping relationships. The server can generate a user bitmask based on the tag data in the full table of primary key mapping relationships.

[0082] In this way, user data can be unified across different upstream systems.

[0083] Specifically, in one implementation, the above-described merging process can be implemented using the following code: "# Step 1: Join the underage tag table on the account side with the risk control table to obtain the complete tags on the account side:" acc_join_df = acc_minors_df.join(acc_risk_df, on="account_id"); # Step 2: Left join the mapping table and supplement the corresponding urs identifier for each account: acc_join_df = acc_join_df.join(mapping_df, on="account_id", how="left"); # Step 3: The same applies to the URS side—join the tag table and the risk control table internally, then join the mapping table on the left: urs_join_df = urs_minors_df.join(urs_risk_df, on="urs") urs_join_df = urs_join_df.join(mapping_df, on="urs", how="left"); # Step 4: Perform a dual-key full external join to ensure that no user is missed if they exist on either side: merge_df = acc_join_df.join(urs_join_df, on="user_id", how="full_outer"); # Step 5: Fill empty values ​​in the ML probability field with "0.0" to ensure the correctness of string comparisons in the UDF: merge_df = merge_df.fillna("0.0", subset=["minor_prob_value"])"; Specifically, by setting `fillna("0.0")`, the ML probability value (corresponding to the probability evaluation dimension) field is used in the UDF for threshold determination using string comparison (max_value>= "0.9"). Filling empty values ​​with "0.0" ensures the correctness of string comparison within the range of "0.0" to "1.0", while avoiding anomalies when None is involved in the comparison.

[0084] Additionally, in some cases, when mapping tag data from mobile games to PC games, a full mapping can be performed, allowing user tag data on mobile games to be mapped to all other PC games and recorded to form tag data on the PC games. When mapping tag data from PC games to mobile games, the mapping is performed on the mobile game with the most recent activity time or update time (this can be achieved by sorting by exectime DESC and taking row_number=1), so that user tag data on PC games is only mapped to the mobile game that the user has recently used and recorded to form tag data on the mobile game.

[0085] In some implementations, after merging multiple tag data corresponding to each user to generate a corresponding user bitmask, the process may include: Based on all user bitmasks generated on the current date and all user bitmasks generated on the previous date, variables are updated in the pending storage file, which is used to generate the user information storage file.

[0086] The data processing method of this application embodiment can be implemented by the data processing system 100 of this application embodiment. Specifically, please refer to... Figure 3 Server 110 can be used to update variables in a pending storage file based on all user bitmasks generated on the current date and all user bitmasks generated on the previous date. The pending storage file is used to generate a user information storage file.

[0087] In this way, while achieving daily updates of user datasets, the pressure of data transmission and writing during updates can be reduced.

[0088] Specifically, the generated user bitmasks can be partitioned and stored according to their generation date. Given all user bitmasks generated on a given day, the determined bitmasks can be compared with those generated the previous day to identify differences (additions, modifications, and deletions) between the two days. Based on these differences, variables are updated in the pending storage file. This avoids updating all user bitmasks generated on the same day to the pending storage file, reducing the pressure of data transfer and writing during updates.

[0089] In some implementations, updating variables in the pending storage file may include: User bitmasks generated on the current date but not on the previous date will be replaced and updated in the pending storage file; and / or User bitmasks generated for both the current and previous dates, but with different bit patterns, will be replaced and updated in the pending storage file; and / or User bitmasks that were generated on the previous date but not on the current date will be updated in a reduced manner in the pending storage file.

[0090] The data processing method of this application embodiment can be implemented by the data processing system 100 of this application embodiment. Specifically, please refer to... Figure 3 Server 110 can be used to: replace and update user bitmasks generated on the current date but not on the previous date in the pending storage file; and / or replace and update user bitmasks generated on both the current date and the previous date but with different bit patterns in the pending storage file; and / or reduce the amount of user bitmasks generated on the previous date but not on the current date in the pending storage file.

[0091] This allows for a specific scheme to update variables based on the user bitmask.

[0092] Specifically, user bitmasks generated on the current date but not on the previous date are only generated within the current date and should be stored as new additions. User bitmasks generated on both the current and previous dates but with different bit patterns indicate that the user bitmasks were modified on the current date, causing changes in the values ​​of the corresponding bits. They should be used as replacements to replace the version generated on the previous date in the pending storage file. User bitmasks generated on the previous date but not on the current date indicate that the account corresponding to the user bitmask may have been canceled or reset, rendering the user information in the user bitmask invalid. Therefore, they can be deleted from the pending storage file as deletions.

[0093] In addition, in some implementations, the user identifier and bit pattern can be compared simultaneously for the user bitmask. The user identifier can be used to determine whether the same user's user bitmask was generated on the previous date, and the bit pattern can be used to determine whether the versions of the user bitmasks on the two dates are consistent if it is determined that the same user's user bitmask was generated.

[0094] In some implementations, user bitmasks generated on the previous date but not on the current date may include, prior to decremental updates in the pending storage file: Filter out user bitmasks with all zero bits from all user bitmasks generated on the previous date.

[0095] The data processing method of this application embodiment can be implemented by the data processing system 100 of this application embodiment. Specifically, please refer to... Figure 3 Server 110 can be used to filter out user bitmasks with all zero bits in all user bitmasks generated on the previous date.

[0096] This avoids generating meaningless deletion commands.

[0097] Specifically, when obtaining a user bitmask generated within the previous date, user bitmasks with all zero bits can be filtered out, that is, user bitmasks where all bits are zero. Therefore, when comparing by date, deletion instructions will not be generated for these user bitmasks with all zero bits.

[0098] In some implementations, before updating variables in the pending storage file, based on all user bitmasks generated on the current date and all user bitmasks generated on the previous date, the process may include: Get the first total data volume of all user bitmasks generated on the current date, and the second total data volume of all user bitmasks generated on the previous date; If the first total data volume is greater than or equal to the first data volume threshold, and the second total data volume is greater than or equal to the second data volume threshold, then it is determined to start updating the variables of the storage file to be stored.

[0099] The data processing method of this application embodiment can be implemented by the data processing system 100 of this application embodiment. Specifically, please refer to... Figure 3 Server 110 can be used to: obtain a first total data volume of all user bitmasks generated on the current date, and a second total data volume of all user bitmasks generated on the previous date; and determine to start updating variables of the target storage file if the first total data volume is greater than or equal to a first data volume threshold and the second total data volume is greater than or equal to a second data volume threshold.

[0100] This avoids data errors when generating user information storage files.

[0101] Specifically, in one implementation, after determining all user bitmasks in the partition for the current day, all user bitmasks in the partition for the previous day can be determined. Then, the total data volume in the two partitions is compared with a threshold. In this way, when importing data on a daily basis to generate user information storage files, it can be determined that the data in both partitions are non-empty and complete. This avoids the situation where empty or incomplete data is used to calculate and generate the final user information storage file when the user information storage file is generated on server 110 due to incompleteness or abnormal interruption. This also prevents the user data from being mistakenly marked as "deleted" in a centralized manner, making it impossible to call the user bitmask corresponding to the user.

[0102] In some implementations, the data processing method may include: Obtain incremental data of the user bitmask; The incremental data is fused with the corresponding existing version at the bit level to obtain the fused user bitmask. Update the corresponding user bitmask storage data based on the merged user bitmask.

[0103] The data processing method of this application embodiment can be implemented by the data processing system 100 of this application embodiment. Specifically, please refer to... Figure 3 Server 110 can be used to: acquire incremental data of the user bitmask; perform bit-level fusion processing between the incremental data and the corresponding existing version to obtain the fused user bitmask; and update the corresponding user bitmask storage data based on the fused user bitmask.

[0104] This makes operations that frequently update user datasets inherently idempotent.

[0105] Specifically, in one implementation, the user information storage file can be polled at a preset polling frequency to check if there is corresponding incremental data for the user bitmask. The update frequency of the incremental data can be greater than the daily update frequency but less than the preset polling frequency. The update frequency of the incremental data can be 1 hour, meaning that every hour, all user data acquired within that hour is used to update the user dataset. The preset polling frequency can be 5 minutes per polling, meaning that after completing one check of incremental data for the user bitmask, another check of incremental data for the user bitmask is performed after 5 minutes, and this cycle repeats.

[0106] In one implementation, the existing version of the same user bitmask is 010000000, and the corresponding incremental data is 000010000. The two bitmasks are subjected to a bitwise OR operation, and the resulting fused user bitmask is 010010000.

[0107] Based on the above implementation method, when incremental data is determined to exist, the incremental data of the user bitmask can be bitwise ORed with the existing version to obtain a fused user bitmask; then, the user bitmask storage data is updated according to the fused user bitmask. Since the bitwise OR operation always results in 010010000 regardless of how many times it is repeated, it is inherently idempotent. This eliminates the need for heavyweight operations such as reading, parsing, and merging JSON lists, and also removes the risk of dirty writes during concurrent updates.

[0108] In some implementations, when incremental data is determined to exist, performing bit-level fusion processing between the incremental data and the corresponding existing version may include: If at least one new tag data of a user is obtained, determine whether the user dimension corresponding to the at least one new tag data has a first preset dimension; If at least one new label data is found to have a first preset dimension in the user dimension, the preset bits corresponding to the first preset dimension are set in the all-zero bit mask, and the preset bits corresponding to the second preset dimension are set to zero, to generate incremental data of the user bit mask.

[0109] The data processing method of this application embodiment can be implemented by the data processing system 100 of this application embodiment. Specifically, please refer to... Figure 3Server 110 can be used to: when at least one new tag data of a user is obtained, determine whether the user dimension corresponding to the at least one new tag data has a first preset dimension; when it is determined that the user dimension corresponding to the at least one new tag data has a first preset dimension, set the preset bit corresponding to the first preset dimension in the all-zero bit mask, and set the preset bit corresponding to the second preset dimension to zero, to generate incremental data of the user bit mask.

[0110] This avoids overwriting results from other user dimensions generated through daily updates.

[0111] Specifically, in one implementation, the first preset dimension may include a real-name authentication dimension and an age exceeding limit dimension, and the second preset dimension may include a high-risk account dimension. When new tag data is obtained through hourly updates, if there is tag data corresponding to the real-name authentication dimension and the age exceeding limit dimension, valid values ​​will be generated for these dimensions; if there is tag data corresponding to the high-risk account dimension, no processing will be performed or a zero value will be generated for that dimension, so that in the incremental data of the generated user bitmask, only the bits corresponding to the real-name authentication dimension and the age exceeding limit dimension have valid values, while the bits corresponding to the high-risk account dimension are zero.

[0112] Based on the above, the user information corresponding to the first preset dimension can be considered as needing to be confirmed in a timely manner and having relatively low importance, while the user information corresponding to the second preset dimension can be considered as needing to be fully confirmed and having high importance. Furthermore, the user information corresponding to the second preset dimension will be updated and maintained in a daily update direction, so that daily updates and hourly updates can be performed on different user dimensions respectively.

[0113] In some implementations, the data processing method may include: Get the update time corresponding to all incremental data of the user bitmask within the day, and the last update time of the user bitmask storage data; If the last update time of the user bitmask storage data is determined to be no earlier than the update time of all incremental data, the last update time of the user bitmask storage data is updated to the latest update time of all incremental data, and the current round of incremental data detection process is terminated, waiting for the next round of detection process to be triggered.

[0114] The data processing method of this application embodiment can be implemented by the data processing system 100 of this application embodiment. Specifically, please refer to... Figure 3Server 110 can be used to: obtain the update time corresponding to all incremental data of user bitmask within the day, and the last update time of user bitmask storage data; if it is determined that the last update time of user bitmask storage data is not earlier than the update time corresponding to all incremental data, update the last update time of user bitmask storage data to the latest one of the update times corresponding to all incremental data, terminate the current round of incremental data detection process, and wait for the next round of detection process to be triggered.

[0115] In this way, when updating data hourly, the duplicate processing of user data within the same hourly partition can be avoided.

[0116] Specifically, when updating data hourly, the user data acquired each hour can be stored in a corresponding hourly partition, and an hourly update time will be generated after the data is updated. When polling to detect incremental data, the hourly update time of the latest user information storage file can be detected, and then compared with the most recent update time among all hourly partitions storing user data within the same day. If the user information storage file has been updated according to the most recent hourly partition, the comparison result is that the former is the most recent, or the two are the same, indicating that the current user information storage file is the latest version. The current polling detection does not need to further detect the incremental data of each user bitmask, but directly terminates the current detection and waits for the next round of polling detection.

[0117] In some implementations, the data processing method may include: In response to the tag cancellation command, the specified bits of the merged user bitmask are cleared according to the preset clear bitmask.

[0118] The data processing method of this application embodiment can be implemented by the data processing system 100 of this application embodiment. Specifically, please refer to... Figure 3 Server 110 can be used to: respond to tag revocation commands and clear specified bits of the merged user bitmask according to the preset clear bitmask.

[0119] In this way, it is possible to meet the need to revoke previously updated tags in some business scenarios.

[0120] Specifically, the specified bit clearing process can be performed by first performing a bitwise AND operation on the merged user bitmask according to the preset clear bitmask, and then inverting the bits to obtain data that is the same as the user bitmask before the fusion process, thereby achieving the effect of canceling the previously updated label; this data (inverted result) can then be fused bitwise with the incremental data of the user bitmask that needs to be updated again, thereby achieving the effect of updating with the new incremental data.

[0121] In some implementations, the data processing method may include: In the user dimension corresponding to all non-zero bits of the filtered user bitmask, generate the filtering reason log for the filtered user bitmask based on the one with the highest priority under the preset sorting rules.

[0122] The data processing method of this application embodiment can be implemented by the data processing system 100 of this application embodiment. Specifically, please refer to... Figure 3 Server 110 can be used to: generate a filtering reason log for the filtered user bitmask based on the user dimension corresponding to all non-zero bits of the filtered user bitmask, according to the one with the highest priority under the preset sorting rules.

[0123] This makes it easier to identify the main reasons why a specific user was filtered out in the filtering process.

[0124] Specifically, in one implementation, according to the left-to-right position order in the user bitmask, the user dimensions may include high-risk account dimension, real-name authentication dimension, age limit exceeding dimension, important person dimension, refund behavior dimension, low probability assessment dimension, medium probability assessment dimension, high probability assessment dimension, and facial recognition dimension. Each user dimension has a corresponding priority: high-risk account dimension is 9, real-name authentication dimension is 1, age limit exceeding dimension is 2, important person dimension is 3, refund behavior dimension is 4, low probability assessment dimension is 5, medium probability assessment dimension is 6, high probability assessment dimension is 7, and facial recognition dimension is 8.

[0125] Based on the above, if the obtained comparison bitmask is 010010000, it means that the real-name authentication dimension and the refund behavior dimension are matched. The priority of the refund behavior dimension is higher than that of the real-name authentication dimension. Therefore, the user's filtering reason log will be generated based on the refund behavior dimension. For example, in the scenario of filtering minors, the corresponding filtering reason log may include information such as "This user has had a refund behavior and is suspected to be a minor".

[0126] In another implementation, all user dimensions can be grouped into broad categories, each with a priority level. Then, based on the categories of all matched user dimensions in the comparison bitmask, the category with the highest priority is determined first. Within that category, the user dimension with the highest priority is used to determine the filtering reason log. Category grouping can include mandatory compliance filtering groups and operational discretionary filtering groups, where the priority of the mandatory compliance filtering group can be higher than that of the operational discretionary filtering group.

[0127] In some implementations, the data processing method may include: When the push program is inactive, it is determined whether the current condition meets the first preset release condition based on the first user configuration information and the first push channel. The first preset release condition includes the first filter switch value being false and the first push channel being included in the first channel set. The first user configuration information includes the user dataset and the first filter switch value. If it is determined that the first preset release condition is not met, according to the saved user parameters and the first user information file, the corresponding user identifier is obtained for each of the multiple users, and the user whose corresponding user identifier cannot be obtained is released. The user identifier is used to determine the user bitmask for bitwise AND operation. After completing the first filtering process, users corresponding to the user bitmasks that pass the filter are allowed to pass, and corresponding filtering reason logs are generated based on the user bitmasks that fail the filter. Send the corresponding first push message to all users who have been granted access.

[0128] The data processing method of this application embodiment can be implemented by the data processing system 100 of this application embodiment. Specifically, please refer to... Figure 3 Server 110 can be used to: when the push program is inactive, determine whether the current condition meets the first preset release condition based on the first user configuration information and the first push channel. The first preset release condition includes the first filter switch value being false and the first push channel being included in the first channel set. The first user configuration information includes the user dataset and the first filter switch value. If it is determined that the current condition does not meet the first preset release condition, obtain the corresponding user identifier for each of the multiple users according to the saved user parameters and the first user information file, and release users whose corresponding user identifiers cannot be obtained. The user identifier is used to determine the user bitmask for bitwise AND operation. After completing the first filtering process, release the users corresponding to the user bitmasks that pass the filter, and generate the corresponding filtering reason log based on the user bitmasks that do not pass the filter. Push the corresponding first push message to all users who have been released.

[0129] In this way, content can be pushed in offline scenarios.

[0130] In some implementations, the data processing method may include: When the push notification program is active, it is determined whether the current condition meets the second preset release condition based on at least one message queue obtained. The second preset release condition includes that the user identifier in at least one message queue is empty. If it is determined that the current condition does not meet the second preset release condition, the third preset release condition is determined based on the second user configuration information and the second push channel. The third preset release condition includes the second filter switch value being false and the second push channel being included in the second channel set. The second user configuration information includes the user dataset and the second filter switch value. If it is determined that the current condition does not meet the third preset release condition, in at least one message queue, the corresponding user identifier is obtained for each user. The user identifier is used to determine the user bitmask for bitwise AND operation. After completing the first filtering process, users corresponding to the user bitmasks that pass the filter are allowed to pass, and corresponding filtering reason logs are generated based on the user bitmasks that fail the filter. A second push message will be sent to all users who have been granted access.

[0131] The data processing method of this application embodiment can be implemented by the data processing system 100 of this application embodiment. Specifically, please refer to... Figure 3 Server 110 can be used to: when the push program is active, determine whether the current condition meets the second preset release condition based on at least one message queue, the second preset release condition including that the user identifier in at least one message queue is empty; if it is determined that the current condition does not meet the second preset release condition, determine whether the current condition meets the third preset release condition based on the second user configuration information and the second push channel, the third preset release condition including that the second filter switch value is false and the second push channel is included in the second channel set, the second user configuration information including the user dataset and the second filter switch value; if it is determined that the current condition does not meet the third preset release condition, obtain the corresponding user identifier for each user in at least one message queue, the user identifier is used to determine the user bitmask for bitwise AND operation; after completing the first filtering process, allow the users corresponding to the user bitmasks that pass the filtering, and generate the corresponding filtering reason log based on the user bitmasks that do not pass the filtering; push the corresponding second push message to all users who have been allowed.

[0132] In this way, content can be pushed in online scenarios.

[0133] In addition, the differences in content push in offline and real-time scenarios in this application can be reflected in Table 2 below:

[0134] Table 2 In some implementations, the data processing method may include: After the first filtering process is completed, the user bitmask that has passed the filtering process is subjected to multi-level filtering process in sequence. The multi-level filtering process includes multiple second filtering processes. Each second filtering process corresponds to a different filtering condition. After the first filtering process and the second filtering process are completed, the user bitmask has a corresponding filtering reason value. After completing the multi-level filtering process, the final filtering reason value corresponding to the user bitmask is generated based on the maximum value among all filtering reason values ​​of the user bitmask. Send messages to users whose final filter reason value is empty (user bitmask), and generate filter reason logs for users whose final filter reason value is not empty (user bitmask).

[0135] The data processing method of this application embodiment can be implemented by the data processing system 100 of this application embodiment. Specifically, please refer to... Figure 3 Server 110 can be used to: after completing the first filtering process, sequentially perform multi-level filtering on the filtered user bitmasks, the multi-level filtering process including multiple second filtering processes, each second filtering process corresponding to different filtering conditions, and after completing the first and second filtering processes, the user bitmask has a corresponding filtering reason value; after completing the multi-level filtering process, generate a final filtering reason value corresponding to the user bitmask based on the maximum value among all the filtering reason values ​​of the user bitmask; push messages to users corresponding to user bitmasks with an empty final filtering reason value, and generate filtering reason logs for user bitmasks with a non-empty final filtering reason value.

[0136] In this way, the effect of filtering and judging from multiple factors can be achieved.

[0137] Specifically, in one implementation, multiple second filtering processes may include Darwin control group filtering, duplicate content filtering, and channel frequency filtering. After completing the first filtering process, the user bitmask can be sequentially subjected to Darwin control group filtering, duplicate content filtering, and channel frequency filtering. After each filtering process, a string corresponding to the `filter_reason` field is generated, allowing the filtering result to be passed down the entire filtering chain through a unified `filter_reason` string field. If the string is empty, it indicates that the current filtering result is allowed; if the string is non-zero, it forms the corresponding filtering reason. All filtering stages can be accumulated using the `max()` function (taking the highest priority filtering reason in lexicographical order), meaning that among all filtering stages, the filtering reason log is generated based on the highest priority filtering reason.

[0138] It's important to note that the core semantic of the `filter_reason` field is an existence marker (a non-empty field indicates filtering), and its specific value is used for log readability. The precise filter type is recorded through a separate `filter_type` field, allowing the data side to categorize and statistically analyze the proportion and trends of various filtering reasons.

[0139] In some implementations, the data processing method may include: Based on the type of the current business scenario, a corresponding filter bitmask is determined. Different types of business scenarios have different filter bitmasks.

[0140] The data processing method of this application embodiment can be implemented by the data processing system 100 of this application embodiment. Specifically, please refer to... Figure 3 Server 110 can be used to: determine a corresponding filter bitmask based on the type of the current business scenario. Different types of business scenarios have different filter bitmasks.

[0141] In this way, filtering metrics can be flexibly configured according to different business scenarios.

[0142] Specifically, in one implementation, the first filtering process is used to filter and determine the minor population. The business scenario and the corresponding default filter bitmask can be represented by the following Table 3:

[0143] Table 3 Here, minor_flag represents the channel-level filtering switch. minor_flag being "false" indicates that the channel-level filtering switch is off, and minor_flag being "true" indicates that the channel-level filtering switch is on; minor_mask represents the default filter bitmask used for filtering and determining the minor population.

[0144] In one implementation, if it is necessary to modify the default filter bitmask corresponding to the business scenario, the minor_flag and minor_mask fields corresponding to the business scenario can be modified directly according to Table 3 above, so that no code changes or service restarts are required, and the new configuration can take effect immediately in the next batch of push tasks.

[0145] In some implementations, the data processing method may include: In response to the dimension addition instruction, add the number of bits corresponding to the dimension to be added to the end of the filter bitmask and the user bitmask.

[0146] The data processing method of this application embodiment can be implemented by the data processing system 100 of this application embodiment. Specifically, please refer to... Figure 3Server 110 can be used to: respond to dimension addition instructions by adding a number of bits to the end of the filter bitmask and the user bitmask, corresponding to the number of dimensions to be added.

[0147] In this way, the accuracy of user filtering can be improved without modifying the filtering engine code.

[0148] Specifically, a new user dimension needs to be added as the device fingerprint mark dimension. Then, an item can be added to the FULL_MINOR_FILTER_REASON_LIST in the configuration file: {"rank": 10, "reason": "device_fingerprint"}, so that MINOR_MASK_LEN is automatically adjusted from the original 9 to 10. When generating user bitmasks for user data provided by the upstream system, the new dimension can be encoded into the 10th bit of the mask in the list file (bit[9]). In tasks that require using the device fingerprint mark dimension for filtering, the user bitmask can be manually adjusted (e.g., from "010111100" (9 bits) to "0101111001" (10 bits, with the last bit 1 to enable the new dimension). In filtering, the bitwise AND operation comparison mechanism of the filtering engine and the logic of generating filtering reason logs by bit scanning are naturally compatible with the new length mask and do not require any modification.

[0149] In some implementations, the data processing method may include: In response to task instructions, determine at least one target filtering dimension; A filter bitmask is generated by setting bits according to at least one determined target filtering dimension.

[0150] The data processing method of this application embodiment can be implemented by the data processing system 100 of this application embodiment. Specifically, please refer to... Figure 3 Server 110 can be used to: respond to task instructions, determine at least one target filtering dimension; perform bit setting processing according to the determined at least one target filtering dimension, and generate a filter bitmask.

[0151] In this way, the user dimensions that need to be filtered can be flexibly configured according to the needs of content delivery.

[0152] In some implementations, the data processing method may include: The filter bitmask and user bitmask are stored in binary or decimal format.

[0153] The data processing method of this application embodiment can be implemented by the data processing system 100 of this application embodiment. Specifically, please refer to... Figure 3Server 110 can be used to store filter bitmasks and user bitmasks in binary or decimal format.

[0154] This can improve the comparison efficiency of bitwise AND operations.

[0155] Specifically, storing the filter bitmask and user bitmask in binary or decimal format allows for direct extraction and comparison during the initial filtering process, improving the efficiency of comparisons performed via bitwise AND operations. Furthermore, storing in decimal format avoids the overhead of string-to-integer conversion.

[0156] Optionally, this application also provides a computer device, including a memory and a processor, wherein the memory stores a computer program, and the processor executes the computer program to implement the steps in the above-described method embodiments.

[0157] Figure 4 This is a schematic diagram of the structure of a computer device provided in an embodiment of this application. The computer device may be a terminal or a server. Figure 4 As shown, the computer device 300 includes a processor 301 with one or more processing cores, a memory 302 with one or more computer-readable storage media, and a computer program stored in the memory 302 and executable on the processor. The processor 301 is electrically connected to the memory 302. Those skilled in the art will understand that the computer device structure shown in the figures does not constitute a limitation on the computer device, and may include more or fewer components than shown, or combine certain components, or have different component arrangements.

[0158] The processor 301 is the control center of the computer device 300. It connects various parts of the computer device 300 through various interfaces and lines. By running or loading software programs and / or modules stored in the memory 302, and calling data stored in the memory 302, it performs various functions of the computer device 300 and processes data, thereby performing overall processing of the computer device 300.

[0159] In this embodiment, the processor 301 in the computer device 300 loads the instructions corresponding to the processes of one or more computer programs into the memory 302 according to the following steps, and the processor 301 runs the computer programs stored in the memory 302 to realize various functions: 03: Obtain the filter bitmask and user dataset. Each bit in the filter bitmask corresponds to a user dimension. The user dimension corresponding to the bit in the filter bitmask that is in the preset filter state is the target filter dimension. The user dataset includes multiple user bitmasks. The user bitmask and the filter bitmask have the same length, and the bits in the same position correspond to the same user dimension. The bit in the user bitmask that is in the preset label state corresponds to a user label. 04: Based on the bit states of multiple user bitmasks and filter bitmasks respectively, determine whether the corresponding user matches the target filter dimension and obtain the comparison result; 05: If it is determined that some of the comparison results match the target filtering dimension, the corresponding user bitmask is subjected to the first filtering process.

[0160] For details on the implementation of each of the above operations, please refer to the previous examples, which will not be repeated here.

[0161] Optional, such as Figure 4 As shown, the computer device 300 also includes: a display screen 303, a radio frequency circuit 304, an audio circuit 305, an input unit 306, and a power supply 307. The processor 301 is electrically connected to the display screen 303, the radio frequency circuit 304, the audio circuit 305, the input unit 306, and the power supply 307. Those skilled in the art will understand that... Figure 4 The computer device structure shown does not constitute a limitation on the computer device and may include more or fewer components than shown, or combine certain components, or have different component arrangements.

[0162] The display screen 303 can be used to display a graphical user interface (GUI) and receive operation commands generated by the user interacting with the GUI. The display screen 303 may include a display panel and a touch panel. The display panel can be used to display information input by the user or information provided to the user, as well as various graphical user interfaces of the computer device. These graphical user interfaces can be composed of graphics, text, icons, video, and any combination thereof. The touch panel can be used to collect touch operations performed by the user on or near it (such as operations performed by the user using a finger, stylus, or any suitable object or accessory on or near the touch panel), generate corresponding operation commands, and execute the corresponding program. Optionally, the touch panel may include a touch detection device and a touch controller. The touch detection device detects the user's touch location and the signal generated by the touch operation, and transmits the signal to the touch controller. The touch controller receives touch information from the touch detection device, converts it into touch point coordinates, sends it to the processor 301, and can receive and execute commands from the processor 301. The touch panel can cover the display panel. When the touch panel detects a touch operation on or near it, it transmits the information to the processor 301 to determine the type of touch event. Subsequently, the processor 301 provides corresponding visual output on the display panel according to the type of touch event. In this embodiment, the touch panel and the display panel can be integrated into the display screen 303 to achieve input and output functions. However, in some embodiments, the touch panel and the display screen 303 can be implemented as two independent components to achieve input and output functions. That is, the display screen 303 can also be used as part of the input unit 306 to achieve input functions.

[0163] The radio frequency circuit 304 can be used to transmit and receive radio frequency signals to establish wireless communication with network devices or other computer devices, and to transmit and receive signals with network devices or other computer devices.

[0164] Audio circuitry 305 can be used to provide an audio interface between a user and a computer device via a speaker and a microphone. Audio circuitry 305 converts received audio data into electrical signals, transmits them to the speaker, and the speaker converts them into sound signals for output. Conversely, the microphone converts collected sound signals into electrical signals, which are then received by audio circuitry 305, converted back into audio data, and output to processor 301 for processing. The audio data is then transmitted via radio frequency circuitry 304 to, for example, another computer device, or output to memory 302 for further processing. Audio circuitry 305 may also include an earphone jack to facilitate communication between peripheral headphones and the computer device.

[0165] The input unit 306 can be used to receive input numbers, characters, or object feature information (such as fingerprints, irises, facial information, etc.), and to generate keyboard, mouse, joystick, optical, or trackball signal inputs related to user settings and function control.

[0166] Power supply 307 is used to supply power to various components of computer device 300. Optionally, power supply 307 can be logically connected to processor 301 through a power management system, thereby enabling functions such as charging, discharging, and power consumption management through the power management system. Power supply 307 may also include one or more DC or AC power supplies, recharging systems, power fault detection circuits, power converters or inverters, power status indicators, and other arbitrary components.

[0167] although Figure 4 As not shown in the diagram, computer equipment 300 may also include a camera, sensor, wireless fidelity module, Bluetooth module, etc., which will not be described in detail here.

[0168] This application also provides a computer-readable storage medium for storing a computer program. This computer-readable storage medium can be applied to a computer device, and the computer program causes the computer device to execute the corresponding flow in the data processing method of the embodiments of this application; for brevity, further details are omitted here.

[0169] This application also provides a computer program product including computer instructions stored in a computer-readable storage medium. A processor of a computer device reads the computer instructions from the computer-readable storage medium and executes the computer instructions, causing the computer device to perform the corresponding flow in the data processing method of the embodiments of this application. For simplicity, further details are omitted here.

[0170] This application also provides a computer program comprising computer instructions stored in a computer-readable storage medium. A processor of a computer device reads the computer instructions from the computer-readable storage medium and executes the computer instructions, causing the computer device to perform the corresponding flow in the data processing method of this application. For brevity, further details are omitted here.

[0171] It should be understood that the processor in this application may be an integrated circuit chip with signal processing capabilities. In implementation, the steps of the above method embodiments can be completed by integrated logic circuits in the processor's hardware or by instructions in software form. The processor described above can be a general-purpose processor, a digital signal processor (DSP), an application-specific integrated circuit (ASIC), a field-programmable gate array (FPGA), or other programmable logic devices, discrete gate or transistor logic devices, or discrete hardware components. It can implement or execute the methods, steps, and logic block diagrams disclosed in the embodiments of this application. The general-purpose processor can be a microprocessor or any conventional processor. The steps of the methods disclosed in the embodiments of this application can be directly embodied in the execution of a hardware decoding processor, or executed by a combination of hardware and software modules in the decoding processor. The software modules can be located in random access memory, flash memory, read-only memory, programmable read-only memory, electrically erasable programmable memory, registers, or other mature storage media in the art. This storage medium is located in memory, and the processor reads information from the memory and, in conjunction with its hardware, completes the steps of the above method.

[0172] It is understood that the memory in the embodiments of this application can be volatile memory or non-volatile memory, or may include both volatile and non-volatile memory. The non-volatile memory can be read-only memory (ROM), programmable read-only memory (PROM), erasable programmable read-only memory (EPROM), electrically erasable programmable read-only memory (EEPROM), or flash memory. The volatile memory can be random access memory (RAM), which is used as an external cache. By way of example, but not limitation, many forms of RAM are available, such as Static Random Access Memory (SRAM), Dynamic Random Access Memory (DRAM), Synchronous DRAM (SDRAM), Double Data Rate SDRAM (DDR SDRAM), Enhanced Synchronous DRAM (ESDRAM), Synchlink DRAM (SLDRAM), and Direct Rambus RAM (DR RAM). It should be noted that the memory used in the systems and methods described herein is intended to include, but is not limited to, these and any other suitable types of memory.

[0173] Those skilled in the art will recognize that the units and algorithm steps of the various examples described in conjunction with the embodiments disclosed herein can be implemented in electronic hardware, or a combination of computer software and electronic hardware. Whether these functions are implemented in hardware or software depends on the specific application and design constraints of the technical solution. Those skilled in the art can use different methods to implement the described functions for each specific application, but such implementation should not be considered beyond the scope of this application.

[0174] Those skilled in the art will understand that, for the sake of convenience and brevity, the specific working processes of the systems, devices, and units described above can be referred to the corresponding processes in the foregoing method embodiments, and will not be repeated here.

[0175] In the embodiments of this application, the terms "module" or "unit" refer to a computer program or part of a computer program that has a predetermined function and works with other related parts to achieve a predetermined goal, and can be implemented wholly or partially using software, hardware (such as processing circuitry or memory), or a combination thereof. Similarly, a processor (or multiple processors or memory) can be used to implement one or more modules or units. Furthermore, each module or unit can be part of an overall module or unit that includes the functionality of that module or unit.

[0176] In the several embodiments provided in this application, it should be understood that the disclosed systems, apparatuses, and methods can be implemented in other ways. For example, the apparatus embodiments described above are merely illustrative; for instance, the division of units is only a logical functional division, and in actual implementation, there may be other division methods. For example, multiple units or components may be combined or integrated into another system, or some features may be ignored or not executed. Furthermore, the coupling or direct coupling or communication connection shown or discussed may be through some interfaces; the indirect coupling or communication connection between apparatuses or units may be electrical, mechanical, or other forms.

[0177] The units described as separate components may or may not be physically separate. The components shown as units may or may not be physical units; that is, they may be located in one place or distributed across multiple network units. Some or all of the units can be selected to achieve the purpose of this embodiment according to actual needs.

[0178] In addition, the functional units in this application can be integrated into one processing unit, or each unit can exist physically separately, or two or more units can be integrated into one unit.

[0179] If the aforementioned functions are implemented as software functional units and sold or used as independent products, they can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of this application, in essence, or the part that contributes to the prior art, or a portion of the technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions to cause a computer device (which may be a personal computer or a server) to execute all or part of the steps of the methods described in the various embodiments of this application. The aforementioned storage medium includes various media capable of storing program code, such as USB flash drives, portable hard drives, ROM, RAM, magnetic disks, or optical disks.

[0180] The above description is merely a specific embodiment of this application, but the scope of protection of this application is not limited thereto. Any variations or substitutions that can be easily conceived by those skilled in the art within the scope of the technology disclosed in this application should be included within the scope of protection of this application. Therefore, the scope of protection of this application should be determined by the scope of the claims.

Claims

1. A data processing method, characterized in that, The data processing method includes: Obtain a filter bitmask and a user dataset. Each bit in the filter bitmask corresponds to a user dimension. The user dimension corresponding to the bit in the filter bitmask that is in a preset filtering state is the target filtering dimension. The user dataset includes multiple user bitmasks. The user bitmasks and the filter bitmasks have the same length, and the bits in the same position correspond to the same user dimension. The bits in the user bitmask that are in a preset label state correspond to a user label. Based on the bit states of the multiple user bitmasks and the filter bitmasks respectively, determine whether the corresponding user matches the target filter dimension and obtain the comparison result; If it is determined that a portion of the comparison result identifier matches the target filtering dimension, the corresponding user bitmask is subjected to a first filtering process.

2. The data processing method as described in claim 1, characterized in that, The user dimension includes at least one of the following: identity attribute dimension, behavioral risk control dimension, and probability assessment dimension: the identity attribute dimension is used to represent the user's basic identity identification information, the behavioral risk control dimension is used to represent the user's historical behavioral tag information, and the probability assessment dimension is used to represent the probability level information obtained based on user behavior prediction.

3. The data processing method as described in claim 2, characterized in that, The data processing method includes: The acquired raw user data is subjected to probability prediction processing to determine the target probability assessment dimension. The user dimension includes multiple probability assessment dimensions, and the target probability assessment dimension is one of the multiple probability assessment dimensions. Each probability assessment dimension has a corresponding probability prediction degree. Set the bits in the user bitmask corresponding to the target probability evaluation dimension, and clear the bits corresponding to the other probability evaluation dimensions.

4. The data processing method as described in claim 1, characterized in that, The data processing method includes: Acquire tag data for multiple users, where each user has multiple tag data, and each tag data corresponds to a user dimension; The multiple tag data corresponding to each user are merged to generate a corresponding user bitmask; The multiple user bitmasks are stored to generate a user information storage file, which includes the user dataset.

5. The data processing method as described in claim 4, characterized in that, Before merging the multiple tag data corresponding to each user, the process includes: Perform data quality verification on the tag data of the aforementioned multiple users; After verifying that the tag data of the multiple users has passed the verification, it is determined that the multiple tag data corresponding to each user will be merged.

6. The data processing method as described in claim 5, characterized in that, The data quality verification of the tag data of the multiple users includes: Detect whether the number of tag data for multiple users under the same user dimension is greater than or equal to the corresponding data volume threshold, wherein the data volume threshold is determined by the user dimension and the data source type of the tag data; If the detection results for all user dimensions are yes, then the multiple tag data are determined to have passed the verification.

7. The data processing method as described in claim 6, characterized in that, The data processing method includes: If the detection result corresponding to some of the user dimensions is negative, the tag data of the multiple users are backtracked according to the corresponding user dimensions.

8. The data processing method as described in claim 4, characterized in that, The acquisition of tag data from multiple users includes: Obtain all primary keys for each user, where each primary key has multiple tag data corresponding to a user, and the multiple tag data corresponding to different primary keys of the same user are stored independently. The step of merging multiple tag data corresponding to each user to generate a corresponding user bitmask includes: Based on the multiple tag data corresponding to each of the primary keys, bitwise OR fusion processing is performed to generate the user bitmask corresponding to the user.

9. The data processing method as described in claim 4, characterized in that, The data processing method includes: Obtain the incremental data of the user bitmask; The incremental data is fused with the corresponding existing version at the bit level to obtain the fused user bitmask. Update the corresponding user bitmask storage data according to the fused user bitmask.

10. The data processing method as described in claim 9, characterized in that, Before performing bit-level fusion processing between the incremental data and the corresponding existing version, the process includes: If at least one new tag data of the user is obtained, determine whether the user dimension corresponding to the at least one new tag data has a first preset dimension; If the user dimension corresponding to the at least one new label data has the first preset dimension, the preset bit corresponding to the first preset dimension is set in the all-zero bit mask, and the preset bit corresponding to the second preset dimension is set to zero, to generate incremental data of the user bit mask.

11. The data processing method as described in claim 9, characterized in that, The data processing method includes: Obtain the update time corresponding to all incremental data of the user bitmask within the same day, and the last update time of the user bitmask storage data; If it is determined that the last update time of the user bitmask storage data is not earlier than the update time corresponding to all incremental data, the last update time of the user bitmask storage data is updated to the latest update time corresponding to all incremental data, and the current round of incremental data detection process is terminated, waiting for the next round of detection process to be triggered.

12. A data processing system, characterized in that, The data processing system includes at least one server, the server being used for: Obtain a filter bitmask and a user dataset. Each bit in the filter bitmask corresponds to a user dimension. The user dimension corresponding to the bit in the filter bitmask that is in a preset filtering state is the target filtering dimension. The user dataset includes multiple user bitmasks. The user bitmasks and the filter bitmasks have the same length, and the bits in the same position correspond to the same user dimension. The bits in the user bitmask that are in a preset label state correspond to a user label. Based on the bit states of the multiple user bitmasks and the filter bitmasks respectively, determine whether the corresponding user matches the target filter dimension and obtain the comparison result; If it is determined that a portion of the comparison result identifier matches the target filtering dimension, the corresponding user bitmask is subjected to a first filtering process.

13. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores a computer program adapted for loading by a processor to perform the data processing method as described in any one of claims 1-11.

14. A computer device, characterized in that, The computer device includes a processor and a memory, the memory storing a computer program, and the processor executing the data processing method according to any one of claims 1-11 by calling the computer program stored in the memory.

15. A computer program product comprising computer instructions, characterized in that, When the computer instructions are executed by the processor, they implement the data processing method according to any one of claims 1-11.

Images