Cluster codebook updating method and system, semantic label generation method and system

Abstract

Embodiments of the present specification provide a cluster codebook updating method and system, a semantic identifier generating method and system. The cluster codebook updating method comprises: obtaining a current user representation vector and a current cluster codebook, wherein the current cluster codebook is constructed according to historical user representation vectors, and comprises a plurality of cluster centers and corresponding state identifiers divided according to a cluster level; determining an uncovered user representation vector from the current user representation vector based on a distance between the current user representation vector and a center vector of the plurality of cluster centers; determining a new cluster center based on the uncovered user representation vector; merging the new cluster center and the plurality of cluster centers already in the current cluster codebook, updating the corresponding state identifiers of the cluster centers, and obtaining a target cluster codebook. The dynamic expansion and continuous updating of the cluster codebook are realized, the adaptability of the codebook to the evolution of the user group is enhanced, and the generation accuracy and efficiency of the user semantic identifier are improved.

Description

Technical Field

[0001] The embodiments in this specification relate to the technical field, and in particular to a clustering codebook update method, a clustering codebook update system, a semantic identifier generation method, and a semantic identifier generation system. Background Technology

[0002] With the expansion of recommendation systems and the continuous accumulation of user behavior data, clustering quantization encoding based on user representation vectors has become a key technology to support large-scale user semantic representation and efficient retrieval. The quality and update efficiency of the clustering codebook directly affect the encoding accuracy of user representations and the performance of downstream tasks.

[0003] Currently, related technologies typically employ residual quantization clustering to convert user behavior representations into continuous user representation vectors. These vectors are then clustered hierarchically across multiple levels, forming a cluster codebook composed of multiple cluster centers and their codes. The user representation vectors are discretized by calculating the distance to each cluster center. The cluster codebook remains fixed after training. However, to adapt to continuous changes in the user base, the codebook is usually retrained entirely or online clustering methods are used to incrementally update the cluster centers to accommodate new data distributions.

[0004] However, among the aforementioned technical methods, static clustering codebooks cannot respond to the continuous evolution of user groups. New user groups are forcibly mapped to existing cluster centers with semantically mismatched values, leading to a continuous degradation in encoding quality. While full retraining can cover all user representation vectors, it causes a large number of user encoding jumps, disrupting the historical correlation between the encodings established by downstream models and user behavior, resulting in poor accuracy and low generation efficiency of the generated semantic codes. Therefore, there is an urgent need for a clustering codebook update method to improve the accuracy and efficiency of semantic label generation. Summary of the Invention

[0005] In view of this, embodiments of this specification provide a clustering codebook update method. One or more embodiments of this specification also relate to a semantic identifier generation method, a clustering codebook update system, a semantic identifier generation system, a computing device, a computer-readable storage medium, and a computer program product, to solve the aforementioned technical problems.

[0006] According to a first aspect of the embodiments of this specification, a clustering codebook update method is provided, comprising: Obtain the current user representation vector and the current cluster codebook. The current cluster codebook is constructed based on the historical user representation vectors and includes multiple cluster centers divided according to the clustering level and the corresponding status identifiers. Based on the distance between the current user representation vector and the center vectors of multiple cluster centers, determine the uncovered user representation vectors from the current user representation vector; Based on the uncovered user representation vectors, new cluster centers are determined; The newly added cluster centers are merged with the existing cluster centers in the current cluster codebook, and the status flags of each cluster center are updated to obtain the target cluster codebook.

[0007] According to a second aspect of the embodiments of this specification, a semantic identifier generation method is provided, comprising: Obtain the target user representation vector; Based on the target clustering codebook, the target user representation vector is encoded and mapped to obtain the target semantic identifier. The target clustering codebook is updated according to the clustering codebook update method described above.

[0008] According to a third aspect of the embodiments of this specification, a clustering codebook update system is provided, comprising: The first acquisition module is used to acquire the current user representation vector and the current cluster codebook. The current cluster codebook is constructed based on the historical user representation vector and includes multiple cluster centers divided according to the clustering level and the corresponding status identifiers. The detection module is used to determine the uncovered user representation vectors from the current user representation vector based on the distance between the current user representation vector and the center vectors of multiple cluster centers; The training module is used to determine new cluster centers based on the uncovered user representation vectors; The update module is used to merge newly added cluster centers with multiple existing cluster centers in the current cluster codebook, and update the status flags corresponding to each cluster center to obtain the target cluster codebook.

[0009] According to a fourth aspect of the embodiments of this specification, a semantic identifier generation system is provided, comprising: The second acquisition module is used to acquire the target user representation vector; The encoding mapping module is used to encode and map the target user representation vector based on the target clustering codebook to obtain the target semantic identifier. The target clustering codebook is updated according to the clustering codebook update method described above.

[0010] According to a fifth aspect of the embodiments of this specification, a computing device is provided, comprising: Memory and processor; The memory is used to store computer-executable instructions, and the processor is used to execute the computer-executable instructions, which, when executed by the processor, implement the steps of the above method.

[0011] According to a sixth aspect of the embodiments of this specification, a computer-readable storage medium is provided that stores computer-executable instructions that, when executed by a processor, implement the steps of the above-described method.

[0012] According to a seventh aspect of the embodiments of this specification, a computer program product is provided, including computer-executable instructions that, when executed by a processor, implement the steps of the above-described method.

[0013] One embodiment of this specification implements a clustering codebook update method, comprising: obtaining the current user representation vector and the current clustering codebook, wherein the current clustering codebook is constructed based on historical user representation vectors, including multiple cluster centers divided according to clustering levels and corresponding status identifiers; determining uncovered user representation vectors from the current user representation vectors based on the distance between the current user representation vectors and the center vectors of the multiple cluster centers; determining new cluster centers based on the uncovered user representation vectors; merging the new cluster centers with the multiple existing cluster centers in the current clustering codebook, and updating the status identifiers corresponding to each cluster center to obtain the target clustering codebook.

[0014] By acquiring the current user representation vector and the current cluster codebook, which is divided into multiple cluster centers and corresponding state identifiers according to the clustering hierarchy, the codebook update process has a complete data and structural input foundation. By determining the uncovered user representation vector based on the distance between the current user representation vector and the center vectors of multiple cluster centers, the codebook coverage gap is accurately identified in a quantitative way, providing a clear target for codebook expansion. By determining the new cluster centers based on the uncovered user representation vectors, the codebook expansion direction is driven by the actual coverage gap, and the new cluster centers can accurately match the distribution characteristics of the uncovered user group. By merging the new cluster centers with multiple existing cluster centers in the current cluster codebook and updating the state identifiers corresponding to each cluster center to obtain the target cluster codebook, the codebook coverage capability is expanded while maintaining the stability of existing cluster centers and their encoded semantics. This avoids encoding jumps from disrupting the historical association memory of downstream recommendation models, thereby achieving dynamic expansion and continuous updating of the cluster codebook while maintaining the stability of the existing encoded semantics. This enhances the codebook's adaptability to the evolution of user groups, and thus improves the accuracy and efficiency of user semantic identifier generation. Attached Figure Description

[0015] Figure 1 This is a flowchart illustrating a clustering codebook update method provided in one embodiment of this specification; Figure 2 This is a schematic diagram of the scheduling framework of a clustering codebook update method provided in one embodiment of this specification; Figure 3This is a flowchart of a semantic identifier generation method provided in one embodiment of this specification; Figure 4 This is a schematic diagram of the structure of a clustering codebook update system provided in one embodiment of this specification; Figure 5 This is a schematic diagram of the structure of a semantic tag generation system provided in one embodiment of this specification; Figure 6 This is a structural block diagram of a computing device provided in one embodiment of this specification. Detailed Implementation

[0016] Many specific details are set forth in the following description to provide a full understanding of this specification. However, this specification can be implemented in many other ways than those described herein, and those skilled in the art can make similar extensions without departing from the spirit of this specification. Therefore, this specification is not limited to the specific implementations disclosed below.

[0017] The terminology used in one or more embodiments of this specification is for the purpose of describing particular embodiments only and is not intended to be limiting of the one or more embodiments of this specification. The singular forms “a,” “described,” and “the” as used in one or more embodiments of this specification and the appended claims are also intended to include the plural forms unless the context clearly indicates otherwise. It should also be understood that the term “and / or” as used in one or more embodiments of this specification refers to and includes any or all possible combinations of one or more associated listed items.

[0018] It should be understood that although the terms first, second, etc., may be used to describe various information in one or more embodiments of this specification, such information should not be limited to these terms. These terms are only used to distinguish information of the same type from one another. For example, first may also be referred to as second without departing from the scope of one or more embodiments of this specification, and similarly, second may also be referred to as first. Depending on the context, the word "if" as used herein may be interpreted as "when," "when," or "in response to a determination."

[0019] Furthermore, it should be noted that the user information (including but not limited to user device information, user personal information, etc.) and data (including but not limited to data used for analysis, data stored, data displayed, etc.) involved in one or more embodiments of this specification are obtained through open-source datasets or public datasets that comply with their license agreements, or are obtained with full authorization from the relevant parties. Moreover, the collection, use and processing of the relevant data must comply with the relevant laws, regulations and standards of the relevant countries and regions, and corresponding operation entry points are provided for users to choose to authorize or refuse.

[0020] First, the terms and concepts used in one or more embodiments of this specification will be explained.

[0021] Residual quantization clustering is a multi-level progressive quantization method that discretizes a continuous vector into a combination of multi-level encoded values ​​by progressively clustering and encoding the residual between the input vector and the quantization result of the previous level.

[0022] K-Means Clustering is a partition-based clustering algorithm that iteratively assigns samples to the clusters of the nearest cluster centers by pre-specifying the number of cluster centers, until the cluster centers converge or the preset number of iterations is reached.

[0023] K-Means++ is an initial cluster center selection strategy for K-means clustering. It selects samples that are far apart from each other as initial cluster centers according to the distance probability distribution, thereby improving the stability and convergence speed of the clustering results.

[0024] A deep neural network (DNN) is an artificial neural network containing multiple hidden layers that performs hierarchical feature extraction and representation learning on input data through layer-by-layer nonlinear transformations.

[0025] Kernel density estimation (KDE) is a non-parametric probability density estimation method that estimates the continuous probability density distribution of data in space by placing kernel functions around data points and summing them.

[0026] Principal Component Analysis (PCA) is a data dimensionality reduction method that uses linear transformation to project high-dimensional data onto several orthogonal directions with the largest variance in order to preserve the main structural information of the data.

[0027] The elbow method is a method for determining the number of clusters. It compares the clustering errors under different candidate clustering numbers and determines the number of clusters corresponding to the inflection point where the decreasing trend of clustering error changes significantly.

[0028] Outlier detection is a technique for identifying data objects in a dataset that significantly deviate from the main distribution, used to discover anomalous data that does not conform to the behavior patterns of the vast majority of data.

[0029] The Local Outlier Factor (LOF) is a density-based outlier detection algorithm that calculates the degree of outlier status of a sample by comparing its local density with that of its neighboring samples. Samples with a local density significantly lower than that of their neighboring samples are identified as outliers.

[0030] Hierarchical clustering is a clustering method that constructs a hierarchical clustering structure by iteratively merging or splitting clusters. It does not require pre-specifying the number of clusters and can truncate at different levels to obtain clustering results of different granularities.

[0031] Density-based clustering is a clustering method based on sample density distribution. It identifies densely connected sample regions as clusters, can discover cluster structures of arbitrary shapes, and marks samples in low-density regions as noise points.

[0032] The silhouette coefficient is a clustering effectiveness evaluation index that comprehensively measures the cohesion of a sample with its own cluster and the separation with neighboring clusters. The closer the value is to one, the better the clustering effect.

[0033] Median Absolute Deviation (MAD) is a robust statistic of dispersion, obtained by calculating the median of the absolute values ​​of the differences between a sample and the median, and is insensitive to extreme outliers.

[0034] A cold start refers to the process of starting a system or model when there is a lack of available historical data or experience during the initial operation phase.

[0035] With the expansion of recommendation systems and the continuous accumulation of user behavior data, clustering quantization encoding based on user representation vectors has become a key technology to support large-scale user semantic representation and efficient retrieval. The quality and update efficiency of the clustering codebook directly affect the encoding accuracy of user representations and the execution effect of downstream tasks.

[0036] Currently, related technologies typically employ residual quantization clustering to convert user behavior representations into continuous user representation vectors. These vectors are then clustered hierarchically across multiple levels, forming a cluster codebook composed of multiple cluster centers and their codes. The user representation vectors are discretized by calculating the distance to each cluster center. The cluster codebook remains fixed after training. However, to address continuous changes in the user base, a full retraining of the codebook or incremental updates to the cluster centers using online clustering methods are typically employed to adapt to new user representation vector distributions.

[0037] However, static clustering codebooks cannot respond to the continuous evolution of the user group. Newly emerging user groups are forcibly mapped to existing cluster centers with semantically mismatched values, leading to a continuous degradation in encoding quality. While full retraining can cover all user representation vectors, it causes a large number of user codes to jump, disrupting the historical correlation between the encodings established by downstream models and user behavior. Online clustering updates can continuously adjust cluster centers, but this causes the cluster center positions to drift, gradually shifting the encoded semantics. None of the above methods can incrementally expand the clustering codebook to adapt to the evolution of the user group while maintaining the stability of the existing clustering codebook's encoded semantics.

[0038] This specification provides a clustering codebook update method, and also relates to a semantic identifier generation method, a clustering codebook update system, a semantic identifier generation system, a computing device, a computer-readable storage medium, and a computer program product, which are described in detail in the following embodiments.

[0039] Specifically, see Figure 1 , Figure 1 A flowchart of a clustering codebook update method according to an embodiment of this specification is shown, such as... Figure 1 As shown, the clustering codebook update method specifically includes the following steps.

[0040] Step 102: Obtain the current user representation vector and the current cluster codebook. The current cluster codebook is constructed based on the historical user representation vector and includes multiple cluster centers divided according to the clustering level and the corresponding status identifiers.

[0041] The clustering codebook update method provided in one or more embodiments of this specification can be applied to various scenarios that require discretization encoding and dynamic maintenance based on user representation vectors. It is particularly suitable for the dynamic maintenance and incremental update of user semantic identifiers in large-scale recommendation systems. For example, it can be applied to the scenario of incremental training and deployment of the quantization encoding model of user representation in the transaction service domain, and it can also be applied to other links involving the generation of user semantic identifiers using residual quantization clustering models, and it can also be applied to combined scenarios of the above-mentioned multiple scenarios.

[0042] The execution subject of the clustering codebook update method provided in the embodiments of this specification can be the server-side device of the recommendation system, the terminal device with data processing capabilities, or a system composed of server-side devices and terminal devices. The embodiments of this specification do not limit this.

[0043] User representation vectors are continuous vectors obtained by numerically representing user behavioral characteristics. They serve as the basic data units in the process of updating the clustering codebook and encoding mapping. Specifically, user representation vectors can include fixed-dimensional dense vectors output by deep neural network encoding multidimensional behavioral data of users within a preset time window. Behavioral data can include aggregated features of various behavioral types such as browsing, clicking, adding to cart, favoriting, and placing orders. User representation vectors can cover the representation vector set of all active users on the platform or a subset of users who meet preset activity conditions. User representation vectors can serve as the direct object for distance calculation and encoding mapping in the clustering codebook, and their distribution characteristics reflect the interest status and evolution trend of the user group. Furthermore, user representation vectors are the bridge connecting user behavior and discrete encoding. Through the clustering codebook, the continuous vector space is discretized into an encoding space with business semantics. The dimension and encoding precision of the user representation vector directly determine the representation quality of user semantic identifiers.

[0044] The current user representation vector is a user representation vector acquired within the current update cycle and used to evaluate the coverage of the current cluster codebook. Specifically, the current user representation vector can include the representation vectors of active users on the platform collected within the current time window. The collection time window can be the most recent statistical period, such as the most recent day or the most recent week. The current user representation vector can be sequentially linked to historical user representation vectors in the time dimension. The current user representation vector can be used as input data in the cluster codebook update process. By calculating the distance between it and the center vectors of existing cluster centers, it is determined whether the current cluster codebook covers the distribution range of the current user group. Furthermore, the difference in distribution between the current user representation vector and historical user representation vectors reflects the overall evolution trend of user group interests. When new user groups emerge or the interests of existing user groups diverge, the distribution of the current user representation vector will deviate from the clustering structure determined by the historical user representation vectors, thereby triggering the incremental update process of the cluster codebook.

[0045] Historical user representation vectors are used to construct the current clustering codebook, representing the behavioral characteristics and interest distribution of user groups within a historical time window. Specifically, historical user representation vectors can include user representation vectors corresponding to training samples collected during the current clustering codebook construction phase, or user representation vectors accumulated during the codebook's historical update cycle. These historical user representation vectors accumulate chronologically and participate in the construction and quality evaluation process of the clustering codebook. Historical user representation vectors can serve as input data for layer-by-layer clustering training, determining the cluster center positions at each clustering level by performing layer-by-layer clustering based on their distribution characteristics. Furthermore, the user group distribution reflected by historical user representation vectors constitutes the coverage benchmark of the current clustering codebook, and the coverage of historical user representation vectors by the current clustering codebook can serve as a basis for codebook quality evaluation and pre-deployment verification.

[0046] A clustering codebook is an encoded dictionary organized hierarchically by clustering levels, consisting of multiple cluster centers and their state identifiers. It is used to map continuous user representation vectors to discrete semantic identifiers. Specifically, the clustering codebook can include multiple clustering levels, each containing multiple cluster centers. Each cluster center occupies a corresponding encoding slot in the codebook and has a corresponding state identifier. The user representation vector is identified by calculating its distance to the cluster centers of each clustering level, determining the nearest cluster center in each level, and then combining the cluster center numbers of each level in hierarchical order to obtain the discrete semantic identifier corresponding to the user representation vector. The clustering codebook can be constructed using residual quantization clustering, where each clustering level further clusters the clustering residuals of the previous level to progressively refine the granularity of the user representation vector partitioning. Furthermore, the cluster codebook is the core data structure for generating user semantic identifiers. Its coverage and the quality of the cluster centers directly determine the encoding accuracy of user semantic identifiers. The cluster codebook needs to be dynamically expanded while maintaining the stability of existing encoded semantics in order to adapt to the ever-evolving user group.

[0047] The current cluster codebook is the currently active version of the cluster codebook and serves as the processing object and benchmark for the cluster codebook update process. Specifically, the current cluster codebook is constructed based on historical user representation vectors, including multiple cluster centers divided according to cluster levels and their corresponding state identifiers. The current cluster codebook can be the initial codebook constructed after cold start training or the target cluster codebook obtained after the last incremental update. The current cluster codebook can be used as input data for coverage detection and incremental updates. By calculating the distance between the current user representation vector and the center vectors of each cluster center in the current cluster codebook, user representation vectors not covered by the current cluster codebook are determined. Furthermore, the positions and state identifiers of each cluster center in the current cluster codebook constitute the benchmark state for codebook updates. During incremental updates, the positions of existing cluster centers remain unchanged to ensure that the encoding results of users with assigned semantic identifiers do not change due to codebook updates.

[0048] Clustering hierarchy is a multi-level, progressively refined clustering organization structure in the clustering codebook, with each clustering level corresponding to a granular partition of the user representation space. Specifically, the clustering hierarchy can include a progressive structure from the first level to the last level. The first level performs coarse-grained overall clustering of the user representation vectors, the second level further clusters the residual vectors generated by the first level's clustering, and so on. Subsequent levels sequentially perform finer-grained clustering of the residuals from the previous level's clustering. The cluster center numbers of each clustering level are combined in hierarchical order to form the user's semantic identifier.

[0049] In specific application scenarios, the total number of clustering levels can be determined based on actual business needs and clustering granularity requirements. For example, it can include 1 to 3 frozen levels at the beginning and the remaining dynamic levels. The cluster centers of the frozen levels correspond to the major categories of the user group and are generally relatively stable with little need for expansion. Furthermore, the number of clustering levels determines the encoding length and representation accuracy of the semantic identifier. More clustering levels result in higher semantic identifier discriminability, but also increase the corresponding codebook maintenance cost and computational complexity. In practical applications, a reasonable configuration is needed based on the user group size and business accuracy requirements. Each clustering level operates relatively independently. Adding a cluster center to one clustering level will not interfere with the codebook structure of other clustering levels. The purpose of adding a cluster center is to make the clustering of the corresponding clustering level more accurate; other clustering levels do not need adjustment if their clustering is already good.

[0050] Cluster centers are representative vectors for each cluster in the clustering codebook, used to characterize the central tendency and typical features of user representation vectors within the corresponding cluster. Specifically, cluster centers can be fixed-dimensional coordinate vectors determined through iterative optimization during the construction or updating of the clustering codebook. Each cluster center corresponds to an encoding slot. The user representation vector is assigned to a cluster center number by calculating the distance between it and the center vectors of each cluster center. User representation vectors corresponding to the same cluster center constitute the cluster corresponding to that cluster center. Cluster centers can serve as reference benchmarks for the discretization encoding of user representation vectors, and their positional accuracy directly affects the representation quality of semantic identifiers. Furthermore, during the incremental update of the clustering codebook, keeping the positions of existing cluster centers unchanged is a key constraint to ensure that the historical association memory of downstream models is not destroyed. Newly added cluster centers are only used to cover user groups that were not previously fully represented, without making global adjustments to existing cluster centers.

[0051] Status identifiers are coding status markers corresponding to cluster centers in the clustering codebook, used to characterize the current availability and lifecycle stage of the cluster centers. Specifically, status identifiers can include three types: active, inactive, and reserved. The active status indicates that the corresponding cluster center currently has a user affiliation and the representation quality meets the requirements, and can participate normally in production coding and incremental training. The inactive status indicates that the corresponding cluster center does not participate in production due to substandard quality assessment or has been split and replaced, but the corresponding coding slot is reserved for reuse by new cluster centers. The reserved status indicates that the corresponding coding slot is reserved for future expansion but has not yet been assigned to a cluster center.

[0052] Furthermore, status identifiers can be used to manage the availability of each coding slot during the cluster codebook update process. When a new cluster center is added to the codebook, the corresponding status identifier is updated to the active state. Through the coordinated management of the three states of active, inactive, and reserved, the cluster codebook has the ability to dynamically adjust while keeping the total amount of coding controllable. The reuse of inactive coding slots avoids the disorderly expansion of the codebook size, and the reserved coding slots provide space for limited codebook expansion.

[0053] In practical applications, obtaining the current user representation vector and the current cluster codebook can be achieved through various methods.

[0054] One alternative approach is to obtain the current user representation vector and the current cluster codebook using a centralized batch loading method. Specifically, user representation vectors of all active users within the current time window can be read in batches from the platform's user feature storage system. Simultaneously, the current version of the cluster codebook data structure and the status identifier information corresponding to each cluster center can be loaded from the codebook management service. The user representation vectors and the cluster codebook are then encapsulated together as input data for the codebook update task. During the batch reading process, format validation and deduplication can be performed on the user representation vectors. Only valid and unique user representation vectors are included in the input data, and the version number and last update time of the cluster codebook are recorded synchronously, providing complete metadata information for the codebook update process. Obtaining input data through a centralized batch loading method offers good data consistency and is simple to implement, making it suitable for offline, timed codebook update scenarios.

[0055] Another alternative approach is to obtain the current user representation vector and the current clustering codebook based on an incremental data synchronization mechanism. Specifically, the synchronization position marker at the time of the last codebook update can be recorded. Only user representation vectors that have been added or changed since the last codebook update are retrieved. Simultaneously, a snapshot version of the current clustering codebook is read from the distributed codebook storage to reduce data transfer volume and storage access pressure while ensuring data integrity. After incremental retrieval, the incremental user representation vector is merged with the existing user representation vectors cached locally to construct a complete set of current user representation vectors. During the merging process, duplicate vectors can be deduplicated. Obtaining input data through an incremental synchronization mechanism can significantly reduce the data loading overhead of each update task, making it suitable for scenarios with a large user base and high codebook update frequency.

[0056] Furthermore, the current user representation vector and the current cluster codebook can be obtained based on streaming access. Specifically, user representation vectors calculated in real time can be continuously received, and a codebook update process can be triggered after accumulating to a preset batch size. The current cluster codebook can then be read from the memory cache to improve the timeliness of data acquisition.

[0057] In this step, by acquiring the current user representation vector and the current cluster codebook, a complete input data foundation is provided for the cluster codebook update process. By acquiring the current cluster codebook, which is divided into multiple cluster centers and corresponding state identifiers according to the clustering level, the codebook update process can be carried out based on the currently effective codebook structure, providing an accurate comparison benchmark for subsequent coverage detection. By acquiring the current user representation vector, which represents the current distribution of the user group, the codebook update process can perceive the latest evolutionary state of the user group, providing data basis for identifying codebook coverage gaps. By using the current cluster codebook constructed based on historical user representation vectors as the processing object, the codebook update process can be carried out while maintaining the existing coding system, laying the foundation for subsequent incremental expansion.

[0058] Step 104: Based on the distance between the current user representation vector and the center vectors of multiple cluster centers, determine the uncovered user representation vectors from the current user representation vector.

[0059] The centroid vector is a fixed-dimensional coordinate vector corresponding to a cluster center, used to calculate the distance between the user representation vector and the cluster center. Specifically, the centroid vector can be the centroid coordinates determined through iterative optimization during the construction or updating of the cluster codebook. Each cluster center has a unique corresponding centroid vector, and the dimension of the centroid vector is consistent with the dimension of the user representation vector. The distance between them can be calculated using Euclidean distance, cosine distance, or other distance metrics. The centroid vector can serve as the basis for coverage detection. By comparing the distance between the current user representation vector and each centroid vector, it can be determined whether the corresponding user representation vector is sufficiently covered by existing cluster centers. Furthermore, the centroid vector can remain frozen during the incremental update of the cluster codebook, meaning that the coordinates of the centroid vectors of existing cluster centers are not modified due to the addition of new user representation vectors, ensuring that the encoding results of users with assigned semantic identifiers do not change due to codebook updates.

[0060] Distance is a measure of the difference between a user representation vector and the center vector of a cluster in vector space, used to gauge the proximity between them. Specifically, distance can include various metrics such as Euclidean distance and cosine distance. Euclidean distance measures numerical positional differences by calculating the square root of the sum of the squares of the differences in each dimension of the two vectors, while cosine distance measures directional differences by calculating the cosine of the angle between the two vectors. In practical applications, Euclidean distance can be chosen as the metric for the distance between a user representation vector and a cluster center. Furthermore, distance can serve as a quantitative basis for determining whether a user representation vector is covered by existing cluster centers. The smaller the distance between a user representation vector and its nearest cluster center, the closer the corresponding user representation vector is to the user group represented by that cluster center. The distances between all current user representation vectors and their nearest cluster centers can form a distance distribution. The statistical characteristics of this distance distribution can reflect the overall matching degree between the current user group and the existing cluster structure.

[0061] Uncovered user representation vectors are a subset of user representation vectors that are not fully covered by existing cluster centers. Specifically, they are user representation vectors whose distances to the center vectors of all existing cluster centers do not meet the coverage condition. Uncovered user representation vectors can include those whose distance to the center vector of the nearest cluster center exceeds a preset coverage distance threshold, or those falling into sparsely distributed regions of existing cluster centers. Their common characteristic is that the existing clustering codebook cannot provide them with the required precision for discretization. Uncovered user representation vectors can serve as the basis for determining new cluster centers; their distribution characteristics determine the location and number of new cluster centers. Furthermore, the emergence of uncovered user representation vectors usually stems from the emergence of new user groups or the differentiation of interests within existing user groups. Their size and distribution range reflect the degree of coverage gap of the current clustering codebook for the current user group, and are a direct signal triggering incremental expansion of the clustering codebook.

[0062] In practical applications, determining the uncovered user representation vectors from the current user representation vector based on the distance between the current user representation vector and the center vectors of multiple cluster centers can be achieved through various methods.

[0063] One alternative approach is to determine uncovered user representation vectors based on a distance threshold comparison. Specifically, the Euclidean distance between each user representation vector in the current user representation vector and the center vectors of each cluster center in the current clustering codebook can be calculated one by one. The minimum distance for each user representation vector is determined, and this minimum distance is compared with a preset coverage distance threshold. If the minimum distance exceeds the preset coverage distance threshold, the corresponding user representation vector is determined as an uncovered user representation vector. During the distance calculation process, the distance calculation between each user representation vector and each cluster center can be performed in parallel to accelerate the process, and the nearest cluster center number corresponding to each user representation vector is recorded to provide reference information for determining subsequent new cluster centers. Determining uncovered user representation vectors through a distance threshold comparison, using the distance threshold as the sole criterion, has clear logic and a unified judgment standard, making it suitable for codebooks with a moderate number of cluster centers.

[0064] Another alternative approach is to determine uncovered user representation vectors based on outlier detection. Specifically, outlier scores can be calculated based on the distance distribution between each user representation vector and the center vector of its nearest cluster. For example, an outlier determination interval can be constructed based on the mean and standard deviation of the distance distribution. User representation vectors whose distance deviates from the determination interval are identified as outlier vectors, and user representation vectors whose outlier scores exceed a preset outlier threshold are identified as uncovered user representation vectors. Optionally, a local outlier factor can be used to calculate the local outlier degree of each user representation vector relative to its neighboring user representation vectors. User representation vectors whose local outlier degree meets a preset condition are identified as uncovered user representation vectors. Determining uncovered user representation vectors through outlier detection identifies anomalous vectors deviating from existing cluster centers from a statistical distribution perspective. This approach has a lower dependence on the distance threshold setting and can adapt to detection needs of different codebook sizes.

[0065] Another alternative approach is to determine uncovered user representation vectors based on kernel density estimation. Specifically, kernel density estimation can be performed on the overall distribution of current user representation vectors to obtain their density distribution in the vector space. User representation vectors located in low-density regions and with large distances from the center vectors of the nearest cluster centers are identified as uncovered user representation vectors, balancing distance and distribution density criteria. Optionally, the density threshold and distance threshold of kernel density estimation can be jointly set, identifying uncovered user representation vectors when they simultaneously meet both low-density and long-distance conditions, thus improving the reliability of the determination. Determining uncovered user representation vectors through kernel density estimation can identify user groups in sparsely distributed areas of existing cluster centers, making it suitable for scenarios with uneven distribution of user interests.

[0066] Furthermore, uncovered user representation vectors can be determined based on a comprehensive assessment of coverage and residuals. Specifically, the coverage rate of current user representation vectors to existing cluster centers can be calculated. When the coverage rate is lower than a preset standard, user representation vectors with larger residuals are identified as uncovered user representation vectors. The residuals can represent the degree of unexplained difference between a user representation vector and its nearest cluster center.

[0067] In this step, the proximity between the current user representation vector and the center vectors of multiple cluster centers is calculated, quantifying the proximity between the user representation vector and the cluster centers into a comparable value, providing a quantitative basis for coverage determination. By determining the uncovered user representation vectors from the current user representation vectors based on distance, the coverage gaps in the cluster codebook are accurately identified in a data-driven manner, avoiding the computational overhead of re-clustering all user representation vectors. By using the uncovered user representation vectors as the objects of subsequent incremental updates, the expansion range of the cluster codebook is limited to the areas where coverage gaps actually exist, providing a clear object basis for determining new cluster centers.

[0068] Step 106: Determine new cluster centers based on the uncovered user representation vectors.

[0069] The newly added cluster centers are new centroid vectors determined based on the uncovered user representation vectors and used to supplement the coverage of the current cluster codebook. Specifically, the newly added cluster centers can include the center vectors of each cluster obtained after clustering the uncovered user representation vectors. Each newly added cluster center corresponds to a dense region in the distribution space of the uncovered user representation vectors, and its location is determined by the distribution characteristics of the uncovered user representation vectors. The newly added cluster centers can expand the coverage of the cluster codebook without modifying the positions of existing cluster centers, providing discretized coding with sufficient accuracy for user groups that were previously not adequately covered. Furthermore, the number of newly added cluster centers needs to be reasonably determined based on the size and dispersion of the uncovered user representation vectors. Too few centers will not be able to fully compensate for the coverage gap, while too many centers may lead to codebook redundancy and decreased coding discriminability.

[0070] In practical applications, determining new cluster centers based on uncovered user representation vectors can be achieved through various methods.

[0071] One alternative approach is to determine new cluster centers by performing clustering operations on the representation vectors of uncovered users. Specifically, clustering operations can be performed on the representation vectors of uncovered users, and the centroid vectors of each cluster obtained after clustering can be determined as new cluster centers. The number of clusters is determined based on the size and dispersion of the representation vectors of uncovered users. Optionally, before performing the clustering operation, the range of the number of clusters can be estimated based on the distribution characteristics of the representation vectors of uncovered users, and the number of clusters within the preset range that meets preset conditions can be selected to perform the clustering operation, ensuring that the clustering results match the actual distribution structure of the uncovered user group. By determining new cluster centers through clustering operations, the location of the new cluster centers can accurately reflect the distribution characteristics of the uncovered user group.

[0072] Another alternative approach is to determine new cluster centers through a step-by-step iterative method. Specifically, the vector with the highest distribution density among the uncovered user representation vectors can be selected as the initial centroid. The centroid position is then iteratively adjusted until convergence, and the converged centroid vector is determined as the new cluster center. This iterative process is repeated until the uncovered user representation vectors are fully covered. Optionally, after determining a new cluster center in each iteration, the coverage of the remaining uncovered vectors can be reassessed. If the size of the remaining uncovered vectors meets a preset condition, the next new cluster center can be determined, until all uncovered user representation vectors are covered. By determining new cluster centers through a step-by-step iterative method, and reassessing the distribution of the remaining uncovered vectors after each new cluster center is added, the positioning accuracy of the new cluster centers is relatively high.

[0073] Another alternative method is to determine new cluster centers based on distribution direction guidance. Specifically, the principal component analysis results of the uncovered user representation vectors can be used to determine the main direction of the distribution. Sampling along the principal component direction at preset intervals determines the initial positions of the new cluster centers, which are then adjusted locally to become the final new cluster centers. After determining the main direction, clustering can be performed on the projections of the uncovered user representation vectors onto the plane perpendicular to the main direction to determine the distribution positions of each new cluster center along the main direction. This combined positional information along the main direction then determines the complete set of new cluster centers. Determining new cluster centers based on distribution direction guidance, utilizing the main direction information to guide the layout of the new cluster centers, is suitable for scenarios where the uncovered user representation vectors are linearly or ellipsoidally distributed.

[0074] In addition, new cluster centers can be determined based on the extreme value vector method. Specifically, the vector farthest from the existing cluster centers in the uncovered user representation vectors can be directly determined as the new cluster center, with the farthest vector representing the location of the area with the most severe coverage gap.

[0075] In this step, new cluster centers are determined based on the representation vectors of uncovered users, so that the expansion of the cluster codebook is driven by the actual coverage gap. The new cluster centers can accurately match the distribution characteristics of the uncovered user group. By using the representation vectors of uncovered users as the basis for determining new cluster centers, codebook redundancy caused by blindly adding cluster centers is avoided, and the codebook expansion range is kept consistent with the evolution trend of the user group. By determining new cluster centers, incremental content is provided for subsequent merging with existing cluster centers to form the target cluster codebook, realizing the adaptive expansion of the cluster codebook while maintaining the existing structure.

[0076] Step 108: Merge the newly added cluster centers with the existing cluster centers in the current cluster codebook, and update the status flags corresponding to each cluster center to obtain the target cluster codebook.

[0077] The target clustering codebook is an updated codebook obtained by merging newly added cluster centers with multiple existing cluster centers in the current clustering codebook and updating the status labels corresponding to each cluster center. It serves as the encoding benchmark for subsequent semantic label determination. Specifically, the target clustering codebook can include a complete set of all original cluster centers and newly added cluster centers. Each cluster center corresponds to an updated status label. The status labels of newly added cluster centers are updated to active status, while the status labels of existing cluster centers remain unchanged or are updated based on quality assessment results.

[0078] Furthermore, the target clustering codebook can serve as an updated encoding dictionary for assigning semantic identifiers to user representation vectors. Users determine their assigned cluster center number combination by calculating the distance between their representation vector and the cluster centers at each layer in the target clustering codebook. The target clustering codebook expands its coverage while maintaining the stability of the existing encoding system. Users already assigned semantic identifiers do not experience changes in their encoding results due to codebook updates, while newly covered users receive newly assigned semantic identifiers, ensuring a smooth transition during codebook updates.

[0079] In practical applications, merging newly added cluster centers with multiple existing cluster centers in the current clustering codebook can be achieved through various methods.

[0080] One alternative approach is to merge cluster centers using an append-and-merge method. Specifically, new cluster centers can be appended to the current cluster center set in the clustering codebook. Each new cluster center is assigned a corresponding coding slot, while the positions and status identifiers of existing cluster centers remain unchanged. The merged complete cluster center set and its status identifier information are then encapsulated into the target clustering codebook and persistently stored. After appending, the total number of cluster centers in the target clustering codebook and the distribution of cluster centers at each clustering level can be verified to confirm that the merging result meets the codebook capacity and hierarchical structure constraints. Merging cluster centers using this append-and-merge method leaves the existing coding system completely unaffected, achieving simplicity and backward compatibility.

[0081] Another alternative approach is to merge cluster centers using a collaborative method of merging and quality maintenance. Specifically, while merging new cluster centers, a quality assessment can be performed on existing cluster centers. The status of existing cluster centers that fail the quality assessment is updated to "invalid," while the status of newly added cluster centers is set to "active." The target cluster codebook is then generated based on all updated statuses. During the merging process, newly added cluster centers can be preferentially assigned to the coding slots corresponding to the invalid status, thus simultaneously reusing invalid slots and keeping the codebook capacity manageable. By merging cluster centers through this collaborative approach of merging and quality maintenance, codebook quality is maintained while expanding coverage capabilities, and invalid coding slots can be reused in subsequent updates.

[0082] Another alternative approach is to merge cluster centers based on consistency checks. Specifically, after merging, a global consistency check can be performed on the target cluster codebook to verify that there is no overlap in coverage areas between newly added cluster centers and existing cluster centers. If overlap exists, the positions of the newly added cluster centers can be adjusted or overlapping cluster centers can be merged. After the check passes, the target cluster codebook is released as the new effective version. Before release, a uniqueness check is performed on the target cluster codebook to confirm that the codes of each cluster center are not duplicated and that the allocation status of each coding slot corresponds one-to-one with the cluster centers. Merging cluster centers through consistency checks ensures that the coverage areas of each cluster center in the target cluster codebook do not conflict, avoiding ambiguity in the encoding of user representation vectors.

[0083] Furthermore, cluster centers can be merged based on version-based management. Specifically, a change log can be recorded for the target cluster codebook relative to the current cluster codebook. The change log includes information on newly added cluster centers, changes to status identifiers, and coding slot allocation. It also supports rolling back to the previous version in case of update errors.

[0084] Once the merging is complete, the status identifiers corresponding to each cluster center can be updated to obtain the target cluster codebook.

[0085] One alternative method is to update the status identifiers of each cluster center in batches. Specifically, based on the current status of each cluster center and the merging results, the updated status identifiers of each cluster center can be determined. Newly added cluster centers are updated to an active state, existing active cluster centers remain unchanged, and cluster centers that fail the quality assessment are updated to an invalid state. The updated status identifiers are then combined with the merged cluster center set to obtain the target cluster codebook. Furthermore, a consistency check can be performed on the batch update results of the status identifiers to confirm the accurate correspondence between the status identifiers of each coding slot and the cluster centers. Updating status identifiers in batches provides a clear update process and traceable results.

[0086] Another alternative approach is to update state identifiers based on state transition rules. Specifically, batch updates can be performed based on state identifier transition rules. After merging newly added cluster centers, the state identifiers of newly added cluster centers in the coding slots corresponding to the failed state are updated to active state. If there are insufficient failed coding slots, the remaining newly added cluster centers are allocated to the coding slots corresponding to the reserved state, and their state identifiers are updated synchronously. The target cluster codebook is obtained based on the updated state identifiers. Optionally, the state identifiers of cluster centers whose quality assessment indicators do not meet the preset thresholds can also be updated to failed state, combining the quality assessment results, so that state updates and quality maintenance are performed in tandem. By updating state identifiers through state transition rules, the state update process and the coding slot allocation process are performed in tandem, and the state transition relationship is clear and controllable.

[0087] In addition, after the status identifier is updated, an integrity check can be performed on the target cluster codebook to verify the correspondence between the status identifier of each coding slot and the cluster center. If there are any abnormalities, the target cluster codebook can be output after correction.

[0088] In this step, by merging the newly added cluster centers with multiple existing cluster centers in the current cluster codebook, the codebook coverage is expanded while maintaining the positions of existing cluster centers, avoiding encoding jumps caused by full retraining. By updating the status identifiers corresponding to each cluster center, the newly added cluster center is incorporated into the encoding system in an active state, and the state transitions of invalid and reserved encoding slots are recorded, providing structured information for subsequent codebook quality maintenance and capacity management. By obtaining the target cluster codebook, an updated encoding benchmark that meets the coverage accuracy requirements and has stable encoding semantics is provided for subsequent semantic identifier generation, realizing a closed loop for dynamic maintenance of the cluster codebook.

[0089] In this embodiment, by acquiring the current user representation vector and the current cluster codebook, which is divided into multiple cluster centers and corresponding state identifiers according to the clustering hierarchy, the codebook update process has a complete data and structural input foundation. By determining the uncovered user representation vector based on the distance between the current user representation vector and the center vectors of multiple cluster centers, the codebook coverage gap is accurately identified in a quantitative manner, providing a clear target for codebook expansion. By determining the new cluster centers based on the uncovered user representation vectors, the codebook expansion direction is driven by the actual coverage gap, and the new cluster centers can accurately match the distribution characteristics of the uncovered user group. By merging the new cluster centers with the existing multiple cluster centers in the current cluster codebook and updating the state identifiers corresponding to each cluster center, the target cluster codebook is obtained. While expanding the codebook coverage capability, the stability of the existing cluster centers and their encoded semantics is maintained, avoiding encoding jumps that would damage the historical association memory of the downstream recommendation model. This achieves dynamic expansion and continuous updating of the cluster codebook while maintaining the stability of the existing encoded semantics, enhancing the codebook's adaptability to the evolution of the user group, and thus improving the accuracy and efficiency of user semantic identifier generation.

[0090] In an optional embodiment of this specification, before obtaining the current user representation vector and the current clustering codebook, the method further includes: Obtain historical user representation vectors; based on historical user representation vectors, perform cluster training layer by layer according to the clustering level to obtain initial cluster centers; Set the status flag of each initial cluster center to active state to obtain the current cluster codebook.

[0091] Cluster training is the process of performing layer-by-layer clustering on user representation vectors to determine cluster centers. Specifically, cluster training can include performing clustering operations sequentially from the first to the last clustering level, based on historical user representation vectors. The first clustering level performs a global clustering of historical user representation vectors, and subsequent clustering levels cluster the clustering residuals of the previous level. Each clustering level yields its corresponding cluster centers. The clustering operation can be implemented using K-means clustering, iteratively optimizing to minimize the distance between samples and cluster centers. Cluster training serves as the core step in constructing the cluster codebook, determining the cluster center positions for each clustering level within the codebook upon completion. Furthermore, cluster training is performed once during the cold start phase, using historical user representation vectors as training samples. The resulting cluster codebook becomes the current codebook used in production and is continuously maintained through incremental updates.

[0092] The initial cluster centers are obtained through cluster training and determined during the initial construction phase of the cluster codebook. Specifically, the initial cluster centers can include all cluster centers obtained after the first cluster training at each cluster level. Each initial cluster center corresponds to a coding slot, and its position is determined by the distribution characteristics of historical user representation vectors. The initial cluster centers serve as the fundamental components of the current cluster codebook, and their quality directly determines the initial coding accuracy of the codebook. Furthermore, all initial cluster centers are active after cold start training. Subsequently, a quality evaluation mechanism marks substandard cluster centers as invalid, and an incremental update mechanism is used to fill coverage gaps.

[0093] The active state is a state type in the state identifier, indicating that the corresponding cluster center is currently in a normal and usable state and can participate in production encoding and incremental training. Specifically, cluster centers with the active state identifier can normally participate in the encoding mapping of user representation vectors, and their corresponding encoding slots can be hit by user representation vectors. The active state is the main state in the life cycle of a cluster center; when a new cluster center is added to the codebook, its state identifier is set to active. The active state can be used to distinguish between cluster centers that are normally serving and those that are not serving in the codebook. Furthermore, the active state, along with the inactive state and the reserved state, constitutes a three-state state management mechanism. The set of active cluster centers determines the actual encoding service capability of the codebook at present.

[0094] In practical applications, historical user representation vectors can be obtained through various methods.

[0095] One alternative approach is to obtain historical user representation vectors through batch reading based on historical time windows. Specifically, all user representation vectors within a preset historical time window can be read in batches from the platform's user feature storage system. The historical time window can be the most recent period before codebook construction, such as the last 30 days or the last 90 days. The read user representation vectors are used as sample data for clustering training. During the batch reading process, user representation vectors for each time partition can be read sequentially, and after reading, the user representation vectors for each partition are merged and deduplicated to form a complete clustering training sample set. Obtaining historical user representation vectors through batch reading using historical time windows can provide complete user group distribution information, which is suitable for the data requirements of the initial codebook construction during the cold start phase.

[0096] Another alternative approach is to acquire historical user representation vectors through offline data pipeline accumulation. Specifically, historical user representation vectors can be obtained from offline data pipelines, and these vectors from each time partition can be accumulated and merged chronologically. Once the accumulated data reaches a preset size, it can be output as a clustering training sample set. During the accumulation process, user representation vectors can be weighted according to user activity, controlling the sample set size while maintaining the sample distribution characteristics. Acquiring historical user representation vectors through offline data pipeline accumulation leverages existing offline data processing infrastructure to achieve continuous accumulation of historical samples.

[0097] Furthermore, historical user representation vectors can be obtained based on sampling methods. Specifically, sampling can be performed on all historical user representation vectors, and representative samples can be extracted according to a preset sampling ratio as clustering training data, thereby reducing the computational cost of clustering training while maintaining the sample distribution characteristics.

[0098] Once historical user representation vectors are obtained, clustering training can be performed layer by layer according to the clustering hierarchy based on the historical user representation vectors to obtain the initial cluster centers.

[0099] One alternative approach is to perform clustering training based on a progressive clustering method. Specifically, clustering operations can be performed sequentially from the first to the last clustering level. The first level uses K-means clustering based on historical user representation vectors to obtain the first-level cluster centers. The residuals between each historical user representation vector and the first-level cluster centers are calculated, and the second-level clustering is performed based on these residual vectors to obtain the second-level cluster centers. This process continues until all clustering levels are completed, and the cluster centers obtained at each level are used as the initial cluster centers. After each level of clustering is completed, a convergence check can be performed on the cluster centers. The iteration of the corresponding level is terminated early if the change in the position of the cluster centers is less than a preset threshold. By performing clustering training through a progressive clustering method, the determination of cluster centers at each level has a clear progressive relationship, strictly corresponding to the structural characteristics of residual quantization clustering.

[0100] Another alternative approach is to perform cluster training based on parallel clustering. Specifically, before training, the number of cluster centers at each clustering level can be estimated based on the overall distribution of historical user representation vectors. Then, cluster training at each level can be performed in parallel. After each level of clustering is completed, the consistency of the clustering results at each level can be verified through residual correlation. Furthermore, the historical user representation vectors can be distributed to multiple computing nodes in segments, with each node performing clustering operations in parallel. Then, the local cluster centers of each node can be globally merged and adjusted to obtain the initial cluster centers for each clustering level. Performing cluster training through parallel clustering shortens the overall time of cold start training and is suitable for scenarios with large historical user representation vectors.

[0101] Another alternative approach is to perform clustering training using a two-stage clustering method. Specifically, initial clustering at each level can be performed first using a sample set of historical user representation vectors. Then, the initial clustering results are applied to the full set of historical user representation vectors for center fine-tuning. This approach controls computational overhead while ensuring the accuracy of the cluster centers' fit to the full sample set. The center fine-tuning stage can have an upper limit on the number of iterations; after convergence or reaching the upper limit, the initial cluster centers for each level are output. This two-stage clustering training method controls computational overhead while maintaining the accuracy of cluster center fitting, making it suitable for scenarios with high sample size and dimensionality.

[0102] Furthermore, cluster training can be performed based on a distributed clustering framework. Specifically, historical user representation vectors can be segmented and clustered using a distributed clustering framework, with clustering operations at each clustering level executed in parallel on multiple computing nodes. Then, the local cluster centers of each node are globally merged and adjusted to obtain the initial cluster centers.

[0103] Once the initial cluster centers are obtained, the status flags of each initial cluster center can be set to active status to obtain the current cluster codebook.

[0104] One alternative approach is to set the status identifiers using a unified batch setting method. Specifically, after cluster training is complete, all initial cluster centers can be traversed, and the status identifiers of each initial cluster center can be uniformly set to active. Each initial cluster center and its status identifier can then be encapsulated into the current cluster codebook for persistent storage. During the encapsulation process, a version identifier and build time information can be appended to the current cluster codebook for easier version management during subsequent codebook updates. Setting the status identifiers using a unified batch setting method completes the initial state setting through a consistent batch operation, achieving simplicity and ensuring state consistency.

[0105] Another alternative approach is to set status identifiers based on training synchronization. Specifically, a status identifier can be synchronously recorded for each cluster center during cluster training. Upon completion of training, the cluster centers carry an active status identifier, and the current cluster codebook is directly generated based on the training results, eliminating the need for batch status setting after training. Furthermore, abnormal cluster centers can be marked with special status identifiers during training, and these cluster centers with special status identifiers can be reviewed before generating the current cluster codebook. Setting status identifiers through training synchronization reduces additional data traversal overhead, and the initial state of the cluster centers is generated synchronously with the training results.

[0106] In addition, a status check can be performed after the current cluster codebook is generated, traversing all coding slots to confirm that the status identifiers of each cluster center are active, and correcting any abnormal statuses to ensure that the current cluster codebook is online in a fully available state.

[0107] In a specific application scenario, after obtaining the current cluster codebook, a simulated incremental update can be performed on the current cluster codebook based on the historical user representation vectors to determine the coverage of the current cluster codebook with the historical user representation vectors. If the coverage meets the preset verification conditions, the current cluster codebook is determined to have passed the verification and is put into production to ensure that the put-in codebook can cover all historical user representation vectors.

[0108] In the embodiments of this specification, historical user representation vectors are obtained to provide a complete training sample foundation for the initial construction of the cluster codebook. Initial cluster centers are obtained by performing cluster training layer by layer according to the clustering level based on the historical user representation vectors. The initial structure of the codebook is established through residual quantization clustering, and each clustering level corresponds to the division of user groups at different granularities. The current cluster codebook is obtained by setting the status flag of each initial cluster center to an active state, so that the codebook completes the cold start and goes online with all codes available. This is connected with the steps of obtaining the current user representation vectors and the current cluster codebook, forming a complete cluster codebook maintenance link from cold start training to continuous incremental updates.

[0109] In an optional embodiment of this specification, after obtaining the current user representation vector and the current clustering codebook, the method further includes: Based on historical user representation vectors, the quality of each cluster center in the current clustering codebook is evaluated to obtain the quality evaluation index of each cluster center. If the quality assessment indicators do not meet the preset threshold, the status identifier of the corresponding cluster center will be updated to the failure status.

[0110] Quality assessment is the process of evaluating the quality of each cluster center in a clustered codebook. Specifically, quality assessment may include calculating and analyzing the distribution characteristics of samples within the cluster corresponding to each cluster center based on historical user representation vectors, obtaining quality assessment indicators for each cluster center, and comparing these indicators with preset threshold values ​​to determine whether each cluster center meets quality requirements. Quality assessment can serve as the core mechanism for clustered codebook quality control, and the assessment results are used to determine the status of the cluster centers. Furthermore, quality assessment can be performed layer by layer according to the clustering hierarchy and can be periodically repeated during codebook maintenance. Low-quality codes are marked as invalid through quality assessment, preventing low-quality cluster centers from affecting production coding performance.

[0111] Quality assessment metrics are quantitative indicators characterizing the quality of cluster centers, used to measure the quality level of the clusters corresponding to those centers. Specifically, quality assessment metrics may include boundary stability metrics, minimum clustering percentage metrics, coefficient of variation metrics, and residual explained rate metrics, etc. Each quality assessment metric characterizes the quality features of cluster centers from different dimensions, and each metric corresponds to a preset threshold. Quality assessment metrics serve as the numerical carrier of quality assessment results; by comparing each quality assessment metric with its corresponding threshold, it can be determined whether the cluster centers meet the quality requirements. Furthermore, the selection of quality assessment metrics can cover multiple quality dimensions such as sample size, cluster dispersion, boundary stability, and residual explained rate of cluster centers, to comprehensively assess whether the cluster centers can represent a meaningful user group.

[0112] Preset threshold values ​​are pre-defined critical values ​​used to determine whether a quality assessment indicator meets the required standards. Specifically, preset threshold values ​​can be set separately for each quality assessment indicator, and different indicators may have different preset threshold values. These threshold values ​​can be determined based on the business scenario and codebase size, or they can be differentiated based on the clustering level. The preset threshold values ​​serve as the criterion for quality assessment; the comparison between the quality assessment indicator and the preset threshold value determines whether a cluster center is marked as invalid. Furthermore, in specific application scenarios, preset threshold values ​​can be adjusted layer by layer according to the clustering level using attenuation coefficients to accommodate the natural decrease in the discriminative power of deep cluster centers, preventing deep cluster centers from being incorrectly marked as invalid.

[0113] Failure status is a type of status identifier that indicates the corresponding cluster center is not currently participating in production coding, but the corresponding coding slot is reserved for reuse by new cluster centers. Specifically, cluster centers marked as failure status can be cluster centers that fail quality assessments or cluster centers that have been split and replaced. Cluster centers in the failure status do not participate in production coding and incremental training, but the coding slots they occupy are reserved. New cluster centers can be assigned to the coding slots corresponding to the failure status and the status identifier will be updated. Failure status can serve as a key state for cluster codebook quality control and capacity management. Furthermore, by marking low-quality cluster centers as failure status, low-quality coding is prevented from occupying codebook expression capacity and affecting production performance, while the reuse mechanism of failure coding slots keeps codebook capacity controllable.

[0114] In practical applications, quality assessment of each cluster center in the current clustering codebook can be performed based on historical user representation vectors to obtain quality assessment indicators for each cluster center. This can be achieved through various methods.

[0115] One alternative approach is to calculate quality assessment indicators based on cluster distribution statistics. Specifically, the distribution of sample user representation vectors within each cluster corresponding to each cluster center can be statistically analyzed. Based on the distance distribution between the sample user representation vectors within each cluster and the center vector of the cluster center, the boundary stability index, minimum clustering percentage index, coefficient of variation index, and residual explanation rate index corresponding to each cluster center can be calculated to obtain the quality assessment indicators for each cluster center. During the indicator calculation process, normalization can be performed on each indicator to ensure that the numerical range of each quality assessment indicator is consistent, facilitating subsequent unified comparison with preset indicator thresholds. Calculating quality assessment indicators through cluster distribution statistics provides evaluation results based on the actual distribution of historical user representation vectors, offering a clear statistical basis.

[0116] Another alternative approach is to obtain quality assessment metrics based on batch computation of the distance matrix. Specifically, based on the distance matrix between historical user representation vectors and the center vectors of each cluster center, the quality features of the corresponding clusters for each cluster center can be calculated in batches. Quality assessment metrics for all cluster centers can be obtained simultaneously through matrix operations. Furthermore, parallel computing frameworks can be used to accelerate matrix operations during the distance matrix calculation process, and the distance matrix can be compressed for storage to reduce memory usage. Obtaining quality assessment metrics through batch computation of the distance matrix improves the execution efficiency of quality assessment and is suitable for codebases with a large number of cluster centers.

[0117] Another alternative approach is to obtain quality assessment indicators based on a hierarchical evaluation method. Specifically, quality assessment can be performed layer by layer according to the clustering hierarchy. After the assessment at the current level is completed, the sample range for the next level is determined based on the assessment results. Quality assessment indicators are calculated only for the cluster centers corresponding to samples that passed the previous level's assessment. During the hierarchical evaluation process, the filtering results of samples at each level can be recorded, providing a basis for sample selection in subsequent levels. Obtaining quality assessment indicators through a hierarchical evaluation method ensures that the assessment results at each level are interconnected, conforming to the hierarchical dependency relationship of residual quantization clustering.

[0118] Furthermore, quality assessment metrics can be obtained based on incremental evaluation. Specifically, quality assessment metrics can be recalculated only for cluster centers newly added or whose status has changed in the most recent codebook update, while historical assessment results can be reused for cluster centers whose status has not changed, thereby reducing the computational overhead of periodic quality assessments.

[0119] Once the quality assessment indicators for each cluster center are obtained, the status identifier of the corresponding cluster center can be updated to the invalid state if the quality assessment indicators do not meet the preset threshold.

[0120] One alternative method is to perform failure determination based on direct comparison of indicator thresholds. Specifically, the quality assessment indicators of each cluster center can be compared one by one with preset indicator thresholds. If any quality assessment indicator fails to meet the corresponding preset indicator threshold, the status identifier of the corresponding cluster center is updated to a failure state, and the status identifier of the corresponding coding slot is updated simultaneously. During the comparison process, the indicator values ​​and threshold comparison results of each cluster center can be recorded to form a detailed quality assessment record for subsequent review and traceability. Performing failure determination through direct comparison of indicator thresholds is simple and the determination results are traceable.

[0121] Another alternative approach is to perform failure determination based on a multi-indicator joint judgment method. Specifically, the quality assessment indicators of each cluster center can be compared with preset indicator thresholds. The number of indicators that do not meet the corresponding preset thresholds can be counted. When the number of indicators reaches a preset number, the status identifier of the corresponding cluster center is updated to a failed state. Optionally, different weights can be assigned to different types of quality assessment indicators, and failure determination can be performed based on the weighted number of non-compliant indicators. Performing failure determination through a multi-indicator joint judgment method reduces the risk of misjudgment caused by a single indicator.

[0122] Another alternative approach is to perform failure determination based on a confidence-protected approach. Specifically, before determining failure, a sample size confidence check can be performed on each cluster center. Failure determination is performed only if the number of user representation vectors corresponding to the cluster center meets a preset condition; otherwise, failure determination is skipped and the status label remains unchanged. Optionally, cluster centers with insufficient samples can be marked as pending evaluation, and failure determination can be re-executed after sufficient samples have accumulated. Performing failure determination using a confidence-protected approach avoids misjudgments caused by evaluation results based on a small number of samples.

[0123] In addition, a quality assessment report can be output after the failure status update is completed, recording the quality assessment indicators, preset indicator thresholds and status identifier update results of each cluster center, so that operators can review and audit the quality assessment process.

[0124] In the embodiments of this specification, the quality of each cluster center in the current clustering codebook is evaluated based on historical user representation vectors, so that the quality status of the clustering codebook is systematically characterized in the form of quantitative indicators, providing an objective basis for the identification of low-quality codes. When the quality evaluation indicators do not meet the preset indicator thresholds, the status identifier of the corresponding cluster center is updated to the invalid state, so that the low-quality codes are removed from production services, avoiding the continuous occupation of codebook expression capacity and affecting coding quality by low-quality cluster centers. Through the synergy of the quality evaluation mechanism and the incremental update mechanism, the continuous guarantee of codebook quality is achieved, so that the clustering codebook always maintains the controllability of coding quality during dynamic expansion.

[0125] In one optional embodiment of this specification, the quality assessment metrics include at least one of the following: boundary stability metric, minimum clustering percentage metric, coefficient of variation metric, and residual explanation rate metric. If the quality assessment indicators do not meet the preset threshold, the status identifier of the corresponding cluster center will be updated to an invalid state, including: If the number of quality assessment indicators that do not meet the corresponding preset threshold reaches the preset number, the status identifier of the corresponding cluster center will be updated to the invalid state.

[0126] Boundary stability index is a metric that characterizes the proportion of samples within a cluster corresponding to a cluster center that are far from the decision boundary, and is used to evaluate the stability of cluster center encoding. Specifically, the boundary stability index can include the proportion of samples within a cluster whose distance from the center vector of adjacent cluster centers is greater than a preset margin threshold. A lower boundary stability index value indicates that more samples are located near the cluster boundary, and these samples are easily perturbed and reassigned to adjacent clusters, leading to encoding instability. The boundary stability index can serve as a quantitative basis for evaluating the encoding stability of cluster centers. Further, in specific application scenarios, the boundary distance between each sample within a cluster and its adjacent cluster centers can be calculated, and the proportion of samples whose boundary distance is less than a preset margin threshold can be counted. A higher proportion indicates a lower boundary stability index. When the boundary stability index is lower than the corresponding preset threshold, a large number of samples are located near the cluster boundary, and the corresponding cluster center may be judged as substandard.

[0127] The minimum clustering percentage metric is an indicator that characterizes the proportion of the cluster size corresponding to a cluster center relative to the total number of samples, and is used to assess the sufficiency of the sample size of cluster centers. Specifically, the minimum clustering percentage metric can include the proportion of the number of samples in the cluster corresponding to the cluster center to the total number of training samples in the corresponding clustering level. The lower the minimum clustering percentage metric value, the smaller the sample size of the cluster corresponding to the cluster center, and the less representative it is. The minimum clustering percentage metric can serve as a quantitative basis for assessing the sufficiency of the sample size of cluster centers. Furthermore, clusters with too small a sample size may only represent noise or individual samples, and do not have statistical representativeness. If the minimum clustering percentage metric is lower than the corresponding preset threshold, the corresponding cluster center may be judged as substandard.

[0128] The coefficient of variation (CV) is an indicator that characterizes the dispersion of samples within a cluster corresponding to a cluster center, and is used to assess the representativeness of the cluster center to the samples within the cluster. Specifically, the CV can be the ratio of the standard deviation to the mean of the distance between the center vector of a sample within a cluster and the center vector of the cluster center. A higher CV value indicates that the samples within the cluster are more dispersed, and the cluster center is less representative of the samples. The CV can serve as a quantitative basis for assessing the cluster cohesion of cluster centers. Further, in specific applications, the distance sequence from all samples within a cluster to the center vector of the cluster center can be calculated, and the CV can be calculated based on the mean and standard deviation of the distance sequence. If the CV is higher than a corresponding preset threshold, it indicates that the corresponding cluster center cannot effectively represent the samples within its cluster, and the cluster center quality is substandard.

[0129] The residual explanation rate (RER) is an indicator that characterizes the proportion of residuals in the samples covered by a cluster center that can be explained, and is used to evaluate the quantification effect of the cluster center. Specifically, the RER can include the proportion of residuals in the samples covered by a cluster center that are explained after being represented by the corresponding cluster center. The lower the RER value, the less effectively the cluster center can capture the residual information of the samples it covers, and the worse the quantification effect. The RER can serve as a quantitative basis for evaluating the quantification effect of cluster centers. Furthermore, in a residual quantification clustering structure, each clustering level clusters the residuals of the previous clustering level. The RER directly reflects the ability of the current level cluster center to capture residual information. If the RER is lower than a corresponding preset threshold, it indicates that the corresponding cluster center's ability to explain the residuals is insufficient, and the cluster center quality is substandard.

[0130] In practical applications, when the number of quality assessment indicators that do not meet the corresponding preset threshold reaches a preset number, the status identifier of the corresponding cluster center is updated to the invalid state, which can be achieved through various methods.

[0131] One alternative approach is to perform failure determination based on a multi-indicator counting method. Specifically, for each cluster center, the boundary stability index, minimum clustering percentage index, coefficient of variation index, and residual explained rate index are compared with their corresponding preset index thresholds. The number of indices that do not meet the preset index thresholds is counted. When the number of indices that do not meet the preset thresholds reaches a certain number, the status of the corresponding cluster center is updated to a failed state. In specific application scenarios, four quality assessment indicators can be used simultaneously for evaluation. The preset number can be two, meaning that when at least two of the four quality assessment indicators do not meet their corresponding preset index thresholds, the status of the corresponding cluster center is updated to a failed state. During the statistical process, non-compliance can be recorded separately according to index type, forming quality deficiency information for each cluster center. Performing failure determination through a multi-indicator counting method reduces the risk of misjudgment by a single indicator through joint determination of multiple indicators, and the evaluation results have high robustness.

[0132] Another alternative approach is to perform failure determination based on a comprehensive scoring method. Specifically, each quality assessment indicator can be normalized and then weighted and summed to obtain a comprehensive quality score for the cluster centers. This comprehensive quality score is then compared to a preset scoring threshold. If the comprehensive quality score does not meet the threshold, the corresponding cluster center's status is updated to a failed state. Optionally, weighting coefficients can be set based on the degree of influence of each quality assessment indicator on coding quality, with indicators having a higher weight for greater influence. Performing failure determination through a comprehensive scoring method integrates multi-dimensional indicators into a single score, resulting in a simple determination process suitable for batch evaluation scenarios with a large number of cluster centers.

[0133] Another alternative method is to perform failure determination based on a hierarchical approach. Specifically, the determination can be performed hierarchically according to the importance of the indicators. First, the minimum clustering percentage and coefficient of variation indicators are evaluated. If neither meets the corresponding preset threshold, the indicator is directly marked as failed. Otherwise, the boundary stability and residual explained rate indicators are evaluated. If the number of indicators failing to meet the corresponding preset threshold reaches a preset number, the indicator is marked as failed. During the hierarchical determination process, the results of each level can be recorded, and when the final determination is that the indicator is not failed, the compliance status of each level is output. Performing failure determination through a hierarchical approach reduces unnecessary indicator calculations and improves evaluation efficiency.

[0134] In addition, a confidence protection mechanism can be used to perform failure determination. If the number of samples corresponding to a cluster center is insufficient, the quality assessment and failure determination of the corresponding cluster center can be skipped, thus avoiding the cluster center being incorrectly marked as failed due to unstable indicators calculated based on a small number of samples.

[0135] In the embodiments of this specification, at least one of the following indicators—boundary stability, minimum clustering percentage, coefficient of variation, and residual explanation rate—is used to assess the quality of cluster centers. This characterizes the quality of cluster centers from multiple dimensions, including coding stability, sample size, cluster dispersion, and residual explanation capability, providing a multi-faceted quantitative basis for identifying low-quality cluster centers. By updating the status of the corresponding cluster center to a failed state when the number of quality assessment indicators that do not meet the corresponding preset threshold reaches a preset number, the robustness of the quality assessment is improved by using a multi-indicator joint judgment method to avoid misjudgment caused by fluctuations in a single indicator. By standardizing the comparison between the quality assessment indicators and the preset threshold, the failure judgment process is made objective and reproducible, ensuring that low-quality cluster centers are reliably identified and removed from production coding services.

[0136] In one optional embodiment of this specification, a quality assessment of each cluster center in the current clustering codebook is performed, including: Identify the cluster centers in each cluster level that are marked as active; Following the hierarchical order of clustering, quality assessments are performed layer by layer for each active cluster center to obtain quality assessment indicators for each cluster center.

[0137] The hierarchical order refers to the arrangement of clustering levels in the clustering codebook from coarse to fine granularity. Specifically, the hierarchical order can include the arrangement from the first layer to the last layer. The first layer is the layer that performs overall clustering of user representation vectors, and the last layer is the layer that performs the finest-grained clustering of clustering residuals. The hierarchical order determines the execution order of clustering codebook construction and quality assessment. The hierarchical order can serve as the basis for performing quality assessment layer by layer. Evaluating layer by layer according to the hierarchical order ensures that the evaluation results of previous layers have a correct impact on the evaluation input of subsequent layers. Furthermore, in specific application scenarios, quality assessment is performed sequentially from front to back according to the hierarchical order. The sample user representation vectors corresponding to cluster centers with a state marked as invalid in previous layers will not participate in the quality assessment of subsequent layers, thereby avoiding interference from invalid codes on subsequent layer evaluations.

[0138] In practical applications, determining the cluster centers that are in an active state at each cluster level can be achieved through various methods.

[0139] One alternative approach is to determine the active cluster centers based on a full traversal of state identifiers. Specifically, this involves traversing all cluster centers at each clustering level in the current clustering codebook, reading the state identifier of each cluster center one by one, and filtering out the cluster centers marked as active to form the set of cluster centers to be evaluated at each clustering level. Furthermore, during the traversal, the number of cluster centers in active, inactive, and reserved states at each clustering level can be simultaneously counted to form the coding state distribution information for each clustering level, providing a reference for quality assessment and scheduling. Determining the active cluster centers through a full traversal of state identifiers is simple and yields complete results.

[0140] Another alternative approach is to determine the cluster centers of active states based on a state index query. Specifically, the state index table of coding slots can be used to quickly query the coding slots marked as active in each cluster level, and the cluster centers corresponding to the active coding slots can be identified as the cluster centers of active states. An inverted list of active coding slots can be built in the state index table according to the cluster level, supporting fast retrieval of active cluster centers by level. Determining the cluster centers of active states through a state index query avoids a full traversal and has higher filtering efficiency in scenarios with a large number of cluster centers.

[0141] Furthermore, the active cluster centers can be determined based on the active list maintenance method. Specifically, during the loading and updating of the cluster codebook, the list of active cluster centers at each cluster level can be maintained synchronously, and the list can be updated synchronously when the status identifier changes. During quality assessment, the maintained list of active cluster centers can be directly read to further reduce the filtering overhead.

[0142] In practical applications, quality assessments are performed layer by layer for each active cluster center according to the hierarchical order of the clustering levels to obtain the quality assessment indicators for each cluster center. This can also be achieved through various methods.

[0143] One alternative approach is to perform quality assessment based on a progressive, layer-by-layer evaluation method. Specifically, quality assessment can be performed layer by layer, starting from the first layer. After the current layer's assessment is completed, the status identifiers of each cluster center at that layer are updated based on the assessment results. The range of samples participating in the next layer's quality assessment is then determined based on the updated status identifiers. That is, the user representation vectors of samples corresponding to cluster centers whose status identifiers are in a failed state in previous layers do not participate in the quality assessment of subsequent layers. This process continues until all clustering layers' quality assessments are completed. During this layer-by-layer progression, the assessment progress and sample filtering status at each layer can be recorded, forming a layer-by-layer assessment trajectory record. By performing quality assessment through a progressive, layer-by-layer evaluation method, the assessment results at each layer are interconnected with the sample selection, conforming to the layer-by-layer dependency characteristics of residual quantization clustering.

[0144] Another alternative approach is to perform quality assessment based on a pre-filtering evaluation method. Specifically, sample filtering can be pre-calculated based on the current state identifiers of all clustering levels, determining the sample range for quality assessment at each level in one go. Then, quality assessment is performed independently on the active cluster centers of each level in hierarchical order. After pre-filtering, statistical verification can be performed on the filtered sample range to confirm that the sample size for each level meets the assessment requirements. Performing quality assessment through pre-filtering decouples sample filtering from index calculation, facilitating parallel scheduling of assessment tasks at each level.

[0145] Another alternative approach is to perform quality assessment based on task scheduling. Specifically, quality assessment tasks can be scheduled hierarchically, treating the assessment of active cluster centers at each clustering level as an independent processing unit. After completing the assessment at the current level and updating the status identifier, the next level's assessment task is triggered based on the update result. Optionally, priorities and timeout controls can be set for each level's assessment tasks. If an assessment task encounters an error, exception information is recorded, and subsequent levels of assessment continue. Performing quality assessment through task scheduling facilitates progress tracking and exception handling during the assessment process.

[0146] In addition, a distributed computing framework can be used to schedule the quality assessment tasks of active cluster centers at each clustering level in parallel. While maintaining the hierarchical execution logic, multiple computing nodes can be used to accelerate the index calculation process at each level in parallel.

[0147] In the embodiments of this specification, by identifying the cluster centers in each clustering level that are in an active state, the quality assessment object is limited to the cluster centers that are currently actually participating in the coding service. This avoids invalid assessments of failed and reserved coding slots and reduces the computational overhead of quality assessment. By performing quality assessments layer by layer on each active cluster center according to the hierarchical order of the clustering levels, the assessment results of the preceding levels can constrain the assessment sample range of the following levels. This ensures that the quality assessment process matches the hierarchical structure of residual quantization clustering, thereby improving the accuracy and execution efficiency of quality assessment.

[0148] In one optional embodiment of this specification, the preset index threshold is set based on the clustering level. If the quality assessment indicators do not meet the preset threshold, the status identifier of the corresponding cluster center will be updated to an invalid state, including: For any clustering level, if the quality assessment index does not meet the preset index threshold corresponding to that clustering level, the status identifier of the corresponding cluster center will be updated to the failure state.

[0149] In practical applications, setting preset index thresholds based on clustering levels can be achieved through various methods.

[0150] One alternative approach is to set preset index thresholds based on hierarchical decay coefficients. Specifically, a corresponding preset index threshold can be configured for each clustering level, with stricter thresholds set for earlier clustering levels and more lenient thresholds for later levels. In practical applications, the preset index thresholds for each clustering level can be determined based on a baseline threshold and a decay coefficient. That is, the preset index threshold for the nth clustering level can be equal to the baseline threshold multiplied by the decay coefficient n. -1 The attenuation coefficients are set according to the characteristics of each quality assessment indicator to adapt to the natural decrease in the discriminative power of deep cluster centers as the number of layers increases. The attenuation coefficients for each cluster level can be set with a range to prevent excessive attenuation that would lead to overly lenient assessment standards for later levels. By setting preset indicator thresholds through hierarchical attenuation coefficients, each cluster level adopts an assessment standard that matches its own clustering granularity.

[0151] Another alternative approach is to set preset threshold values ​​based on historical evaluation statistics. Specifically, preset threshold values ​​can be determined based on historical quality evaluation results for each cluster level. Statistical analysis is performed on the distribution of quality evaluation indicators in historical evaluations for each cluster level, and the quantile values ​​that meet the preset distribution conditions are determined as the preset threshold values ​​for the corresponding cluster level. During the statistical process, data from abnormal evaluation periods can be removed to avoid the impact of abnormal fluctuations on threshold determination. Setting preset threshold values ​​through historical evaluation statistics, and adaptively determining thresholds based on historical data, can better reflect the actual quality distribution characteristics of each cluster level.

[0152] Another option is to set preset indicator thresholds based on business configuration calibration methods. Specifically, operations personnel can manually configure preset indicator thresholds for each cluster level based on business experience, and calibrate the thresholds according to a preset period. When there are significant changes in business scenarios or user group structure, the preset indicator thresholds for each cluster level can be readjusted. During the threshold calibration process, changes in online coding quality indicators can be referenced, and the preset indicator thresholds for the corresponding cluster level can be appropriately tightened when coding quality indicators decline.

[0153] Furthermore, preset index thresholds can be set based on the hierarchical sample distribution. Specifically, preset index thresholds can be determined based on the sample size and distribution characteristics of each cluster level. More stringent preset index thresholds can be set for cluster levels with larger sample sizes and stable distributions, while more lenient preset index thresholds can be set for cluster levels with smaller sample sizes and fluctuating distributions.

[0154] In practical applications, for any clustering level, if the quality assessment index does not meet the preset index threshold corresponding to the clustering level, the status identifier of the corresponding cluster center will be updated to the failure state. This can also be achieved through various methods.

[0155] One alternative approach is to perform failure determination based on a hierarchical threshold adaptation method. Specifically, when performing a quality assessment of a specific clustering level, the preset threshold values ​​corresponding to that clustering level are first read. The quality assessment indicators of each cluster center within that level are then compared one by one with the preset threshold values. If a quality assessment indicator does not meet the corresponding preset threshold value, the status of the corresponding cluster center is updated to a failure state. During the determination process, the clustering level to which each cluster center belongs and the preset threshold values ​​used can be recorded, forming a hierarchical assessment and determination record. By performing failure determination using a hierarchical threshold adaptation method, the determination process strictly corresponds to the hierarchical configuration.

[0156] Another alternative approach is to perform failure determination based on a threshold mapping query. Specifically, a mapping relationship between cluster levels and preset indicator thresholds can be established in advance. During quality assessment, the preset indicator threshold is queried based on the cluster level to which the cluster center belongs, and then the quality assessment indicator is compared with the query result for failure determination. The mapping relationship allows for maintaining thresholds for each cluster level according to the type of quality assessment indicator, supporting different level threshold configurations for different indicators. Failure determination via threshold mapping query is suitable for codebase structures with a large number of cluster levels.

[0157] In addition, after the failure determination of each cluster level is completed, inter-level verification can be performed on the determination results to verify whether the failure ratio of each cluster level is within the preset reasonable range. If the failure ratio is abnormal, the preset index threshold of the corresponding cluster level can be reviewed and adjusted.

[0158] In a specific application scenario, if the quality assessment results of all cluster centers in any clustering level do not meet the corresponding preset index threshold, all cluster centers in the corresponding clustering level can be abandoned. That is, the corresponding clustering level does not produce effective semantic codes, in order to prevent the cascading failure of the preceding level from propagating to the subsequent level and avoid cascading collapse leading to the overall failure of the clustering codebook.

[0159] In the embodiments of this specification, by setting the preset index thresholds based on the corresponding clustering levels, the quality assessment criteria are matched with the hierarchical characteristics of residual quantization clustering, thus preventing deep cluster centers from being incorrectly marked as invalid due to a natural decrease in discriminability. By updating the status identifier of the corresponding cluster center to the invalid state when the quality assessment index does not meet the preset index thresholds corresponding to the corresponding clustering level, the status determination of the cluster center is based on the adaptation criteria of its level, which improves the fairness and accuracy of quality assessment across different clustering levels and ensures the stable operation of the codebook quality assessment mechanism.

[0160] In an optional embodiment of this specification, after determining the cluster centers whose state is marked as active at each cluster level, the method further includes: Determine the number of sample user representation vectors corresponding to each cluster center whose state is marked as active in each cluster level; If the number of sample user representation vectors corresponding to any cluster center is less than the preset clustering threshold, skip that cluster center and perform quality assessment. If the total number of sample user representation vectors corresponding to any cluster level is less than the preset level threshold, the quality assessment of that cluster level is skipped.

[0161] Sample user representation vectors are user representation vector samples used in cluster codebook construction or quality assessment. Specifically, sample user representation vectors can include user representation vectors from historical user representation vectors that participated in cluster training at the corresponding cluster level. The sample user representation vectors corresponding to each cluster center are the user representation vectors assigned to the corresponding cluster center during cluster training. The set of sample user representation vectors constitutes the data foundation for cluster codebook construction and quality assessment. Sample user representation vectors can serve as a data source for calculating quality assessment metrics. Boundary stability metrics, minimum clustering percentage metrics, coefficient of variation metrics, and residual explained rate metrics are all calculated based on the distribution characteristics of sample user representation vectors. Furthermore, sample user representation vectors and current user representation vectors have different roles. Current user representation vectors are used for coverage detection and incremental updates, while sample user representation vectors are used for quality assessment. In specific application scenarios, quality assessment is performed based on historical user representation vectors to avoid logical conflicts between using current user representation vectors for quality assessment and incremental detection.

[0162] A preset clustering threshold is a pre-defined lower limit on the number of sample user representation vectors corresponding to a cluster center. It is used to determine whether the sample size of a cluster center is sufficient to support a reliable quality assessment. Specifically, the preset clustering threshold can be compared with the number of sample user representation vectors corresponding to each active cluster center. If the number of sample user representation vectors corresponding to a cluster center is less than the preset clustering threshold, the quality assessment of that cluster center is skipped. In specific application scenarios, the preset clustering threshold can be tens to hundreds of values. Furthermore, the preset clustering threshold can serve as a single-encoding dimension criterion for confidence protection mechanisms. When the sample size is too small, the quality assessment results are unstable, and the quality assessment indicators calculated based on a small number of samples may fluctuate randomly. Skipping quality assessments can prevent normal cluster centers from being incorrectly marked as invalid.

[0163] The preset level threshold is a lower limit for the total number of sample user representation vectors corresponding to a pre-defined clustering level. It is used to determine whether the total number of samples at a clustering level is sufficient to support the quality assessment of all cluster centers at that level. Specifically, the preset level threshold can be compared with the total number of sample user representation vectors corresponding to each clustering level. If the total number of sample user representation vectors at a clustering level is less than the preset level threshold, the quality assessment of that clustering level is skipped. In specific application scenarios, the preset level threshold can be the product of the number of cluster centers at the corresponding clustering level and the preset level threshold. The preset level threshold can serve as the level dimension criterion for the confidence protection mechanism. Furthermore, by using both single cluster center and clustering level dual-reset confidence protection, the instability of quality assessment due to insufficient sample size is avoided, ensuring the credibility of the failure markers.

[0164] In practical applications, the number of sample user representation vectors corresponding to each cluster center with an active state in each cluster level can be determined by various methods.

[0165] One alternative approach is to determine the number of sample user representation vectors based on sample attribution statistics. Specifically, this involves iterating through the sample user representation vectors within each cluster corresponding to an active cluster center, counting the number of sample user representation vectors corresponding to each cluster center, and summing the sample counts for each cluster level to obtain the total number of sample user representation vectors at each cluster level. During the statistical process, the distribution characteristics of the samples corresponding to each cluster center can be recorded simultaneously, providing data support for subsequent quality assessment index calculations. Determining the number of sample user representation vectors through sample attribution statistics yields accurate results and facilitates hierarchical summarization.

[0166] Another alternative approach is to determine the number of sample user representation vectors based on a counting cache maintenance method. Specifically, during cluster training or codebook updates, sample count information for each cluster center can be maintained. When the sample partitioning results are updated, the sample user representation vector counts for each cluster center are updated synchronously. During quality assessment, the maintained count information is directly read. When the count information is updated, an update timestamp can be recorded, and re-counting is triggered when the count information exceeds a preset time limit. Determining the number of sample user representation vectors through a counting cache maintenance method avoids redundant counting and significantly reduces statistical overhead in scenarios with large sample sizes.

[0167] Furthermore, the number of sample user representation vectors can be determined based on a distributed parallel statistical approach. Specifically, the number of sample user representation vectors at each cluster level can be aggregated in parallel using a distributed statistical framework, and the local statistical results on each computing node can be globally merged to obtain the sample quantity information for each cluster center and cluster level.

[0168] In practical applications, if the number of sample user representation vectors corresponding to any cluster center is less than the preset clustering threshold, the corresponding cluster center can be skipped for quality assessment. This can also be achieved through various methods.

[0169] One alternative approach is to skip cluster centers with insufficient samples based on a center-by-center threshold comparison. Specifically, the number of user representation vectors corresponding to each active cluster center is compared with a preset clustering threshold. If the number of samples is less than the preset threshold, the corresponding cluster center is marked as skipped from evaluation, its quality evaluation index is not calculated, and its status remains unchanged. The number of samples and the preset clustering threshold are recorded for each skipped cluster center, forming a record of the reason for skipping evaluation, which can be used as a reference for re-evaluation after subsequent sample accumulation. Skipping cluster centers with insufficient samples using a center-by-center threshold comparison is simple and provides a unified judgment standard.

[0170] Another alternative approach is to skip cluster centers with insufficient samples using a batch filtering method. Specifically, active cluster centers can be first sorted according to the number of user representation vectors in the sample data. Cluster centers with fewer than a preset clustering threshold are then filtered out in batches and their quality assessment is skipped. After batch filtering, quality assessment can be performed on the remaining cluster centers, including only those with sufficient samples. Skipping cluster centers with insufficient samples through batch filtering reduces the number of center-by-center comparisons and is suitable for scenarios with a large number of cluster centers.

[0171] In addition, the reason for skipping the cluster center quality assessment and the corresponding sample quantity information can be recorded and output to the quality assessment log for reference when reassessing after sufficient sample accumulation.

[0172] In practical applications, skipping the quality assessment of the corresponding clustering level when the total number of sample user representation vectors corresponding to any clustering level is less than the preset level threshold can also be achieved through various methods.

[0173] One alternative approach is to skip clustering levels with insufficient samples based on a hierarchical total quantity pre-detection method. Specifically, before performing quality assessment for a given clustering level, the number of user representation vectors corresponding to all active cluster centers within that level is first aggregated. This total number of samples is then compared to a preset level threshold. If the total number of samples is less than the threshold, the quality assessment of all cluster centers within that level is skipped, while the status identifiers of each cluster center remain unchanged. Only after the hierarchical total quantity pre-detection passes is the quality assessment of each cluster center within that level performed, ensuring that the assessment is conducted under conditions of sufficient samples. By skipping clustering levels with insufficient samples through hierarchical total quantity pre-detection, meaningless assessments are avoided for levels with insufficient samples.

[0174] Another alternative approach is to skip clustering levels with insufficient samples based on a dynamic cumulative judgment method. Specifically, during the center-by-center quality assessment, the total number of samples for the corresponding level can be accumulated simultaneously. Level skipping is not triggered until the accumulated sample total reaches a preset level threshold. If insufficient samples are detected, the assessment of the remaining clustering centers at the corresponding level is terminated. After termination, the validity of the completed assessment results is verified, and assessment results calculated based on insufficient samples are marked as low-confidence results. By skipping clustering levels with insufficient samples through a dynamic cumulative judgment method, invalid assessment tasks can be terminated promptly.

[0175] In addition, after skipping the clustering level quality assessment, a level sample size alarm can be output to prompt the operations staff to supplement the sample data, and the corresponding clustering level supplementary assessment process can be triggered after sufficient sample data has been accumulated.

[0176] In the embodiments of this specification, by determining the number of sample user representation vectors corresponding to each active cluster center, a data foundation is provided for the confidence judgment of the quality assessment; by skipping the quality assessment of the corresponding cluster center when the number of samples in the cluster center is less than a preset clustering threshold, the unstable indicators calculated based on a small number of samples are avoided from causing normal cluster centers to be incorrectly marked as invalid; by skipping the quality assessment of the corresponding cluster level when the total number of samples in the cluster level is less than a preset level threshold, unreliable assessments are avoided at levels with insufficient data scale. Through the dual reset confidence protection of single cluster center and cluster level, the stable operation of the quality assessment mechanism and the credibility of the invalidation marking are ensured in scenarios with limited data scale.

[0177] In one optional embodiment of this specification, determining uncovered user representation vectors from the current user representation vector based on the distance between the current user representation vector and the center vectors of multiple cluster centers includes: Calculate the distance between each user representation vector in the current user representation vector and the center vector of multiple cluster centers; Based on the statistical characteristics of the distance between each current user representation vector and the center vectors of multiple cluster centers, the uncovered user representation vectors are determined from the current user representation vectors.

[0178] Statistical features are numerical indicators that provide a general description of the distribution characteristics of a set of data, used to characterize the distribution pattern of the data at an overall level. Specifically, statistical features may include the mean and median reflecting the central tendency of the data; the standard deviation and coefficient of variation reflecting the dispersion of the data; the interquartile range reflecting the shape of the data distribution; and the tail quantiles and maximum values ​​reflecting extreme cases of the data. In the embodiments of this specification, statistical features can be used to provide a general description of the distance distribution between the current user representation vector and the center vector of the cluster center. Statistical features can serve as a quantitative basis for coverage determination. By comparing the statistical features of the distance distribution with preset coverage conditions, the degree of matching between the current user representation vector and the existing cluster structure can be determined. Further, in specific application scenarios, one of the statistical features—the ratio of the tail quantile to the median, the ratio of the interquartile range to the median, the ratio of the maximum value to the median, and the coefficient of variation—can be selected for determination. Different statistical features have different sensitivities to extreme outliers, and can be flexibly selected according to the business scenario.

[0179] In practical applications, the distance between each user representation vector in the current user representation vector and the center vector of multiple cluster centers can be calculated using various methods.

[0180] One alternative approach is to calculate the distance using a sample-by-sample precise calculation method. Specifically, the Euclidean distance between each user representation vector and the center vectors of each cluster can be calculated one by one, and the minimum distance corresponding to each user representation vector can be determined. That is, the distance between each user representation vector and the center vector of its nearest cluster is used as the distance corresponding to the corresponding user representation vector. During the distance calculation process, the nearest cluster center number and distance value corresponding to each user representation vector can be recorded simultaneously to form a detailed distance record for use in coverage determination and subsequent analysis. Calculating the distance using a sample-by-sample precise calculation method offers high accuracy and is suitable for codebooks with a moderate number of cluster centers.

[0181] Another alternative approach is to calculate distances using batch matrix operations. Specifically, the current user representation vectors can be organized into a vector matrix. Batch matrix operations can then be used to calculate the distance matrix between all user representation vectors and the center vectors of all cluster centers at once. The minimum distance for each user representation vector can then be extracted based on this distance matrix. During the matrix operations, a parallel computing framework can be used to perform block-wise calculations on the distance matrix, and numerical verification can be performed after the calculations are complete to eliminate outliers. Calculating distances using batch matrix operations fully leverages the parallel capabilities of matrix operations, significantly improving distance calculation efficiency in scenarios with a large number of user representation vectors.

[0182] Another alternative approach is to calculate distance based on a vector retrieval index. Specifically, a vector retrieval index can be pre-built based on the center vectors of cluster centers. This index quickly locates the nearest cluster center for each user representation vector, calculating only the distance between the user representation vector and the nearest cluster center, significantly reducing the computational complexity through approximation. A retrieval precision parameter can be set within the vector retrieval index to balance retrieval accuracy and computational efficiency, and the index is periodically updated to maintain consistency between the index and the cluster centers. Calculating distance using a vector retrieval index is suitable for scenarios with a large number of cluster centers or high real-time computation requirements.

[0183] In addition, distance can be calculated based on sampling estimation. Specifically, the current user representation vector can be sampled first, and the distance between the sampled subset of user representation vectors and the center vector of the cluster center can be calculated. Then, the coverage of the full set of user representation vectors can be inferred based on the distance distribution of the sampled subset, so as to reduce the overall overhead of distance calculation.

[0184] Once the distance is calculated, the uncovered user representation vectors can be determined from the current user representation vectors based on the statistical characteristics of the distance between each current user representation vector and the center vectors of multiple cluster centers.

[0185] One alternative method is to determine the uncovered user representation vectors based on the tail distance ratio. Specifically, the ratio of the tail quantile to the median of the distances corresponding to all current user representation vectors can be calculated. This ratio is then compared with a threshold corresponding to a preset coverage condition. If the ratio meets the preset coverage condition, it is determined that an uncovered user group exists, and the uncovered user representation vectors are determined based on the distances corresponding to each user representation vector. In a specific application scenario, the tail quantile can be the 99th quantile. The larger the ratio of the 99th quantile to the median, the more it indicates the existence of a large number of tail users far from all cluster centers, i.e., a potential new user group. Optionally, different tail quantile configurations can be used for different cluster levels during the ratio comparison process to adapt to the distribution characteristics of each cluster level. Determining the uncovered user representation vectors using the tail distance ratio method, with the ratio of the extreme tail distance to the overall distance level as the main criterion, is sensitive to outliers distributed far from the cluster centers.

[0186] Another alternative approach is to determine the uncovered user representation vectors based on the degree of dispersion of distance. Specifically, the ratio of the standard deviation to the mean of the distance corresponding to each user representation vector can be calculated, i.e., the coefficient of variation of the distance distribution. If the coefficient of variation exceeds a threshold corresponding to a preset coverage condition, it is determined that there is an uncovered user group, and the uncovered user representation vectors are determined based on distance. During the calculation of the coefficient of variation, robust statistical processing can be performed on the distance distribution, using the absolute deviation of the median instead of the standard deviation to reduce the impact of extreme values ​​on the dispersion measure. Determining the uncovered user representation vectors by using the degree of dispersion of distance, and using the overall dispersion of the distance distribution as the criterion, can detect situations where a large number of samples are far from all cluster centers.

[0187] Another alternative method is to determine the representation vector of uncovered users based on the interquartile range ratio. Specifically, the ratio of the interquartile range to the median can be calculated. If the ratio meets a preset coverage condition, an uncovered user group is identified, and the representation vector of uncovered users is determined based on the distance. During the interquartile range calculation, the interval between the upper and lower quartiles can be used as the measurement range of the main part of the distance distribution to reflect the dispersion of the main data component. Determining the representation vector of uncovered users using the interquartile range ratio method has good robustness to extreme outliers.

[0188] Furthermore, the representation vector of uncovered users can be determined based on the extreme value ratio method. Specifically, the ratio of the maximum distance to the median distance can be calculated, and the extreme values ​​in the distance distribution can be used as criteria. If the extreme value ratio meets the preset coverage conditions, it can be determined that there is an uncovered user group, which can be used to capture the extreme outliers with the largest distances.

[0189] In the embodiments of this specification, the degree of matching between user representation vectors and cluster structures is quantified into a calculable value by calculating the distance between each user representation vector in the current user representation vector and the center vector of multiple cluster centers, providing basic data for coverage determination; by determining the uncovered user representation vectors based on distance-based statistical features, the deviation of the current user group from the existing cluster structure is judged from the distribution level as a whole, avoiding the sensitivity of sample-by-sample threshold comparison to extreme distributions, giving the identification of coverage gaps a holistic perspective, and improving the robustness and efficiency of coverage detection.

[0190] In one optional embodiment of this specification, determining uncovered user representation vectors from the current user representation vectors based on statistical characteristics of the distances between each current user representation vector and the center vectors of multiple cluster centers includes: Based on the position of each cluster level in the current cluster codebook, determine the preset coverage conditions corresponding to each cluster level; Based on the statistical characteristics of the distance between each current user representation vector and the center vector of multiple cluster centers, and the comparison results between the distance and the preset coverage conditions, it is determined whether there is an uncovered user group. If it is determined that there is an uncovered user group, the uncovered user representation vector is determined from the current user representation vector based on the distance between the current user representation vector and the center vectors of multiple cluster centers.

[0191] Preset coverage conditions are pre-defined criteria used to determine whether the current user representation vector is sufficiently covered by the cluster codebook. Specifically, preset coverage conditions can include threshold conditions corresponding to distance statistical features, such as thresholds for the ratio of the tail ternary quantile to the median, the ratio of the interquartile range to the median, the ratio of the maximum value to the median, and the coefficient of variation. Preset coverage conditions can be set based on the position of each cluster level in the current cluster codebook. Preset coverage conditions for earlier cluster levels are more stringent, while those for later cluster levels are more lenient. Preset coverage conditions can serve as a criterion for coverage detection. By comparing the distance statistical features with the preset coverage conditions, it can be determined whether codebook expansion is necessary. Furthermore, in specific application scenarios, the preset coverage conditions for earlier frozen levels are more conservative to suppress invalid expansion triggered by normal fluctuations, while the preset coverage conditions for later dynamic levels are relatively lenient to sensitively respond to changes in the population within deep clusters. The preset coverage conditions for each cluster level can be determined based on a baseline threshold and an adjustment amount that increases with the level position.

[0192] The uncovered user group refers to the user group whose current user representation vector is not fully covered by existing cluster centers. The distance between their representation vector and the center vectors of all existing cluster centers does not satisfy the coverage condition. Specifically, the uncovered user group can include newly emerging user groups that do not have corresponding cluster centers in the existing clustering codebook. Their representation vectors are distributed outside the coverage area of ​​existing cluster centers, and the user representation vectors corresponding to the uncovered user group are called the uncovered user representation vectors. The uncovered user group can serve as the trigger for incremental codebook expansion. By identifying new cluster centers for them, the new group can obtain usable discretized codes. Furthermore, the size and distribution range of the uncovered user group reflect the degree of coverage gap in the current clustering codebook and are important bases for determining whether codebook expansion is needed and determining the expansion scale.

[0193] In practical applications, the preset coverage conditions corresponding to each cluster level can be determined based on the position of each cluster level in the current cluster codebook, which can be achieved through various methods.

[0194] One alternative approach is to determine preset coverage conditions based on hierarchical location configuration. Specifically, preset coverage conditions can be configured for each cluster level, with stricter conditions set for earlier cluster levels and more lenient conditions for later levels. In practical applications, preset coverage conditions for each cluster level can be determined based on a baseline threshold and an adjustment amount that increases with hierarchical location. Earlier frozen levels use more conservative coverage conditions to suppress invalid expansion triggered by normal fluctuations, while later dynamic levels use relatively lenient coverage conditions to respond sensitively to population changes. During configuration, the coverage condition configuration parameters for each cluster level can be recorded to form a hierarchical coverage condition configuration table for querying during coverage determination. Determining preset coverage conditions through hierarchical location configuration ensures that the coverage detection standard matches the location characteristics of the cluster levels.

[0195] Another alternative approach is to determine the preset coverage conditions based on historical expansion frequency statistics. Specifically, preset coverage conditions can be determined based on the historical expansion frequency statistics of each cluster level. More stringent preset coverage conditions can be set for cluster levels with lower historical expansion frequencies, while more lenient preset coverage conditions can be set for cluster levels with higher historical expansion frequencies. During the statistical process, the expansion frequency of each cluster level can be statistically analyzed in segments according to time windows, with the most recent expansion frequency serving as the primary basis for determining the preset coverage conditions. By determining the preset coverage conditions through historical expansion frequency statistics, and adaptively determining the coverage conditions for each level based on historical data, the actual variation patterns of each cluster level can be closely aligned.

[0196] Another alternative approach is to determine the preset coverage conditions based on the hierarchical sample distribution. Specifically, the preset coverage conditions can be determined based on the sample distribution scale of each cluster level. Stricter preset coverage conditions can be set for cluster levels with larger sample sizes and more stable distributions, while more lenient preset coverage conditions can be set for cluster levels with smaller sample sizes and greater distribution fluctuations. This ensures that the coverage conditions match the actual data conditions at each level. When the sample distribution changes, the preset coverage conditions for the corresponding cluster levels can be dynamically adjusted to maintain the adaptability of the coverage conditions to the hierarchical data conditions.

[0197] In addition, preset coverage conditions can be determined based on business configuration calibration methods. Specifically, operations personnel can configure preset coverage conditions for each cluster level based on business experience, and periodically calibrate the preset coverage conditions in conjunction with feedback on online coding performance.

[0198] Given the preset coverage conditions for each cluster level, it is possible to determine whether there are uncovered user groups based on the statistical characteristics of the distance between each current user representation vector and the center vectors of multiple cluster centers, and the comparison results between the preset coverage conditions and the preset coverage conditions.

[0199] One alternative approach is to determine the existence of uncovered user groups based on tail ratio comparison. Specifically, the ratio of the tail quantile to the median of the distance corresponding to the current user's representation vector can be calculated. This ratio is then compared to a preset coverage condition for the corresponding cluster level. If the ratio meets the preset coverage condition, an uncovered user group is identified. In specific applications, the tail quantile can be the 99th quantile. If the ratio of the 99th quantile to the median exceeds the threshold corresponding to the preset coverage condition, an uncovered user group is identified. During the ratio comparison process, outliers in the distance distribution can be filtered, and the ratio can be recalculated after filtering to improve the stability of the ratio estimation. Determining the existence of uncovered user groups by comparing tail ratios, using the extreme tail distance as the primary criterion, can effectively identify tail user groups far from the cluster centers in the distance distribution.

[0200] Another alternative approach is to determine the existence of uncovered user groups based on a dispersion comparison method. Specifically, the coefficient of variation (CV) of the distance distribution can be calculated and compared with a preset coverage condition corresponding to the cluster level. If the CV exceeds a threshold corresponding to the preset coverage condition, an uncovered user group is identified. Robust statistics can be used instead of standard deviation in the CV calculation process to reduce the interference of extreme outliers on the dispersion measure. Determining the existence of uncovered user groups by comparing dispersion, using the overall dispersion of the distance distribution as the criterion, can sensitively trigger the determination even when a large number of samples are far from all cluster centers.

[0201] Another alternative method is to determine the existence of uncovered user groups based on interquartile range (IQR) comparison. Specifically, the ratio of the IQR to the median can be calculated, and this ratio can be compared with preset coverage conditions. If the ratio meets the preset conditions, an uncovered user group is identified. During the IQR comparison, skewness information of the distance distribution can be incorporated; if the ratio meets the preset coverage conditions and the distribution shows a significant right skewness, an uncovered user group is identified. Using IQR comparison to determine the existence of uncovered user groups, and measuring the dispersion of the main data component with IQR, provides good robustness against extreme outliers.

[0202] In addition, the existence of uncovered user groups can be determined based on extreme value comparison. Specifically, the ratio of the maximum distance to the median distance can be calculated. If the extreme value ratio meets the preset coverage conditions, it can be determined that there are uncovered user groups, which can be used to capture the extreme outliers with the largest distances.

[0203] If it is determined that there is an uncovered user group, the uncovered user representation vector can be determined from the current user representation vector based on the distance between the current user representation vector and the center vectors of multiple cluster centers.

[0204] One alternative approach is to determine the uncovered user representation vectors based on a distance-based sorting and truncation method. Specifically, after identifying an uncovered user group, the distances corresponding to each user representation vector in the current user representation vector list are sorted, and the user representation vectors with the highest distances (a predetermined percentage) are identified as uncovered user representation vectors. In specific application scenarios, when using the ratio of the 99th quartile to the median for judgment, the user representation vector with the largest distance (1%) can be selected as the uncovered user representation vector; when using the ratio of the interquartile range to the median, the user representation vector with the largest distance (25%) can be selected as the uncovered user representation vector. During the sorting and truncation process, sorting and truncation can be performed separately for each cluster level, ensuring that the size of the uncovered user representation vectors at each cluster level matches the corresponding judgment metric. By determining the uncovered user representation vectors through distance-based sorting and truncation, the size of the uncovered user representation vectors corresponds to the selected judgment metric.

[0205] Another alternative approach is to determine the uncovered user representation vectors based on a distance threshold. Specifically, all user representation vectors exceeding a preset distance threshold can be identified as uncovered user representation vectors, directly dividing covered and uncovered user representation vectors by the distance threshold. During the distance threshold division process, the statistical characteristics of the distance distribution can be referenced to determine the value of the preset distance threshold, ensuring that the threshold division result aligns with the distribution pattern. Determining uncovered user representation vectors using the distance threshold method offers intuitive rules, as the range of uncovered user representation vectors is directly determined by the distance threshold.

[0206] Another alternative approach is to determine uncovered user representation vectors based on density region identification. Specifically, low-density regions in the current user representation vector distribution can be identified using kernel density estimation methods, and user representation vectors falling into these low-density regions are identified as uncovered user representation vectors. During the low-density region identification process, distance criteria can be incorporated to identify user representation vectors that simultaneously meet both low-density and long-distance conditions as uncovered user representation vectors. By determining uncovered user representation vectors through density region identification and combining this method with distribution density information, sparse regions that are difficult to cover using distance criteria can be identified.

[0207] In addition, distance sorting can be combined with distance threshold determination. First, candidate vectors are coarsely screened based on the distance threshold, and then the uncovered user representation vectors are determined from the candidate vectors based on the distance sorting.

[0208] In the embodiments of this specification, the preset coverage conditions corresponding to each cluster level are determined based on the position of each cluster level in the current cluster codebook, so that the coverage detection standard matches the positional characteristics of the cluster level. The first level uses conservative conditions to suppress invalid expansion triggered by normal fluctuations, while the second level uses lenient conditions to sensitively respond to population changes. The existence of uncovered user groups is determined by comparing the statistical features based on distance with the preset coverage conditions, so that the identification of new groups has clear quantitative criteria and can be flexibly selected among various statistical features according to business scenarios. After determining the existence of uncovered user groups, the uncovered user representation vector is determined based on distance, which provides a precise target for determining the new cluster centers and realizes the connection between automatic discovery of new groups and codebook expansion.

[0209] In one optional embodiment of this specification, determining new cluster centers based on uncovered user representation vectors includes: Keep the center vectors of the multiple cluster centers that already exist in the current clustering codebook unchanged, and determine the number of clusters based on the distribution characteristics of the uncovered user representation vectors; Based on the number of clusters, the uncovered user representation vectors are clustered to obtain the center vector of each cluster, and the center vector of each cluster is determined as the new cluster center.

[0210] Distribution characteristics are properties describing the distribution of a set of vectors in a vector space, used to characterize the aggregation, diffusion, and structural information of data in space. Specifically, distribution characteristics may include the location of dense regions of vector distribution, the degree of dispersion of distribution, the directionality of distribution, and peak structure in distribution. In the embodiments of this specification, distribution characteristics can be used to describe the distribution of uncovered user representation vectors in the vector space to determine a reasonable number of clusters. Distribution characteristics can serve as the basis for determining the number of clusters. By analyzing the distribution characteristics of uncovered user representation vectors, the number of clusters that can cover their dense regions can be determined. Further, in specific application scenarios, the number of clusters can be automatically determined using the elbow method based on the distribution characteristics of uncovered user representation vectors. That is, the clustering error is calculated for different candidate cluster numbers, and the number of clusters corresponding to the inflection point where the decreasing trend of clustering error changes significantly is determined as the number of clusters.

[0211] The number of clusters is the number of clusters determined when clustering uncovered user representation vectors, thus determining the number of new cluster centers. Specifically, the number of clusters can be determined based on the distribution characteristics of the uncovered user representation vectors, including the number of densely distributed regions within them. The value of the number of clusters can be dynamically determined based on the size and dispersion of the uncovered user representation vectors. The number of clusters can be used as a parameter input for the clustering operation, which divides the uncovered user representation vectors into a corresponding number of clusters. Furthermore, determining the number of clusters requires a balance between sufficient coverage and codebook redundancy. Too few clusters will result in insufficient coverage of the uncovered user representation vectors with the new cluster centers, while too many clusters will lead to codebook redundancy and decreased coding discriminability.

[0212] Clusters are sample groups obtained by clustering the representation vectors of uncovered users. Each cluster contains a group of user representation vectors with similar distributions. Specifically, clusters can include groups obtained by dividing the representation vectors of uncovered users through clustering operations. Each cluster has a corresponding center vector. The distance between the user representation vectors within a cluster and the cluster center vector is small, and the center vectors of different clusters maintain a certain degree of distinguishability. Clusters can serve as the source of new cluster centers, and the center vectors of each cluster are determined as new cluster centers. Furthermore, the quality of cluster division directly affects the representation accuracy of the uncovered user group by the new cluster centers. The more concentrated the samples within a cluster, the stronger the representativeness of the corresponding new cluster centers.

[0213] In practical applications, the number of clusters can be determined by keeping the center vectors of the existing cluster centers in the current clustering codebook unchanged and determining the number of clusters based on the distribution characteristics of the uncovered user representation vectors. This can be achieved through various methods.

[0214] One alternative approach is to determine the number of clusters based on a combination of freeze constraints and the elbow method. Specifically, during codebook updates, the center vectors of existing cluster centers can be frozen, preventing them from participating in iterative cluster updates. Simultaneously, the number of clusters is determined based on the distribution characteristics of uncovered user representation vectors. In specific applications, the elbow method can be used to analyze clustering errors in uncovered user representation vectors, identifying the number of clusters corresponding to the inflection point where the clustering error decline trend significantly changes. During the elbow method analysis, the range of candidate cluster numbers can be constrained, limiting the number of clusters to a preset maximum expansion number. By determining the number of clusters through a combination of freeze constraints and the elbow method, the expansion scale is adaptively determined based on data distribution while maintaining the existing code as completely static.

[0215] Another alternative approach is to determine the number of clusters based on density region counting. Specifically, the number of dense regions can be estimated based on the density distribution of the uncovered user representation vectors. The number of dense regions in the distribution of uncovered user representation vectors is then determined as the number of clusters, so that each newly added cluster center corresponds to one dense region. During the dense region counting process, adjacent dense regions can be merged; if the interval between dense regions is less than a preset interval threshold, they can be merged into one dense region. By determining the number of clusters through density region counting, the number of clusters matches the actual distribution structure of the uncovered user group.

[0216] Another alternative method is to determine the number of clusters based on a sample size ratio. Specifically, the number of clusters can be determined based on the size of the uncovered user representation vectors and a preset ratio. For example, the number of clusters can be determined according to the ratio of the total number of uncovered user representation vectors to the preset sample size of a single cluster, ensuring that the sample size of each cluster remains within a reasonable range. During the ratio calculation, the number of clusters can be adjusted based on the dispersion of the uncovered user representation vectors; the higher the dispersion, the more clusters should be added. Determining the number of clusters by sample size ratio is computationally simple and suitable for scenarios where the size of the uncovered user representation vectors is known.

[0217] In addition, the number of clusters can be determined based on the business expansion limit. Specifically, the number of clusters can be determined based on the business's preset expansion scale limit. When the number of dense regions that do not cover user representation vectors exceeds the expansion limit, the number of clusters is limited to the expansion limit to control the expansion range of the codebook.

[0218] Given a fixed number of clusters, the uncovered user representation vectors can be clustered based on the number of clusters to obtain the center vector of each cluster, and the center vector of each cluster can be determined as the new cluster center.

[0219] One alternative approach is to determine new cluster centers based on K-means clustering. Specifically, the number of clusters can be used as the parameter for the number of cluster centers. K-means clustering is performed on the uncovered user representation vectors, and the sum of squared distances between each sample and its cluster center vector is minimized through iterative optimization. After clustering convergence, the center vectors of each cluster are obtained and determined as the new cluster centers. During the clustering initialization phase, K-means addition can be used to select initial cluster centers to improve the stability of the clustering results. During the clustering iteration process, a maximum number of iterations and a convergence threshold can be set. Determining new cluster centers using K-means clustering, based on a mature K-means clustering algorithm, ensures a stable and controllable clustering process.

[0220] Another alternative approach is to determine new cluster centers based on hierarchical clustering. Specifically, hierarchical clustering can be used to cluster uncovered user representation vectors. Starting with each user representation vector as an independent cluster, the two closest clusters are iteratively merged until the number of clusters reaches the required number. The center vector of each merged cluster is then determined as the new cluster center. During the hierarchical clustering process, the distance information of each merge can be recorded, and the reasonableness of the number of clusters can be verified based on the jumps in the merge distance. Determining new cluster centers using hierarchical clustering eliminates the need to pre-specify initial cluster centers, and the clustering results are insensitive to initial conditions.

[0221] Another alternative approach is to determine new cluster centers based on density clustering. Specifically, density clustering can be used to cluster uncovered user representation vectors. Based on the density distribution of these vectors, dense regions are automatically identified and clusters are formed. The center vector of each cluster is then determined as the new cluster center. After density clustering, the cluster size is validated, and clusters that are too small are merged or removed. Determining new cluster centers using density clustering can identify cluster structures of arbitrary shapes, making it suitable for scenarios with complex distributions of uncovered user representation vectors.

[0222] In addition, parallel clustering can be performed on uncovered user representation vectors based on a distributed clustering framework. The uncovered user representation vectors are distributed to multiple computing nodes to perform local clustering. Then, the local clustering results are globally merged and the center is adjusted to obtain the center vector of each cluster.

[0223] In the embodiments of this specification, by keeping the center vectors of multiple existing cluster centers in the current clustering codebook unchanged, the existing encoded semantics are made completely static, and the encoding results of users with assigned semantic identifiers do not change due to codebook updates, thus ensuring the historical association memory of downstream models. By determining the number of clusters based on the distribution characteristics of the representation vectors of uncovered users, the number of new cluster centers is matched with the actual distribution structure of the uncovered user group, avoiding codebook redundancy or insufficient coverage. By clustering the representation vectors of uncovered users based on the number of clusters and determining the center vector of each cluster as the new cluster center, the new cluster centers accurately cover the uncovered user group, realizing the adaptive expansion of the clustering codebook under the premise of stable encoded semantics.

[0224] In an optional embodiment of this specification, after determining the new cluster centers based on the uncovered user representation vectors, the method further includes: Distribution detection is performed on the sample user representation vectors within the clusters corresponding to each cluster center; If the distribution of sample user representation vectors within a cluster corresponding to any cluster center shows multiple peaks, the sample user representation vectors within the cluster are divided into a preset number of sub-clusters. The center vector of each sub-cluster is determined as the new cluster center.

[0225] Distribution detection is the process of detecting the distribution pattern of user representation vectors of samples within a cluster corresponding to a cluster center. Specifically, distribution detection may include analyzing the distribution of distances between all user representation vectors within a cluster and the center vector of its cluster center, determining whether the distribution exhibits a unimodal or multimodal structure. A unimodal distribution indicates that the samples within the cluster are homogeneous, while a multimodal distribution indicates that the samples within the cluster have diverged. Kernel density estimation methods can be used to estimate the density of the sample distribution to identify the peak structure. Distribution detection can serve as a core step in split population detection, and the detection results determine whether clusters need to be split. Furthermore, the execution order of distribution detection follows the determination of new cluster centers based on uncovered user representation vectors; that is, new populations are detected first, and new cluster centers are added, followed by split detection of samples in clusters corresponding to existing cluster centers. The two are executed sequentially, and split population detection can be understood as a special case of new population detection.

[0226] Multiple peaks refer to the phenomenon where the distribution of user representation vectors within a cluster corresponding to a cluster center exhibits multiple local density maxima, indicating that the samples within the cluster have differentiated into multiple subgroups. Specifically, multiple peaks can include situations where two or more density maxima appear in the sample distance distribution. For example, the distribution of samples within a cluster may exhibit a bimodal structure, indicating that user groups originally clustered under the same cluster center have significantly differentiated over time, and a single cluster center can no longer accurately describe the differences within the group. Multiple peaks can serve as a trigger condition for cluster splitting. When multiple peaks are detected, sub-clustering is performed on the user representation vectors within the cluster. Furthermore, the initial stage of population splitting typically exhibits a bimodal structure; splitting the bimodal structure can capture the main differentiation, thereby avoiding excessive fragmentation of the codebook.

[0227] Sub-clusters are subgroups obtained by sub-clustering sample user representation vectors within clusters exhibiting multiple peaks. Specifically, sub-clusters can include subgroups obtained by dividing the sample user representation vectors within a cluster into a predetermined number of groups. In specific application scenarios, the predetermined number can be two, meaning the sample user representation vectors within a cluster are fixedly split into two sub-clusters, each with a corresponding center vector. Sub-clusters can serve as the source of new cluster centers; the center vector of each sub-cluster is determined as the new cluster center, and the status flag of the corresponding cluster center is set to active. Furthermore, adopting a conservative splitting strategy of fixedly splitting into two sub-clusters can prevent codebook explosion caused by excessive codebook fragmentation, while simultaneously capturing the main differentiations in the early stages of population splitting.

[0228] In practical applications, the distribution detection of sample user representation vectors within the clusters corresponding to each cluster center can be achieved through various methods.

[0229] One alternative approach is to perform distribution detection based on kernel density estimation. Specifically, the distance distribution between the user representation vector of samples within each cluster and the center vector of their cluster centers can be estimated using kernel density estimation. The number of peaks on the density curve is used to determine if the distribution exhibits multiple peaks. If the density curve has multiple local maxima, the distribution of samples within a cluster is considered to have multiple peaks. During density estimation, the bandwidth parameter can be adaptively selected to ensure the density curve accurately reflects the peak structure of the distribution. A peak significance threshold can be set during peak detection to filter out false peaks with excessively small amplitudes. Distribution detection using kernel density estimation identifies the multi-peak structure of the distribution based on nonparametric density estimation without prior assumptions about the distribution morphology.

[0230] Another alternative approach is to perform distribution detection based on distance histograms. Specifically, multi-peak detection can be performed using a histogram of the distances between the user representation vectors of samples within a cluster and the center vectors of the cluster centers. By statistically analyzing the sample quantity distribution across each distance interval, multiple peak intervals in the histogram are identified. When there are two or more peak intervals, the distribution is determined to have multiple peaks. During the histogram construction process, the width of the distance intervals can be adaptively determined to improve the resolution of peak identification while ensuring statistical stability. Performing distribution detection using distance histograms is simple to implement and provides intuitive detection results.

[0231] Another alternative approach is to perform distribution detection based on clustering effectiveness metrics. Specifically, clustering effectiveness metrics can be used to determine the existence of substructures within clusters. Candidate sub-clustering is performed on samples within each cluster, and clustering effectiveness metrics such as silhouette coefficients are calculated. If the sub-clustering effectiveness metrics are better than the unsplit state, the cluster's sample distribution exhibits multiple peaks. During the clustering effectiveness evaluation process, effectiveness metrics can be calculated for each of the multiple candidate sub-clusters, and the number of candidate sub-clusters with the best effectiveness metrics is selected as the judgment criterion. Performing distribution detection using clustering effectiveness metrics can quantify the degree of structural improvement brought about by splitting.

[0232] In addition, a bimodal fit test can be performed on the distance distribution of samples within a cluster, fitting the distance distribution to a combination of two Gaussian distributions, and determining that the distribution exhibits multiple peaks when the goodness of fit meets the preset conditions.

[0233] If the distribution of sample user representation vectors within a cluster corresponding to any cluster center shows multiple peaks, the sample user representation vectors within the cluster can be divided into a preset number of sub-clusters.

[0234] One alternative approach is to divide the sub-clusters based on a fixed number of sub-clustering methods. Specifically, a fixed-number sub-clustering operation can be performed on all sample user representation vectors within a cluster exhibiting multiple peaks, dividing them into two sub-clusters. The sample user representation vectors within the cluster are then divided into two sub-clusters with these two sub-clusters as the clustering targets. During this fixed-number division process, constraints can be applied to the size ratio of the two sub-clusters to avoid sub-clusters with severely imbalanced sample sizes after the division. Dividing the sub-clusters using a fixed number of sub-clustering methods is simple and controllable, making it suitable for scenarios where the initial stage of population splitting exhibits a bimodal structure.

[0235] Another alternative approach is to divide sub-clusters based on peak locations. Specifically, samples can be divided based on the locations of distribution density peaks, with each peak location serving as the initial center of the sub-cluster. The user representation vectors of samples within a cluster are then assigned to their corresponding sub-clusters based on the distance between the sample and each peak center. Optionally, after allocation, sub-cluster boundary samples can be reassigned, determining their affiliation according to preset rules when the boundary sample is close to two peak centers. Dividing sub-clusters by peak location fully utilizes the peak location information obtained from distribution detection, and the sub-cluster division directly corresponds to the distribution structure.

[0236] Another alternative approach is to divide sub-clusters based on the nearest distance allocation method. Specifically, samples can be allocated based on the distance between the sample and the center of each peak. After determining the peak position, the user representation vector of each sample within a cluster is assigned to the sub-cluster corresponding to the nearest peak, and the boundary samples of the sub-clusters are fine-tuned after allocation. During the distance allocation process, consistency checks can be performed on the allocation results to confirm that each sample belongs to only one sub-cluster. Dividing sub-clusters using the nearest distance allocation method, under the condition that the peak position is known, completes the division based on the nearest distance principle, achieving simplicity.

[0237] In addition, after the sub-clusters are divided, a quality check can be performed on the sub-clusters. If the sample size of the sub-clusters is too small or the overlap between sub-clusters is too high, the splitting method can be adjusted or the sub-clusters can be merged to ensure the quality of the splitting results.

[0238] Once the sub-clusters are divided, the center vector of each sub-cluster can be determined as the new cluster center.

[0239] One alternative approach is to determine the center vectors of sub-clusters based on the mean vector method. Specifically, the mean vector of the user representation vectors of samples within each sub-cluster can be calculated. This mean vector is then used as the center vector of the corresponding sub-cluster, and the center vectors of each sub-cluster are added to the clustering codebook as new cluster centers. The status flag of each new cluster center is set to active. During the mean calculation process, outlier samples within the sub-clusters can be removed, and the mean vector is recalculated based on the remaining samples after removal. Determining the center vectors of sub-clusters using the mean vector method is computationally simple and exhibits good statistical properties.

[0240] Another alternative approach is to determine the center vector of a sub-cluster based on the minimum distance sum. Specifically, the user representation vector of the sample within the sub-cluster that has the smallest sum of distances to all samples can be selected as the center vector of the sub-cluster, and this corresponding center vector is determined as the new cluster center. During the selection process, the distance sum calculation can be performed in parallel to accelerate the process, and sample vectors located in the central region of the sub-cluster can be selected when the distance sums are close. Determining the center vector of the sub-cluster using the minimum distance sum method ensures that the selected center vector comes from real samples and has good robustness to outlier samples.

[0241] Furthermore, the center vector of a sub-cluster can be determined based on a weighted average method. Specifically, weights can be assigned to the user representation vectors of each sample based on the quality index of the samples within the sub-cluster, and the weighted average vector can be calculated as the center vector of the sub-cluster, so that higher-quality samples have a greater weight in the determination of the center vector.

[0242] In the embodiments of this specification, by performing distribution detection on the sample user representation vectors within the clusters corresponding to the cluster centers, clusters that have undergone population differentiation are identified, and split populations are automatically discovered; by dividing the sample user representation vectors within the clusters into a preset number of sub-clusters when multiple peaks are detected in the distribution, the split sub-groups obtain independent coding units; by determining the center vector of each sub-cluster as the new cluster center, accurate coding representations are provided for the differentiated populations, while a conservative splitting strategy of fixed splitting into two sub-clusters is adopted to avoid excessive fragmentation of the codebook, so that the detection of split populations and the detection of new populations form a complete coverage detection mechanism that is serially connected.

[0243] In one optional embodiment of this specification, each cluster center in the current clustering codebook occupies a corresponding coding slot, and the status identifier of each cluster center is the status identifier of the corresponding coding slot. The coding slot includes a reserved coding slot with a status identifier of reserved status and a failed coding slot with a status identifier of failed status. The newly added cluster centers are merged with multiple existing cluster centers in the current cluster codebook, and the status flags corresponding to each cluster center are updated to obtain the target cluster codebook, including: The newly added cluster centers are assigned to the invalid coding slots, and the status flag of the invalid coding slots is updated to active. If there are not enough invalid coding slots, the newly added cluster centers will be assigned to the reserved coding slots, and the status flag of the reserved coding slots will be updated to active.

[0244] A coding slot is a capacity unit in a clustering codebook that accommodates cluster centers. Each cluster center occupies a corresponding coding slot. Specifically, a coding slot can include a fixed number of position units divided according to clustering levels in the clustering codebook. Each coding slot can accommodate one cluster center. The position of a cluster center in the clustering codebook is determined by the coding slot. The status identifier of a cluster center is the status identifier of the corresponding coding slot. The status identifier of a coding slot includes active, inactive, and reserved states. The coding slot can serve as the basic unit for capacity and status management of the clustering codebook. By managing the status of the coding slots, the addition, inactivation, and reservation of cluster centers can be achieved. Furthermore, there is a one-to-one correspondence between coding slots and cluster centers. The code of a cluster center in the codebook is the sequence number of the corresponding coding slot. Through the status management of coding slots, the clustering codebook can dynamically expand while maintaining a controllable total number of codes, making the codebook capacity usage fully auditable.

[0245] Reserved coding slots are coding slots marked as reserved, and are blank slots pre-allocated for future expansion of the clustering codebook. Specifically, reserved coding slots can include coding slots reserved for subsequent expansion during the construction of the clustering codebook. Reserved coding slots are not currently assigned cluster centers and are activated when there are insufficient invalid coding slots. New cluster centers can be assigned to reserved coding slots. Reserved coding slots can serve as a carrier for the limited expansion space of the clustering codebook. Furthermore, the number of reserved coding slots can be determined based on whether there is a high probability of uncovered cluster centers in the business scenario. Reserved coding slots are only activated when invalid coding slots are exhausted to avoid disorderly expansion of the codebook size.

[0246] Failed coding slots are coding slots marked as failed, vacated by cluster centers that are substandard or have been split and replaced. Specifically, failed coding slots can include those reserved after cluster centers are marked as failed. These slots are not currently used for coding production but are reserved for reuse by new cluster centers, which are preferentially assigned to them. Failed coding slots can serve as a carrier for reusing cluster codebook capacity. Furthermore, the reuse mechanism of failed coding slots prevents unlimited codebook growth, makes codebook capacity usage auditable, and allows operators to monitor slot usage at each cluster level at any time.

[0247] The reserved state is a state type in the state identifier, indicating that the corresponding coding slot is reserved for future expansion but has not yet been assigned a cluster center. Specifically, a coding slot with the reserved state identifier is a reserved coding slot. Reserved coding slots are activated when there are insufficient invalid coding slots. After a new cluster center is assigned to a reserved coding slot, the status identifier of the reserved coding slot is updated to the active state. The reserved state can serve as an extended reserve state in the three-state state management mechanism. Furthermore, the reserved state, along with the active state and the invalid state, constitute the three-state management mechanism of the clustering codebook. The active state ensures normal coding service, the invalid state enables the exit of low-quality coding and slot reuse, and the reserved state provides controllable expansion space.

[0248] In practical applications, assigning new cluster centers to failed coding slots and updating the status flag of failed coding slots to active status can be achieved through various methods.

[0249] One alternative approach is to allocate new cluster centers using a slot-by-slot allocation method. Specifically, when merging new cluster centers, the coding slots currently marked as invalid in the cluster codebook are traversed, and the new cluster centers are sequentially written into these invalid coding slots. After writing, the status of the corresponding invalid coding slots is updated to active, ensuring the new cluster centers are included in the coding system in an active state. During the slot-by-slot allocation process, the correspondence between each new cluster center and an invalid coding slot can be recorded, forming a detailed slot allocation record. After allocation, a consistency check is performed on the allocation results to confirm that each coding slot accommodates only one cluster center. By allocating new cluster centers using a slot-by-slot method, the allocation process and status update are executed synchronously.

[0250] Another alternative approach is to allocate new cluster centers using a batch allocation method. Specifically, a list of invalid coding slots in the current clustering codebook can be pre-compiled. After establishing a one-to-one correspondence between new cluster centers and invalid coding slots, the writing of new cluster centers and the updating of their status identifiers can be performed in batches. Before batch allocation, the availability of invalid coding slots can be verified to exclude invalid coding slots with abnormal status identifiers or those already occupied. After batch allocation, the update results of the status identifiers can be reviewed. Allocating new cluster centers in batches reduces the overhead of slot-by-slot operations and is suitable for scenarios with a large number of new cluster centers.

[0251] Another alternative method is to allocate new cluster centers based on a hierarchical matching allocation method. Specifically, new cluster centers can be preferentially allocated to the same failure coding slots as the cluster level to which they belong, ensuring that each cluster center occupies a failure slot within its current cluster level and maintaining the stability of the slot structure across all cluster levels. During the hierarchical matching allocation process, failure coding slots within the same cluster level can be allocated sequentially according to their slot numbers, and the slot usage at each cluster level can be recorded. Allocating new cluster centers through a hierarchical matching allocation method takes into account the consistency of cluster levels during slot allocation, which is beneficial for the long-term stability of the codebook structure.

[0252] In addition, after completing the allocation and status update of the failed coding slots, a slot change record can be output, recording the correspondence between the newly added cluster centers and the failed coding slots, as well as the update results of the status identifiers, for use in codebook change auditing.

[0253] If there are not enough invalid coding slots, the newly added cluster centers can be assigned to the reserved coding slots, and the status flag of the reserved coding slots can be updated to active status.

[0254] One alternative method is to allocate reserved coding slots based on a quantity difference-based allocation method. Specifically, when the number of failed coding slots is less than the number of new cluster centers, the quantity difference between the failed and new cluster centers is calculated. The new cluster centers are first allocated to all failed coding slots, and then the remaining new cluster centers are allocated to reserved coding slots. After allocation, the status flag of the corresponding reserved coding slot is updated to active. During the quantity difference-based allocation process, the remaining capacity of the reserved coding slots can be checked, and a capacity warning message can be output if there are insufficient reserved coding slots. By allocating reserved coding slots using the quantity difference-based allocation method, all new cluster centers are allocated in the order of failure priority and reserved backup.

[0255] Another alternative approach is to allocate reserved coding slots based on a priority-driven allocation method. Specifically, during the merging process, the number of available invalid coding slots is checked first, followed by the number of available reserved coding slots. New cluster centers are allocated and their status flags are updated sequentially, prioritizing invalid slots over reserved slots. During the priority-driven allocation process, an allocation order can be set for each new cluster center, allocating coding slots in descending order of the sample size covered by each new cluster center. This priority-driven allocation method ensures consistency in the order in which reserved coding slots are used.

[0256] In addition, when both the expired coding slots and the reserved coding slots are insufficient to accommodate all the newly added cluster centers, a codebook capacity warning can be triggered, the allocation of the remaining newly added cluster centers can be suspended, and capacity expansion suggestions can be output for operators to make decisions.

[0257] In the embodiments of this specification, by assigning new cluster centers to invalid coding slots and updating the status flag of invalid coding slots to the active state, slots vacated by low-quality coding that does not meet the quality standards are reused, avoiding disorderly growth in the codebook size. By assigning new cluster centers to reserved coding slots when there are insufficient invalid coding slots and updating the status flag of reserved coding slots to the active state, a limited but controllable capacity space is provided for codebook expansion. Through the three-state coding slot management mechanism of active, invalid, and reserved, the unification of the existing coding being completely static and the codebook being dynamically expandable is achieved, solving the fundamental contradiction between incremental training and coding stability. At the same time, the codebook capacity usage is fully auditable, ensuring the long-term maintainability of the clustering codebook.

[0258] In one optional embodiment of this specification, a scheduling framework for a clustering codebook update method is provided. Specifically, see [link to relevant documentation]. Figure 2 , Figure 2 This diagram illustrates a scheduling framework for a clustering codebook update method according to an embodiment of this specification, as shown below. Figure 2 As shown.

[0259] The clustering codebook update process is divided into a cold start phase and a continuous operation phase. The cold start phase includes cold start training, quality assessment, codebook fine-tuning, and manual training verification. The continuous operation phase includes production deployment, incremental training, and manual quality assessment.

[0260] The cold start phase is executed once, while the continuous operation phase is executed cyclically according to a preset update cycle. During the cold start phase, cold start training is performed based on historical user representation vectors, and an initial clustering codebook is obtained by clustering layer by layer according to the clustering hierarchy. A quality assessment is performed on each cluster center in the initial clustering codebook to determine the quality status of each cluster center. Based on the quality assessment results, the clustering codebook is fine-tuned, and the status labels of each cluster center are updated. Manual training verification is then performed, i.e., simulated incremental training is performed on historical samples based on historical user representation vectors to verify the coverage of the clustering codebook for all historical samples. If the coverage meets the preset verification conditions, the clustering codebook is deemed to have passed verification and is deployed to production.

[0261] During the continuous operation phase, the production cluster codebook provides encoding services and performs incremental training according to a preset update cycle. During incremental training, new population detection and split population detection are performed. Based on the detection results, new cluster centers are determined and the cluster codebook is updated. During continuous operation, manual quality assessment is inserted in a timely manner to review the quality status of each cluster center and update the status of cluster centers that fail the quality assessment to the invalid status. The cluster codebook updated after incremental training re-enters the manual training and verification phase, forming a complete scheduling closed loop with alternating scheduling of production encoding and incremental training, and timely insertion of manual quality assessment.

[0262] In the embodiments of this specification, the clustering codebook update process is divided into a cold start phase and a continuous operation phase. This ensures that the codebook undergoes a rigorous admission process during its initial construction, including cold start training, quality assessment, codebook fine-tuning, and manual training verification. This process fully verifies the codebook quality before production deployment, identifies potential problems early, and prevents defective codebooks from entering the production environment. By performing manual training verification based on historical samples, the codebook completes a comprehensive rehearsal of all historical samples before official deployment, ensuring that the deployed codebook possesses complete encoding service capabilities. Furthermore, by inserting timely manual quality assessments and a feedback loop combined with incremental training, the codebook quality is periodically reviewed and failure markers are applied, forming multiple quality assurances. This achieves automated maintenance of the clustering codebook throughout its entire lifecycle, from cold start construction to continuous updates, guaranteeing the encoding quality and semantic stability of the codebook during long-term operation.

[0263] In one optional embodiment of this specification, an embodiment of a semantic identifier generation method is provided. Specifically, see [link to embodiment]. Figure 3 , Figure 3 A flowchart of a semantic identifier generation method according to an embodiment of this specification is shown, such as... Figure 3 As shown, the semantic identifier generation method specifically includes the following steps.

[0264] Step 302: Obtain the target user representation vector; Step 304: Based on the target clustering codebook, encode and map the target user representation vector to obtain the target semantic identifier, wherein the target clustering codebook is updated according to the above clustering codebook update method.

[0265] The target user representation vector is a continuous vector obtained by numerically representing the behavioral characteristics of the user for whom semantic identifiers need to be generated. Specifically, the target user representation vector can include a fixed-dimensional dense vector output by deep neural network encoding the multidimensional behavioral data of the target user within a preset time window. The target user can be any user on the platform for whom semantic identifiers need to be generated. The target user representation vector can be used to determine the cluster center to which the target user belongs in each cluster level by calculating the distance between it and the cluster centers of each cluster level in the clustering codebook. The target user representation vector can be used as input data for semantic identifier generation. Furthermore, the target user representation vector, the current user representation vector, and the historical user representation vector during the clustering codebook update process all belong to the user representation vector system. The difference is that the target user representation vector is oriented towards the application scenario of semantic identifier generation and can be a user representation vector obtained at any time.

[0266] The target clustering codebook is a clustering codebook used to encode and map target user representation vectors. It is the clustering codebook updated according to the aforementioned clustering codebook update method. Specifically, the target clustering codebook can include the target clustering codebook obtained after incrementally updating the current clustering codebook according to the clustering codebook update method, or it can include a clustering codebook continuously maintained through multiple incremental updates. The target clustering codebook includes multiple cluster centers divided according to clustering levels and their corresponding state identifiers. The target clustering codebook can serve as the encoding benchmark for semantic identifier generation, and the target user representation vector is encoded and mapped based on the target clustering codebook. Furthermore, the target clustering codebook continuously expands its coverage capability while maintaining the stability of existing encoded semantics through the clustering codebook update method, so that the semantic identifiers generated based on the target clustering codebook maintain the continuity of historical semantics while covering newly emerging user groups.

[0267] Encoding mapping is the process of mapping continuous target user representation vectors to discrete semantic identifiers. Specifically, encoding mapping may include, based on the target clustering codebook, calculating the distance between the target user representation vector and the center vectors of each cluster center in each clustering level, determining the nearest cluster center in each clustering level, and combining the cluster center numbers determined at each clustering level in hierarchical order to obtain the discrete semantic identifier corresponding to the target user representation vector. Encoding mapping can serve as the core processing step in semantic identifier generation. Furthermore, the positions of existing cluster centers remain stable during the encoding mapping process, thus ensuring that the semantic identifiers generated for the same user at different time points remain consistent, changing only when user interests undergo substantial changes or new cluster centers are added to the codebook, thereby guaranteeing the stability of the semantic identifiers.

[0268] The target semantic identifier is a discrete identifier obtained by encoding and mapping the target user representation vector based on the target clustering codebook. Specifically, the target semantic identifier can include a multi-layered discrete identifier formed by combining cluster center numbers from multiple clustering levels in hierarchical order. Each target semantic identifier corresponds to a user cluster with clear business semantics in the user representation space and can serve as a user representation index for downstream recommendation tasks. The target semantic identifier can be the final output of the semantic identifier generation method. Furthermore, the target semantic identifier is directly related to the coverage capability of the target clustering codebook. After the target clustering codebook expands its coverage through incremental updates, user groups that previously could not obtain precise encoding can obtain target semantic identifiers that match their own semantics, thereby improving the accuracy and coverage of semantic identifier generation.

[0269] In practical applications, the target user representation vector can be obtained through various methods.

[0270] One alternative approach is to obtain the target user representation vector through online encoding. Specifically, behavioral data of the target user within a preset time window can be read from a user feature storage system. This behavioral data is then encoded into a target user representation vector using a pre-deployed deep neural network encoding model. The encoded target user representation vector is then used as input for semantic identifier generation. During the encoding process, missing value imputation and outlier filtering can be performed on the behavioral data. Encoding is performed after data preprocessing, and the numerical range of the target user representation vector is validated after encoding. Obtaining the target user representation vector through online encoding and generating the representation vector based on a unified encoding model results in good consistency of representation results.

[0271] Another alternative approach is to obtain the target user representation vector through real-time computation. Specifically, a real-time computation engine can calculate the target user representation vector in real-time when the target user initiates a request. This vector is generated based on the target user's latest behavioral data, ensuring that the representation vector used for semantic tagging is synchronized with the user's latest state. During real-time computation, incremental aggregation can be performed on the target user's latest behavioral events, and the results of this aggregation can be fused with historical behavioral features to generate the target user representation vector. Obtaining the target user representation vector through real-time computation is suitable for scenarios with high requirements for data timeliness.

[0272] Furthermore, target user representation vectors can be obtained through offline pre-computation. Specifically, pre-computed target user representation vectors can be read from the offline feature pipeline, and the offline computation results can be directly reused during semantic tag generation, reducing online computation overhead and making it suitable for scenarios involving batch generation of semantic tags.

[0273] In practical applications, the target semantic identifier can be obtained by encoding and mapping the target user representation vector based on the target clustering codebook through various methods.

[0274] One alternative approach is to perform encoding mapping based on a layer-by-layer nearest neighbor matching method. Specifically, the distance between the target user representation vector and the center vectors of each cluster center at each cluster level in the target clustering codebook can be calculated to determine the nearest cluster center at each cluster level. The cluster center numbers at each cluster level are then combined in hierarchical order to obtain the target semantic identifier. During the distance calculation process, the distance values ​​of the nearest cluster centers at each cluster level can be recorded. When the distance value exceeds a preset distance threshold, a reliability flag is applied to the encoding mapping result. By performing encoding mapping through layer-by-layer nearest neighbor matching, the encoding result strictly corresponds to the structure of the target clustering codebook.

[0275] Another alternative approach is to perform encoding mapping based on residual recursive quantization. Specifically, the cluster center to which the target user representation vector belongs in the first clustering level can be determined first. Then, based on the residual between the target user representation vector and the center vector of the corresponding cluster center, the cluster centers to which it belongs in subsequent clustering levels are determined layer by layer. The cluster center numbers of each clustering level are combined to obtain the target semantic identifier. During residual recursive quantization, the residuals at each level can be normalized to match the numerical range of the residual vector with the distribution of cluster centers in subsequent levels. Performing encoding mapping using residual recursive quantization is consistent with the residual quantization construction process of the codebook.

[0276] Another alternative approach is to accelerate encoding mapping using a vector retrieval index. Specifically, a vector retrieval index can be pre-built based on the cluster centers of each cluster level in the target clustering codebook. This index accelerates the distance calculation and nearest neighbor search between the target user representation vector and the cluster centers at each level, quickly determining the nearest cluster center at each level. The vector retrieval index can be structured separately for each cluster level, and its structure can be periodically updated to maintain consistency with the clustering codebook. This accelerated encoding mapping using a vector retrieval index significantly improves efficiency, especially in scenarios with a large number of cluster centers.

[0277] In addition, normalization preprocessing can be performed on the target user representation vector before encoding mapping to correct for abnormal numerical ranges of the target user representation vector, so as to ensure the accuracy of distance calculation and nearest neighbor matching.

[0278] In the embodiments of this specification, the target user representation vector is obtained, providing the input data basis for semantic identifier generation; the target semantic identifier is obtained by encoding and mapping the target user representation vector based on the target clustering codebook, so that the continuous user representation vector is transformed into a discrete identifier with business semantics, providing interpretable semantic representation for the target user; the target clustering codebook is updated according to the above-mentioned clustering codebook update method, so that the semantic identifier generation process reuses the results of the incremental update of the clustering codebook, maintaining the stability of the existing encoded semantics while covering newly emerging user groups, thereby improving the accuracy and efficiency of semantic identifier generation.

[0279] Corresponding to the above method embodiments, this specification also provides embodiments of a clustering codebook update system. Figure 4 A schematic diagram of a clustering codebook update system according to one embodiment of this specification is shown. Figure 4 As shown, the clustering codebook update system includes: The first acquisition module 402 is used to acquire the current user representation vector and the current cluster codebook, wherein the current cluster codebook is constructed based on the historical user representation vector and includes multiple cluster centers divided according to the clustering level and the corresponding status identifiers; Detection module 404 is used to determine the uncovered user representation vector from the current user representation vector based on the distance between the current user representation vector and the center vectors of multiple cluster centers; Training module 406 is used to determine new cluster centers based on the uncovered user representation vectors; The update module 408 is used to merge the newly added cluster centers with the multiple existing cluster centers in the current cluster codebook, and update the status identifiers corresponding to each cluster center to obtain the target cluster codebook.

[0280] The first acquisition module is a module in the clustering codebook update system used to acquire the current user representation vector and the current clustering codebook. Specifically, the first acquisition module may include a data reading unit that interfaces with the user feature storage system and the codebook management service. This unit reads the current user representation vector within the current time window, as well as the currently active clustering codebook data structure. The current clustering codebook is constructed based on historical user representation vectors and includes multiple cluster centers divided according to clustering levels and their corresponding status identifiers. The first acquisition module can serve as the data entry point for the clustering codebook update system. Furthermore, the current user representation vector and the current clustering codebook acquired by the first acquisition module serve as input to the detection module, providing a data foundation for subsequent coverage detection.

[0281] The detection module is a component in the clustering codebook update system used to identify uncovered user representation vectors. Specifically, the detection module may include a distance calculation unit and a coverage determination unit. The distance calculation unit calculates the distance between each user representation vector in the current user representation vector and the center vectors of multiple cluster centers. The coverage determination unit determines uncovered user representation vectors based on statistical features of the distances. The detection module can serve as the execution unit for identifying coverage gaps in the clustering codebook. Furthermore, the output of the detection module serves as the input to the training module, providing the target for determining new cluster centers.

[0282] The training module is used in the clustering codebook update system to determine new cluster centers. Specifically, the training module may include a cluster number determination unit and a clustering training unit. The cluster number determination unit determines the number of clusters based on the distribution characteristics of the uncovered user representation vectors, and the clustering training unit clusters the uncovered user representation vectors based on the cluster number to obtain the center vector of each cluster, and determines the center vector of each cluster as the new cluster center. The training module can also serve as a unit for generating extended content of the clustering codebook. Furthermore, the training module performs clustering training while keeping the center vectors of existing cluster centers unchanged, ensuring the stability of existing encoded semantics.

[0283] The update module is a component of the clustering codebook update system used to merge cluster centers and update their status identifiers. Specifically, the update module may include a merging unit and a status update unit. The merging unit merges newly added cluster centers with existing cluster centers in the current clustering codebook, while the status update unit updates the status identifiers of each cluster center to obtain the target clustering codebook. The update module can serve as an output component of the clustering codebook update system. Furthermore, the update module publishes the target clustering codebook as a new, effective version for use by downstream components such as semantic identifier generation.

[0284] The clustering codebook update system provided in this specification's embodiments acquires the current user representation vector and the current clustering codebook through a first acquisition module, providing complete data input for the codebook update process; a detection module determines the uncovered user representation vector based on the distance between the current user representation vector and the center vectors of multiple cluster centers, achieving quantitative identification of codebook coverage gaps; a training module determines new cluster centers based on the uncovered user representation vectors, making codebook expansion driven by actual coverage gaps; and an update module merges the new cluster centers with existing cluster centers and updates the status identifiers corresponding to each cluster center to obtain the target clustering codebook. This expands the codebook coverage capability while maintaining the stability of existing encoded semantics, achieving automated incremental maintenance of the clustering codebook, improving the codebook's adaptability to user group evolution, and ensuring the encoding quality of user semantic identifiers.

[0285] The above is an illustrative scheme of a clustering codebook update system according to this embodiment. It should be noted that the technical solution of this clustering codebook update system and the technical solution of the clustering codebook update method described above belong to the same concept. For details not described in detail in the technical solution of the clustering codebook update system, please refer to the description of the technical solution of the clustering codebook update method described above.

[0286] Corresponding to the above method embodiments, this specification also provides embodiments of a semantic tag generation system. Figure 5 A schematic diagram of the structure of a semantic identifier generation system provided in one embodiment of this specification is shown. Figure 5 As shown, the semantic identifier generation system includes: The second acquisition module 502 is used to acquire the target user representation vector; The encoding mapping module 504 is used to encode and map the target user representation vector based on the target clustering codebook to obtain the target semantic identifier, wherein the target clustering codebook is updated according to the above-mentioned clustering codebook update method.

[0287] The second acquisition module is a module in the semantic tag generation system used to acquire the target user representation vector. Specifically, the second acquisition module may include a data reading unit that interfaces with a user feature storage system or a real-time computing engine to read the target user's behavioral data within a preset time window and encode it into a target user representation vector, or it may directly read a pre-calculated target user representation vector. The second acquisition module can serve as the data entry point for the semantic tag generation system. Furthermore, the target user representation vector acquired by the second acquisition module serves as the input to the encoding mapping module, providing a data foundation for semantic tag generation.

[0288] The encoding mapping module is a component in the semantic identifier generation system used to encode and map the target user representation vector. Specifically, the encoding mapping module may include a distance calculation unit and an identifier combination unit. The distance calculation unit calculates the distance between the target user representation vector and the center vectors of the cluster centers at each cluster level in the target clustering codebook. The identifier combination unit determines the nearest cluster center at each cluster level and combines the cluster center numbers at each level in hierarchical order to obtain the target semantic identifier. The target clustering codebook is updated according to the aforementioned clustering codebook update method. The encoding mapping module can serve as the core processing unit of the semantic identifier generation system. Furthermore, the encoding mapping module performs encoding mapping based on the updated target clustering codebook, ensuring that the generated semantic identifier maintains the continuity of historical semantics while also covering newly emerging user groups.

[0289] The semantic identifier generation system provided in this specification obtains the target user representation vector through the second acquisition module, providing input data for semantic identifier generation; and obtains the target semantic identifier by encoding and mapping the target user representation vector based on the target cluster codebook through the encoding and mapping module, so that the continuous user representation vector is transformed into a discrete identifier with business semantics. The target cluster codebook is updated according to the above-mentioned cluster codebook update method, so that the semantic identifier generation process reuses the results of the incremental update of the cluster codebook, and covers newly emerging user groups while maintaining the stability of existing encoded semantics, thereby improving the accuracy and efficiency of semantic identifier generation.

[0290] The above is an illustrative scheme of a semantic tag generation system according to this embodiment. It should be noted that the technical solution of this semantic tag generation system and the technical solution of the semantic tag generation method described above belong to the same concept. For details not described in detail in the technical solution of the semantic tag generation system, please refer to the description of the technical solution of the semantic tag generation method described above.

[0291] Figure 6 A structural block diagram of a computing device 600 according to one embodiment of this specification is shown. The components of the computing device 600 include, but are not limited to, a memory 610 and a processor 620. The processor 620 is connected to the memory 610 via a bus 630, and a database 650 is used to store data.

[0292] The computing device 600 also includes an access device 640, which enables the computing device 600 to communicate via one or more networks 660. Examples of these networks include a Public Switched Telephone Network (PSTN), a Local Area Network (LAN), a Wide Area Network (WAN), a Personal Area Network (PAN), or a combination of communication networks such as the Internet. The access device 640 may include one or more of any type of wired or wireless network interface (e.g., a network interface controller (NIC)), such as an IEEE 802.11 Wireless Local Area Network (WLAN) wireless interface, a Worldwide Interoperability for Microwave Access (Wi-MAX) interface, an Ethernet interface, a Universal Serial Bus (USB) interface, a cellular network interface, a Bluetooth interface, or a Near Field Communication (NFC) interface.

[0293] In one embodiment of this specification, the above-described components of the computing device 600 and Figure 6 Other components, not shown, can also be connected to each other, for example, via a bus. It should be understood that... Figure 6 The block diagram of the computing device shown is for illustrative purposes only and is not intended to limit the scope of this specification. Those skilled in the art can add or replace other components as needed.

[0294] The computing device 600 can be any type of stationary or mobile computing device, including mobile computers or mobile computing devices (e.g., tablet computers, personal digital assistants, laptop computers, notebook computers, netbooks, etc.), mobile phones (e.g., smartphones), wearable computing devices (e.g., smartwatches, smart glasses, etc.) or other types of mobile devices, or stationary computing devices such as desktop computers or personal computers (PCs). The computing device 600 can also be a mobile or stationary server.

[0295] The processor 620 is configured to execute the following computer-executable instructions, which, when executed by the processor, implement the steps of the above method.

[0296] The above is an illustrative scheme of a computing device according to this embodiment. It should be noted that the technical solution of this computing device and the technical solution of the above method belong to the same concept, and all details not described in detail in the technical solution of the computing device can be referred to the description of the technical solution of the above method.

[0297] An embodiment of this specification also provides a computer-readable storage medium storing computer-executable instructions that, when executed by a processor, implement the steps of the above-described method.

[0298] The above is an illustrative scheme of a computer-readable storage medium according to this embodiment. It should be noted that the technical solution of this storage medium and the technical solution of the above method belong to the same concept, and all details not described in detail in the technical solution of the storage medium can be referred to the description of the technical solution of the above method.

[0299] An embodiment of this specification also provides a computer program product, including computer-executable instructions that, when executed by a processor, implement the steps of the above-described method.

[0300] The above is an illustrative scheme of a computer program product according to this embodiment. It should be noted that the technical solution of this computer program product and the technical solution of the above method belong to the same concept, and all details not described in detail in the technical solution of the computer program product can be referred to in the description of the technical solution of the above method.

[0301] The foregoing has described specific embodiments of this specification. Other embodiments are within the scope of the appended claims. In some cases, the actions or steps recited in the claims may be performed in a different order than that shown in the embodiments and may still achieve the desired result. Furthermore, the processes depicted in the drawings do not necessarily require the specific or sequential order shown to achieve the desired result. In some embodiments, multitasking and parallel processing are possible or may be advantageous.

[0302] The computer instructions include computer program code, which may be in the form of source code, object code, executable file, or certain intermediate forms. The computer-readable medium may include: any entity or device capable of carrying the computer program code, recording media, USB flash drive, portable hard drive, magnetic disk, optical disk, computer memory, read-only memory (ROM), random access memory (RAM), electrical carrier signals, telecommunication signals, and software distribution media, etc. It should be noted that the content included in the computer-readable medium may be appropriately added or removed according to the requirements of patent practice. For example, in some regions, according to patent practice, computer-readable media may not include electrical carrier signals and telecommunication signals.

[0303] It should be noted that, for the sake of simplicity, the foregoing method embodiments are all described as a series of actions. However, those skilled in the art should understand that the embodiments in this specification are not limited to the described order of actions, because according to the embodiments in this specification, some steps can be performed in other orders or simultaneously. Furthermore, those skilled in the art should also understand that the embodiments described in this specification are all preferred embodiments, and the actions and modules involved are not necessarily essential to the embodiments in this specification.

[0304] In the above embodiments, the descriptions of each embodiment have different focuses. For parts not described in detail in a certain embodiment, please refer to the relevant descriptions in other embodiments.

[0305] The preferred embodiments disclosed above are merely illustrative of this specification. Optional embodiments do not exhaustively describe all details, nor do they limit the invention to the specific implementations described. Clearly, many modifications and variations can be made based on the embodiments described herein. These embodiments are selected and specifically described in this specification to better explain the principles and practical applications of the embodiments, thereby enabling those skilled in the art to better understand and utilize this specification. This specification is limited only by the claims and their full scope and equivalents.

Claims

1. A clustering codebook update method, characterized in that, include: Obtain the current user representation vector and the current cluster codebook, wherein the current cluster codebook is constructed based on the historical user representation vector and includes multiple cluster centers divided according to the clustering level and the corresponding status identifiers; Based on the distance between the current user representation vector and the center vectors of the multiple cluster centers, the uncovered user representation vectors are determined from the current user representation vector; Based on the uncovered user representation vectors, new cluster centers are determined; The newly added cluster center is merged with the existing cluster centers in the current cluster codebook, and the status identifiers corresponding to each cluster center are updated to obtain the target cluster codebook.

2. The method according to claim 1, characterized in that, Before obtaining the current user representation vector and the current clustering codebook, the method further includes: Obtain historical user representation vectors; Based on the historical user representation vector, cluster training is performed layer by layer according to the clustering level to obtain the initial cluster centers; Set the status flag of each initial cluster center to active state to obtain the current cluster codebook.

3. The method according to claim 1, characterized in that, After obtaining the current user representation vector and the current clustering codebook, the method further includes: Based on the historical user representation vector, the quality of each cluster center in the current clustering codebook is evaluated to obtain the quality evaluation index of each cluster center. If the quality assessment index does not meet the preset index threshold, the status identifier of the corresponding cluster center will be updated to the failure status.

4. The method according to claim 3, characterized in that, The quality assessment indicators include at least one of the following: boundary stability indicator, minimum clustering percentage indicator, coefficient of variation indicator, and residual explanation rate indicator; When the quality assessment index does not meet the preset index threshold, updating the status identifier of the corresponding cluster center to a failed state includes: If the number of quality assessment indicators that do not meet the corresponding preset threshold reaches the preset number, the status identifier of the corresponding cluster center will be updated to the invalid state.

5. The method according to claim 3, characterized in that, The quality assessment of each cluster center in the current clustering codebook includes: Identify the cluster centers in each cluster level that are marked as active; According to the hierarchical order of the clustering levels, quality assessment is performed layer by layer for each cluster center in the active state to obtain the quality assessment index of each cluster center.

6. The method according to claim 5, characterized in that, The preset indicator threshold is set based on the clustering level; When the quality assessment index does not meet the preset index threshold, updating the status identifier of the corresponding cluster center to a failed state includes: For any clustering level, if the quality assessment index does not meet the preset index threshold corresponding to that clustering level, the status identifier of the corresponding cluster center will be updated to the failure state.

7. The method according to claim 5, characterized in that, After determining the cluster centers that are identified as active in each clustering level, the process further includes: Determine the number of sample user representation vectors corresponding to each cluster center whose state is marked as active in each cluster level; If the number of sample user representation vectors corresponding to any cluster center is less than the preset clustering threshold, skip that cluster center and perform quality assessment. If the total number of sample user representation vectors corresponding to any cluster level is less than the preset level threshold, the quality assessment of that cluster level is skipped.

8. The method according to any one of claims 1-7, characterized in that, The step of determining uncovered user representation vectors from the current user representation vector based on the distance between the current user representation vector and the center vectors of the multiple cluster centers includes: Calculate the distance between each user representation vector in the current user representation vector and the center vector of the multiple cluster centers; Based on the statistical characteristics of the distance between each current user representation vector and the center vector of the multiple cluster centers, the uncovered user representation vectors are determined from the current user representation vectors.

9. The method according to claim 8, characterized in that, The step of determining uncovered user representation vectors from the current user representation vectors based on statistical features of the distance between each current user representation vector and the center vectors of the multiple cluster centers includes: Based on the position of each cluster level in the current cluster codebook, the preset coverage conditions corresponding to each cluster level are determined; Based on the statistical characteristics of the distance between each current user representation vector and the center vector of the multiple cluster centers, and the comparison results between the preset coverage conditions, it is determined whether there is an uncovered user group. If it is determined that there is an uncovered user group, the uncovered user representation vector is determined from the current user representation vector based on the distance between the current user representation vector and the center vector of the multiple cluster centers.

10. The method according to claim 1, characterized in that, The step of determining new cluster centers based on the uncovered user representation vectors includes: Keeping the center vectors of the multiple existing cluster centers in the current clustering codebook unchanged, the number of clusters is determined based on the distribution characteristics of the uncovered user representation vectors; Based on the number of clusters, the uncovered user representation vectors are clustered to obtain the center vector of each cluster, and the center vector of each cluster is determined as the new cluster center.

11. The method according to claim 1, characterized in that, After determining the new cluster centers based on the uncovered user representation vectors, the method further includes: Distribution detection is performed on the sample user representation vectors within the clusters corresponding to each cluster center; If the distribution of sample user representation vectors within a cluster corresponding to any cluster center shows multiple peaks, the sample user representation vectors within the cluster are divided into a preset number of sub-clusters. The center vector of each sub-cluster is determined as the new cluster center.

12. The method according to claim 1, characterized in that, Each cluster center in the current clustering codebook occupies a corresponding coding slot. The status identifier of each cluster center is the status identifier of the corresponding coding slot. The coding slot includes a reserved coding slot with a status identifier of reserved and a failed coding slot with a status identifier of failed. The step of merging the newly added cluster center with the existing cluster centers in the current clustering codebook, and updating the status identifiers corresponding to each cluster center to obtain the target clustering codebook includes: The newly added cluster centers are assigned to the failed coding slots, and the status identifier of the failed coding slots is updated to active. If there are not enough invalid coding slots, the newly added cluster centers will be assigned to the reserved coding slots, and the status flag of the reserved coding slots will be updated to active.

13. A semantic identifier generation method, characterized in that, include: Obtain the target user representation vector; Based on the target clustering codebook, the target user representation vector is encoded and mapped to obtain the target semantic identifier, wherein the target clustering codebook... The clustering codebook is updated according to any one of claims 1-12.

14. A clustering codebook update system, characterized in that, include: The first acquisition module is used to acquire the current user representation vector and the current cluster codebook, wherein the current cluster codebook is constructed based on the historical user representation vector and includes multiple cluster centers divided according to the clustering level and corresponding status identifiers; The detection module is used to determine the uncovered user representation vectors from the current user representation vector based on the distance between the current user representation vector and the center vectors of the multiple cluster centers; The training module is used to determine new cluster centers based on the uncovered user representation vectors; The update module is used to merge the newly added cluster center with the multiple existing cluster centers in the current cluster codebook, and update the status identifiers corresponding to each cluster center to obtain the target cluster codebook.

15. A semantic identifier generation system, characterized in that, include: The second acquisition module is used to acquire the target user representation vector; The encoding mapping module is used to encode and map the target user representation vector based on the target clustering codebook to obtain the target semantic identifier, wherein the target clustering codebook... The clustering codebook is updated according to any one of claims 1-12.

16. A computing device, characterized in that, include: Memory and processor; The memory is used to store computer-executable instructions, and the processor is used to execute the computer-executable instructions, which, when executed by the processor, implement the steps of the method according to any one of claims 1 to 13.

17. A computer-readable storage medium, characterized in that, It stores computer-executable instructions that, when executed by a processor, implement the steps of the method according to any one of claims 1 to 13.

18. A computer program product, characterized in that, It includes computer-executable instructions that, when executed by a processor, implement the steps of the method according to any one of claims 1 to 13.

Images