A method, system and storage medium for dynamic filtering and distribution of data

Abstract

The application relates to a data dynamic filtering and distribution method, system and storage medium. The application performs multi-stage preprocessing filtering on an initial data set by acquiring the initial data set; routes data items filtered through the filtering to a first channel and a second channel respectively, and performs differentiated resource allocation on the first channel and the second channel; establishes a finite state machine for each data item respectively, acquires interaction data and conversion data generated by each data item in a historical data window, and compensates the conversion data in time; based on the interaction data and the compensated conversion data, calculates upper limit confidence and lower limit confidence of conversion efficiency of each data item in the first channel and the second channel, and according to the upper limit confidence and the lower limit confidence, performs dynamic flow transfer of the data items between the two channels, and skips flow transfer calculation for the data items entering the current channel without reaching a preset protection period. Resource preemption is avoided, manual intervention is reduced, resource utilization efficiency and system operation stability are improved.

Description

Technical Field

[0001] This application relates primarily to the field of computers, and in particular to a method, system, and storage medium for dynamic data filtering and distribution. Background Technology

[0002] In computer data processing and network resource dynamic scheduling systems, the filtering of massive amounts of data and the allocation of system resources such as resource quotas are usually accomplished by static rule formulas or manually set fixed thresholds. Before the system runs, the relevant calculation rules and thresholds are pre-written into the code or configuration file; during operation, the system filters the data according to fixed rules and allocates resources based on the calculation results of the formulas. The rules and thresholds are not automatically adjusted based on real-time data feedback.

[0003] However, this existing technology still has the following shortcomings: fixed formulas and static thresholds rely on pre-set historical experience, making it difficult to adaptively adjust to changes in data feature distribution and competitive environment, resulting in weak model generalization ability. Furthermore, single-dimensional static filtering primarily relies on superficial conditions such as traffic characteristics and thresholds for judgment, making it difficult to identify data containing invalid intent, non-target categories, or data lacking conversion value. This causes the system's computing power and resources to be continuously consumed on data nodes without conversion value, resulting in significant ineffective resource waste.

[0004] Furthermore, existing technologies for data filtering and resource allocation often employ a single-channel fixed allocation scheme. All filtered data is placed in the same processing channel or queue, and resources are allocated uniformly according to a single dimension. The system does not distinguish between high-deterministic, high-quality data and new data to be tested, and all types of data share the same resource pool.

[0005] However, in a single-channel architecture, the utilization of existing high-value data and the exploration of new data are intertwined, creating competition between the two within the same resource pool. The system struggles to simultaneously guarantee the supply of resources for highly deterministic data and low-cost trial-and-error with new data: if resources are opened to new data, it may preempt the resource quotas, allocation thresholds, and distribution channels required for core data; conversely, restricting new data resources will suppress the verification and discovery of potentially high-quality data. This mixed allocation easily lowers overall system efficiency, and the directionality and stability of resource allocation are insufficient.

[0006] Furthermore, existing technologies typically employ static databases or static hierarchical management mechanisms for data flow. Once data is placed in a processing queue or resource repository at a certain level, it often remains at that level for an extended period. Unless maintenance personnel review feedback reports and manually transfer the data, the system will not automatically upgrade or downgrade the data based on real-time conversion performance, lacking an automated, full lifecycle management loop.

[0007] However, once the data hierarchy is determined, it becomes relatively fixed. High-performing exploratory data cannot automatically acquire more resources, and the value of high-potential new data is difficult to discover in a timely manner. Conversely, core data with declining performance cannot be downgraded in a timely manner. The system continuously consumes computing resources and distribution channels on poorly performing data nodes, leading to a decline in overall throughput efficiency and resource utilization. Furthermore, hierarchy adjustments rely on manual report verification and manual operation, resulting in long adjustment cycles, high labor costs, and difficulty in adapting to the latency and volatility of transformed data, easily leading to response delays or judgment errors. Summary of the Invention

[0008] One objective of this application is to provide a method, system, and storage medium for dynamic data filtering and distribution, addressing the shortcomings of existing technologies such as static rules and fixed thresholds failing to adapt to environmental changes, single-channel resource allocation leading to resource contention between high-value data and exploratory data, and data hierarchy being fixed and lacking automated full lifecycle flow.

[0009] According to one aspect of this application, a method for dynamic data filtering and distribution is provided, the method comprising: Obtain an initial data set and perform multi-level preprocessing filtering on the data set; The filtered data items are routed to the first channel and the second channel respectively. The first channel and the second channel are independent parallel data processing queues, and differentiated resource allocation is performed on the first channel and the second channel based on the resource scheduling model. A finite state machine is established for each data item, and the interaction data and transformation data generated by each data item within the historical data window are obtained at a preset period. Time alignment compensation is performed on the transformation data. Based on the interactive data and the compensated conversion data, the upper and lower confidence levels of the conversion efficiency of each data item in the first and second channels are calculated. According to the upper and lower confidence levels, the dynamic flow of data items between the two channels is executed, and the flow calculation is skipped for data items that have entered the first or second channel before the preset protection period has expired.

[0010] Optionally, performing multi-level preprocessing filtering on the data set includes: Based on a preset semantic blacklist rule base, the text content of each data item in the dataset is matched, and data items containing global negative word roots are removed. The classification labels of the remaining data items after the removal are analyzed, and the data items whose classification labels do not match the target classification are blocked to obtain the blocked data items.

[0011] Optionally, the multi-stage preprocessing filter further includes: Extract the entity tags from the blocked data items. If the brand corresponding to the entity tag does not belong to the preset core brand set, then isolate the data item to an independent comparison database and do not participate in the data processing flow of the first channel and the second channel. The data items isolated to the comparison database are blacklisted, preventing them from receiving resource allocation and display opportunities. In related display scenarios, the triggering mechanism for the data items in the comparison database is disabled.

[0012] Optionally, the multi-stage preprocessing filter further includes: Extract the traffic characteristics and competition environment characteristics of the blocked data items, and remove data items whose traffic characteristics are less than a preset first threshold to obtain the first filtered data items; Based on the traffic characteristics and competitive environment characteristics of the first screened data items, a comprehensive evaluation factor is calculated, and data items that do not meet the preset conditions are filtered out to obtain the second screened data items; Extract the historical highest consumption boundary and historical lowest consumption boundary of each data item in the second filtering data item. If the historical highest consumption boundary is missing, the historical lowest consumption boundary is missing, or the historical lowest consumption boundary is greater than the industry benchmark value, filter the corresponding data item in the second filtering data item.

[0013] Optionally, the differentiated resource allocation between the first channel and the second channel based on the resource scheduling model includes: The system resource totals with different weights are configured for the first channel and the second channel, wherein the resource total weight of the first channel is greater than the resource total weight of the second channel; A resource-cap-biased allocation strategy is adopted for data items in the first channel, and a resource-cap-biased allocation strategy is adopted for data items in the second channel.

[0014] Optionally, the method further includes: The allocation strategy for the upper limit of the biased resources and the allocation strategy for the lower limit of the biased resources are determined based on the nonlinear evaluation model, respectively. The nonlinear evaluation model takes the historical highest consumption boundary, historical lowest consumption boundary and industry benchmark value of the data item as input, and outputs the corresponding single resource allocation threshold according to the channel to which the data item belongs. Specifically, the single resource allocation threshold of the first channel is biased towards the side of the historical highest consumption boundary, while the single resource allocation threshold of the second channel is biased towards the side of the historical lowest consumption boundary.

[0015] Optionally, routing the filtered data items to the first channel and the second channel respectively includes: The top N data items are selected based on the ranking of traffic characteristics and routed to the first channel, where N is a preset positive integer, and the remaining data items are routed to the second channel; The first channel and the second channel correspond to two independent resource allocation units. Differentiated allocation of system resources is achieved by adjusting the resource quota of each resource allocation unit. The resource allocation unit corresponding to the first channel has a single resource allocation threshold and an overall resource quota that are greater than those of the second channel, and contains fewer data items than the resource allocation unit corresponding to the second channel.

[0016] Optionally, the total amount of system resources configured with different weights includes: Based on the comprehensive evaluation result of the conversion efficiency of the data items in the first channel, the resource quota ratio of the first channel to the total system resources is adjusted so that the resource quota ratio is positively correlated with the comprehensive evaluation result of the conversion efficiency of the first channel, and a lower limit constraint value and an upper limit constraint value of the resource quota ratio are set.

[0017] Optionally, the finite state machine includes an exploratory state, a core state, and a negative elimination state; The dynamic flow includes the promotion flow of data items from the exploration state to the core state, the demotion flow of data items from the core state back to the exploration state, and the elimination flow of data items from the exploration state to the negative elimination state.

[0018] Optionally, the promotion process includes: For a data item in the exploration state, obtain the interaction feature quantity of the data item; When the interaction feature quantity reaches a preset activation threshold, the confidence level of the upper limit of the conversion efficiency of the data item is greater than the statistical benchmark value of the first channel, and the confidence level of the lower limit of the conversion efficiency of the data item is greater than the preset fluctuation range of the statistical benchmark value of the first channel, the data item is migrated from the second channel to the first channel and the state of the data item is updated to the core state. In each circulation cycle, the number of data items migrated to the first channel does not exceed the preset maximum circulation limit. When the number of data items that meet the promotion conditions exceeds the maximum circulation limit, the number of data items corresponding to the maximum circulation limit is selected according to the comprehensive evaluation factor and promoted.

[0019] Optionally, the degradation process includes: For a data item in the core state, when the confidence level of the upper limit of the conversion efficiency of the data item is lower than the preset absolute safety lower limit and lower than the statistical benchmark value of the first channel, the data item is migrated from the first channel to the second channel and the state of the data item is updated to the exploration state.

[0020] Optionally, the obsolescence process includes: For a data item in the exploration state, when the confidence level of the upper limit of the conversion efficiency of the data item is lower than the preset elimination benchmark and lower than the preset fluctuation range of the statistical benchmark value of the second channel, the state of the data item is updated to the negative elimination state, and resources are stopped being allocated to the data item. Add the search terms corresponding to the data items in the negative elimination state to the search term blacklist, and set the search term blacklist through the advertising platform interface so that the search terms no longer receive exposure.

[0021] Optionally, the time alignment compensation for the transformed data includes: Based on the historical conversion delay distribution, a conversion delay distribution function is fitted, and the future conversion volume of data items that have not yet generated complete conversion data within the delayed conversion window is estimated based on the conversion delay distribution function. The estimated conversion rate is combined with the already returned conversion rate to obtain the compensated conversion data.

[0022] Optionally, the step of skipping the flow calculation for data items entering the first or second channel before the preset protection period has expired includes: Record the time when each data item enters the current channel, and determine the duration of each data item entering the current channel based on the time. For data items whose duration is less than the preset protection period, maintain the current state and skip the flow calculation; for data items whose duration is not less than the preset protection period, perform the flow calculation.

[0023] Optionally, before calculating the upper and lower confidence levels of the conversion efficiency, the method further includes: Identify and remove abnormal interaction data generated by web crawlers or cheating bots, and calculate the upper and lower confidence levels of the conversion efficiency based on the removed interaction data; Wherein, the upper confidence level is taken as the upper bound of the conversion efficiency prediction interval, and the lower confidence level is taken as the lower bound of the conversion efficiency prediction interval.

[0024] According to another aspect of this application, a system for dynamic data filtering and distribution is also provided, the system comprising: A multi-level preprocessing and filtering module is used to obtain an initial data set and perform multi-level preprocessing and filtering on the data set; The dual-channel scheduling module is used to route filtered data items to the first channel and the second channel respectively. The first channel and the second channel are independent parallel data processing queues, and differentiated resource allocation is performed on the first channel and the second channel based on the resource scheduling model. The state machine transition module is used to establish a finite state machine for each data item, acquire the interaction data and transformation data generated by each data item within the historical data window according to a preset period, and perform time alignment compensation on the transformation data; calculate the upper and lower confidence levels of the transformation efficiency of each data item in the first and second channels based on the interaction data and the compensated transformation data, and execute the dynamic transition of the data item between the two channels according to the upper and lower confidence levels, and skip the transition calculation for data items that have entered the first or second channel before the preset protection period has expired.

[0025] According to another aspect of this application, an electronic device is also provided, the electronic device comprising: One or more processors; and A memory storing computer-readable instructions, which, when executed, cause the processor to perform operations as described above.

[0026] According to another aspect of this application, a computer-readable medium is also provided, having stored thereon computer instructions that can be executed by a processor to implement the methods described above.

[0027] Compared with existing technologies, this application obtains an initial data set and performs multi-level preprocessing filtering on the data set; routes the filtered data items to a first channel and a second channel respectively, the first channel and the second channel being independent parallel data processing queues, and performs differentiated resource allocation on the first channel and the second channel based on a resource scheduling model; establishes a finite state machine for each data item, and obtains the interaction data and transformation data generated by each data item within the historical data window at a preset period, and performs time alignment compensation on the transformation data; based on the interaction data and the compensated transformation data, calculates the upper and lower confidence levels of the transformation efficiency of each data item in the first channel and the second channel, and performs dynamic flow of data items between the two channels according to the upper and lower confidence levels, and skips the flow calculation for data items that have entered the first channel or the second channel before the preset protection period has expired. It can filter out worthless data in the early stages of the data processing chain to reduce computational load, achieve resource isolation between high-value data and exploratory data through dual-channel routing to avoid resource contention, and build an automated closed loop for the entire data lifecycle based on a confidence-based state machine feedback mechanism to reduce manual intervention, improve resource utilization efficiency and system stability. Attached Figure Description

[0028] To make the above-mentioned objectives, features and advantages of this application more apparent and understandable, the specific embodiments of this application will be described in detail below with reference to the accompanying drawings, wherein: Figure 1 This illustration shows a flowchart of a method for dynamic data filtering and distribution according to one aspect of this application. Figure 2 This invention provides a flowchart illustrating the overall steps of a data dynamic filtering and distribution method based on a multi-channel state machine in one embodiment of this application. Figure 3 This document shows a sub-flowchart of the data confidence calculation and state machine closed-loop transition logic in one embodiment of this application. Figure 4 This diagram illustrates a system architecture for dynamic data filtering and distribution according to another aspect of this application. Figure 5 This diagram illustrates the module logic structure of a system apparatus for dynamic data filtering and distribution according to an embodiment of this application. Figure 6 This diagram illustrates a structural schematic of an electronic device according to another aspect of this application.

[0029] The same or similar reference numerals in the accompanying drawings represent the same or similar parts. Detailed Implementation

[0030] To make the above-mentioned objectives, features and advantages of this application more apparent and understandable, the specific embodiments of this application will be described in detail below with reference to the accompanying drawings.

[0031] Many specific details are set forth in the following description in order to provide a full understanding of this application. However, this application may also be implemented in other ways different from those described herein, and therefore this application is not limited to the specific embodiments disclosed below.

[0032] As indicated in this application and claims, unless the context clearly indicates otherwise, the words "a," "an," "an," and / or "the" are not specifically singular and may include plural forms. Generally speaking, the terms "comprising" and "including" only indicate the inclusion of explicitly identified steps and elements, which do not constitute an exclusive list, and the method or apparatus may also include other steps or elements.

[0033] Figure 1The diagram illustrates a method for dynamic data filtering and distribution according to one aspect of this application, comprising steps S11 to S14. This method enables early removal of worthless data, isolation of dual-channel routing resources, and automated dynamic data flow based on feedback. This reduces ineffective consumption of system computing power and resources, improves the directionality and stability of resource allocation, balances the protection of core data benefits with the need for new data exploration, shortens the response cycle for hierarchical adjustments, reduces human maintenance costs, and adapts to the latency and volatility characteristics of transformed data.

[0034] Step S11: Obtain the initial data set and perform multi-level preprocessing filtering on the data set.

[0035] The data processing system receives a massive initial dataset to be processed via external interfaces, such as advertising platforms, e-commerce systems, or content platforms. This dataset contains data units to be distributed across different business scenarios, including search engine advertising keywords, e-commerce product items, and content recommendation candidates. Each data item is associated with obtainable multi-dimensional attribute parameters. The system performs multi-level preprocessing and filtering on the initial dataset, eliminating worthless data in the early stages of the data processing chain and reducing the computational load on subsequent evaluation and scheduling modules.

[0036] For example, a data processing system acquires approximately 100,000 new data items daily as an initial dataset through an external interface. Each data item carries multi-dimensional attribute parameters such as text content, category tags, traffic characteristics, competitive environment characteristics, and historical consumption boundaries. Specifically, an advertising system for a cross-border e-commerce merchant acquires approximately 100,000 new search keywords daily as an initial dataset through the API interface of a search engine advertising platform. Each keyword carries multi-dimensional attribute parameters such as text content, category tags, average search volume, competition index, and historical bids. The system performs multi-level preprocessing filtering on these 100,000 keywords, removing keywords containing roots without commercial conversion intent, keywords with category tags that do not match the target industry, and keywords with abnormal historical resource consumption boundaries in the early stages. This reduces the amount of data entering the subsequent competition evaluation and resource allocation stages to approximately 20,000 items, significantly reducing the computational load on subsequent neural network model inference and resource scheduling stages.

[0037] Step S12: The filtered data items are routed to the first channel and the second channel respectively. The first channel and the second channel are independent parallel data processing queues, and differentiated resource allocation is performed on the first channel and the second channel based on the resource scheduling model.

[0038] The filtered and retained data items are divided into two sets, which are then routed to the first and second channels, respectively. The first and second channels are independent parallel data processing queues established at the memory or database level. The resource allocation strategies and flow rules of the two channels are completely isolated to avoid data mixing. Based on a resource scheduling model, the system performs differentiated resource allocation for the first and second channels, configuring different weights for the total amount of system resources for each channel.

[0039] For example, the system sorts approximately 20,000 keywords remaining after three levels of filtering in descending order based on the comprehensive score output from the previous competition evaluation. The top N keywords (e.g., N=50) are selected to form the first set and routed to the first channel. The remaining keywords after the Nth position are formed into the second set and routed to the second channel. A larger portion of the total budget (e.g., 70%) is allocated to the first channel, and a smaller portion (e.g., 30%) is allocated to the second channel, ensuring that the budgets of the two channels do not overlap. The resource allocation threshold for each keyword in the first channel is biased towards the historical highest consumption boundary to ensure sufficient resource supply for high-certainty data; the resource allocation threshold for each keyword in the second channel is biased towards the historical lowest consumption boundary to achieve effect verification of new data at the lowest cost.

[0040] Step S13: Establish a finite state machine for each data item, acquire the interaction data and transformation data generated by each data item within the historical data window according to a preset period, and perform time alignment compensation on the transformation data.

[0041] A finite state machine is established for all data items in the campaign. Interaction and conversion data generated by each data item within the historical data window are obtained according to a set calculation period (e.g., daily or weekly). The interaction data includes quantitative data on user interactions with the data item, such as impressions, clicks, page views, and form submissions. The conversion data includes quantitative data on commercially valuable user behaviors such as purchases, registrations, and inquiries. Because conversion data has a time lag (users may click on an ad and complete the conversion several days or even weeks later), time alignment compensation is applied to the conversion data. A conversion delay distribution function is fitted based on the historical conversion delay distribution. This function is used to estimate the amount of delayed conversions that have not yet been reported back within the current statistical period. The estimated conversion amount and the already reported conversion amount are integrated into the compensated conversion data to avoid value assessment bias caused by data delay.

[0042] For example, the calculation cycle is set to run once daily at midnight, with the historical data window covering the past 30 days. Three days after a keyword goes live, the system counts 10 conversions that have been returned. Based on the industry's historical conversion delay distribution (approximately 20% of conversions are returned after 1 day, 50% after 3 days, 90% after 7 days, and 100% after 14 days), the system estimates that approximately 10 conversions are still pending. The compensated conversion data is then adjusted to 20 conversions before being used in subsequent confidence calculations, preventing the true value of the keyword from being underestimated due to incomplete conversion data.

[0043] Step S14: Based on the interactive data and the compensated conversion data, calculate the upper and lower confidence levels of the conversion efficiency of each data item in the first and second channels. According to the upper and lower confidence levels, perform dynamic flow of data items between the two channels, and skip the flow calculation for data items that have entered the first or second channel before the preset protection period has expired.

[0044] Based on the acquired interaction data and the compensated conversion data, a confidence interval evaluation algorithm based on historical data windows is used to calculate the upper and lower confidence levels of the conversion efficiency for each data item. According to these upper and lower confidence levels, dynamic flow of data items between the first and second channels is executed. This includes promoting data items that perform well in the second channel to the first channel, downgrading data items that perform poorly in the first channel to the second channel, and marking data items with extremely poor performance in the second channel as negative and cutting off resource allocation. For data items that have entered the first or second channel but have not yet reached the preset protection period (e.g., 14 days), the system skips this flow calculation and maintains the current state to avoid misjudging new data due to insufficient sample size.

[0045] If a keyword in the second channel reaches a preset activation threshold (e.g., 100 clicks), and its upper confidence level is greater than the statistical benchmark value of the first channel, while its lower confidence level is greater than the preset fluctuation range of the statistical benchmark value of the first channel, the system will promote the keyword from the second channel to the first channel and update its status. If the upper confidence level of a keyword in the first channel is lower than the preset absolute safety lower limit and lower than the statistical benchmark value of the first channel, the system will downgrade it from the first channel to the second channel. If a new keyword has only been in the second channel for 3 days (less than the 14-day protection period), the system will skip this transition calculation, maintain its current status, and wait for sufficient samples to accumulate before participating in the assessment.

[0046] According to the data dynamic filtering and distribution method described in this application, multi-level preprocessing filtering removes worthless data in the early stages of the data link, reducing the computational load of subsequent evaluation and scheduling modules; dual-channel independent routing achieves physical isolation between high-value data and data to be explored, avoiding resource contention between the two types of data; a confidence-based feedback update mechanism forms a closed loop for data processing, reducing the number of manual interventions; and time alignment compensation corrects evaluation biases caused by transformation delays, improving the accuracy of data value assessment.

[0047] In one embodiment of this application, in step S11, based on a preset semantic blacklist rule base, the text content of each data item in the data set is matched, and data items containing global negative word roots are removed; the classification labels of each remaining data item after removal are parsed, and data items whose classification labels do not conform to the target classification are blocked to obtain blocked data items.

[0048] The semantic blacklist rule base pre-stores negative word roots with no commercial conversion value (such as "free", "second-hand", "piracy" etc.). NLP semantic matching is performed on the text content of each data item in the dataset. The rule base stores global negative word roots; if a data item's text matches any negative word root, the data item is directly removed and not proceeded to the next processing step. Then, the category labels (annotations) carried by each data item are parsed and compared with the system's preset target industry categories (such as the category range corresponding to retail, education, and healthcare businesses). If the label does not belong to the target category range, the data item is directly blocked.

[0049] For example, the semantic blacklist rule base of advertisers in the education industry stores negative word roots such as "free," "second-hand," and "piracy." When keywords such as "free English courses" and "second-hand textbooks" are obtained from the search engine platform, these keywords are directly removed by matching negative word roots through text matching. At the same time, the category tags of each keyword (such as the annotations field) are parsed. If the category tag of a keyword is "retail-clothing" while the target category is "education-vocational training," then that keyword is blocked from entering the subsequent processing flow.

[0050] By using semantic blacklist matching and classification label interception, data items containing invalid intent and data items not belonging to the target category are removed before the data enters the subsequent evaluation stage, thereby reducing the amount of unnecessary computation in the subsequent competition evaluation and resource scheduling stages.

[0051] In one embodiment of this application, the multi-level preprocessing filtering further includes: extracting entity tags from the blocked data items; if the brand corresponding to the entity tag does not belong to the preset core brand set, then the data item is isolated to an independent comparison database and does not participate in the data processing flow of the first channel and the second channel; performing blacklisting and blocking processing on the data items isolated to the comparison database, so that the data items in the comparison database no longer obtain resource allocation and display opportunities, and disabling the triggering mechanism for the data items in the comparison database in the associated display scenario.

[0052] The system pre-sets a core brand set that includes proprietary brand identifiers. When the system extracts an entity tag from a data item that is not a proprietary brand, it isolates the data item containing that brand identifier to a separate comparison database. The system then blacklists the data items in the comparison database, setting negative flag rules through the distribution platform interface to prevent these data items from being displayed. Furthermore, it disables triggering for these non-proprietary brand data items in other data item association matching scenarios, preventing non-proprietary brand data items from entering the main channel and consuming system resources. Thus, by isolating non-proprietary brand data in a separate database and performing blacklisting and blocking through entity tag identification, the system avoids non-proprietary brand data interfering with resource allocation in the main channel. Simultaneously, by disabling the association display trigger mechanism, it prevents hidden consumption caused by the distribution platform's intelligent matching function.

[0053] In one embodiment of this application, the multi-level preprocessing filtering further includes: extracting the traffic characteristics and competitive environment characteristics of the blocked data items, removing data items whose traffic characteristics are less than a preset first threshold to obtain a first filtered data item; calculating a comprehensive evaluation factor based on the traffic characteristics and competitive environment characteristics of the first filtered data item, filtering data items whose comprehensive evaluation factor does not meet preset conditions to obtain a second filtered data item; extracting the historical highest consumption boundary and historical lowest consumption boundary of each data item in the second filtered data item, and filtering the corresponding data item in the second filtered data item when the historical highest consumption boundary is missing, the historical lowest consumption boundary is missing, or the historical lowest consumption boundary is greater than the industry benchmark value.

[0054] For data items retained after the previous level of semantic, classification, and entity filtering, two core features are extracted: first, traffic features, namely the average search volume index (AMS) of the data item; and second, competitive environment features, namely the competition index (CI) of the advertising scenario in which the data item is placed. This index reflects the number of entities participating in resource competition in the same scenario and the intensity of competition. First, data items with traffic features below a preset first threshold are eliminated to prevent data items without basic traffic support from entering subsequent processes. Then, the extracted AMS and CI are concatenated into an input feature vector, which is input to a pre-trained competition evaluation neural network model (which can be implemented using a multilayer perceptron regression model). This model is trained based on historical advertising data and can output a comprehensive score representing the survival value of data within the 0-1 range. The system also retains alternative evaluation logic without a neural network as a compatibility solution. The system dynamically generates an interception threshold based on the score distribution of the overall data pool, intercepting data items with a comprehensive score (or comprehensive evaluation factor K value) lower than this dynamic threshold, and retaining only data items with scores meeting the threshold for the next level of filtering.

[0055] The third level of resource consumption boundary filtering: For keywords that pass the competition assessment, their historical highest consumption boundary (HB) and historical lowest consumption boundary (LB) are further obtained. If HB is missing (indicating that the keyword has no historical ad spend data) or LB is greater than the industry benchmark value (AB, such as AB = 10 unit price in the education industry, and LB = 15 unit price for a certain keyword), then the keyword is filtered. Thus, through three progressive screenings—traffic characteristic threshold filtering, comprehensive evaluation factor filtering, and resource boundary filtering—invalid data with low traffic, high traffic with low competition, and data with abnormal resource consumption costs are eliminated, ensuring that the data entering the dual-channel distribution stage all have verifiable traffic value and reasonable resource consumption expectations.

[0056] In another embodiment of this application, when the total number of data items retained after multi-level preprocessing and filtering is less than a preset value N (e.g., the total number of retained items is 20 and N is set to 50), all 20 data items are routed to the first channel, and the second channel does not receive any data items during this period. In the next calculation period, when new data items pass through preprocessing and filtering, the newly added data items are routed to the second channel, ensuring that both channels have data items running. This allows for normal channel initialization even in cold start scenarios or when data volume is insufficient, preventing the system from failing to run due to an empty exploration channel.

[0057] In one embodiment of this application, different weights are assigned to the total amount of system resources for the first channel and the second channel, wherein the total resource weight of the first channel is greater than that of the second channel; a resource-to-upper-level allocation strategy is adopted for the data items in the first channel, and a resource-to-lower-level allocation strategy is adopted for the data items in the second channel.

[0058] For example, allocate 70% of the total budget to the first channel and 30% to the second channel. The resource allocation threshold for each keyword in the first channel is determined based on its historical highest consumption boundary, with the value biased towards the historical highest consumption boundary to ensure that keywords in the first channel receive sufficient resource allocation thresholds and exposure opportunities; the resource allocation threshold for each keyword in the second channel is determined based on its historical lowest consumption boundary, with the value biased towards the historical lowest consumption boundary to minimize the trial and error cost of new keywords.

[0059] By configuring different total resource weights for the two channels and adopting differentiated allocation strategies, we can ensure that high-deterministic data in the first channel receives sufficient resource supply, while simultaneously verifying the effectiveness of new data in the second channel at the lowest resource cost, thus avoiding resource competition between the two types of data.

[0060] In one embodiment of this application, the method further includes: determining the allocation strategy biased towards the upper limit of resources and the allocation strategy biased towards the lower limit of resources based on a nonlinear evaluation model; the nonlinear evaluation model takes the historical highest consumption boundary, the historical lowest consumption boundary and the industry benchmark value of the data item as input, and outputs the corresponding single resource allocation threshold according to the channel to which the data item belongs; wherein, the single resource allocation threshold of the first channel is biased towards the historical highest consumption boundary, and the single resource allocation threshold of the second channel is biased towards the historical lowest consumption boundary.

[0061] The nonlinear evaluation model receives the historical highest consumption boundary, historical lowest consumption boundary, industry benchmark value, and channel attribute (first channel) for a given data item. It outputs a single resource allocation threshold for that data item in the first channel that is biased towards the historical highest consumption boundary. This value is close to the historical highest consumption boundary to ensure the data item receives sufficient resource allocation, while not exceeding a preset multiple of the industry benchmark value to control costs. For the same data item in the second channel, the model outputs a value biased towards the historical lowest consumption boundary to obtain the necessary interactive data for testing at the lowest cost.

[0062] By using a nonlinear evaluation model to output differentiated resource allocation thresholds based on channel attributes, instead of fixed formulas or manual experience settings, resource allocation can adapt to the historical consumption characteristics of different data items, thereby improving the accuracy of resource allocation.

[0063] In one embodiment of this application, the top N data items are selected according to the ranking of traffic characteristics and routed to the first channel, where N is a preset positive integer, and the remaining data items are routed to the second channel. The first channel and the second channel correspond to two independent resource allocation units, and the differentiated allocation of system resources is achieved by adjusting the resource quota of each resource allocation unit. The resource allocation unit corresponding to the first channel has a single resource allocation threshold and an overall resource quota that are greater than those of the second channel, and contains fewer data items than the resource allocation unit corresponding to the second channel.

[0064] The top 50 data items, ranked by their traffic characteristics, are routed to the first channel. The corresponding first resource allocation unit is configured with a daily resource quota of 7000 units and a single-item resource allocation threshold biased towards the historical highest consumption boundary. The remaining approximately 19950 data items are routed to the second channel, with the corresponding second resource allocation unit configured with a daily resource quota of 3000 units and a single-item resource allocation threshold biased towards the historical lowest consumption boundary. When adjusting the resource ratio between channels, the system directly modifies the resource quota values ​​of the two resource allocation units, eliminating the need to adjust the allocation thresholds for each data item individually. For example, in an advertising scenario, this resource allocation unit represents the advertising campaign.

[0065] Dual-channel routing is carried out by two independent resource allocation units. The native budget control mechanism of the resource distribution platform is used to achieve hard isolation of channel resources, reducing the development complexity of cross-channel resource scheduling. At the same time, by configuring differentiated resource allocation thresholds and the number of data items, the calculation scope of the first channel is narrowed, and the resource acquisition priority of the first channel data is improved.

[0066] Following the above embodiments, configuring the total system resources with different weights includes: adjusting the resource quota ratio of the first channel to the total system resources based on the comprehensive evaluation result of the conversion efficiency of the data items in the first channel, so that the resource quota ratio is positively correlated with the comprehensive evaluation result of the conversion efficiency of the first channel, and setting a lower limit constraint value and an upper limit constraint value for the resource quota ratio.

[0067] The system sets a lower limit of 20% (Core:Exploration = 2:8) and an upper limit of 50% (Core:Exploration = 5:5) for the resource allocation ratio in the first channel. When the overall conversion efficiency evaluation result of data items in the first channel is better, the system adjusts the resource allocation ratio of the first channel to 50%; when the overall conversion efficiency evaluation result is poor, the resource allocation ratio is adjusted to 20%. The ratio adjustment is carried out gradually in preset steps to avoid fluctuations in data performance caused by abrupt changes in the budget ratio. Thus, through a dynamic ratio adjustment mechanism with upper and lower limits, resource allocation can be dynamically adjusted according to the actual performance of the first channel, while avoiding extreme ratios that could lead to insufficient exploration resources or damage to core benefits.

[0068] In one embodiment of this application, the finite state machine includes an exploratory state, a core state, and a negative elimination state; the dynamic transitions include a promotion transition where data items migrate from the exploratory state to the core state, a demotion transition where data items fall back from the core state to the exploratory state, and an elimination transition where data items migrate from the exploratory state to the negative elimination state.

[0069] For example, when a keyword first enters the system, it is assigned to the second channel in an exploratory state. After a cooling-off period, the system periodically evaluates its conversion efficiency. If it meets the promotion criteria, its status is updated to the core state and it is migrated to the first channel. If the keyword subsequently declines in performance in the first channel, meeting the downgrade criteria, its status is reverted back to the exploratory state and it is migrated back to the second channel. If the keyword performs extremely poorly in the second channel, meeting the elimination criteria, its status is updated to a negative elimination state and resource allocation is cut off. By defining multiple clear states and different types of flow rules, an automated closed-loop flow mechanism for the entire data lifecycle is constructed, enabling data to automatically migrate between channels based on real-time conversion performance without manual intervention.

[0070] Following the above embodiments, the promotion flow includes: for data items in the exploration state, acquiring the interaction feature quantity of the data item; when the interaction feature quantity reaches a preset activation threshold, the confidence level of the upper limit of the conversion efficiency of the data item is greater than the statistical benchmark value of the first channel, and the confidence level of the lower limit of the conversion efficiency of the data item is greater than a preset fluctuation range of the statistical benchmark value of the first channel, migrating the data item from the second channel to the first channel and updating the state of the data item to the core state; wherein, the number of data items migrated to the first channel in each flow cycle does not exceed a preset maximum single flow quantity limit, and when the number of data items that meet the promotion conditions exceeds the maximum single flow quantity limit, the number of data items corresponding to the maximum single flow quantity limit are selected according to the comprehensive evaluation factor sorting and promoted.

[0071] For example, the activation threshold is set at 100 clicks, and the maximum number of interactions per session is capped at 5. If a keyword in the exploration phase reaches 120 interactions (exceeding the 100-click activation threshold), its upper confidence level is 4% (greater than the statistical benchmark of the first channel at 2.5%), and its lower confidence level is 2% (greater than the preset fluctuation range of the statistical benchmark of the first channel at 1.5%), thus meeting the promotion criteria. If 8 keywords simultaneously meet the promotion criteria within the same period, but the maximum is 5, the system sorts them from highest to lowest according to the comprehensive evaluation factor, selects the top 5 for promotion, and retains the exploration qualification for priority evaluation in the next period. This avoids misjudgment due to small samples by using the interaction feature activation threshold, reduces erroneous promotions caused by short-term fluctuations by comparing double confidence levels with the statistical benchmark and preset fluctuation range, and prevents resource allocation fluctuations caused by a large influx of data into the first channel in a short period by using the maximum number of interactions per session and the sorting and queuing mechanism.

[0072] In one embodiment of this application, the degradation process includes: for a data item in the core state, when the confidence level of the upper limit of the conversion efficiency of the data item is lower than a preset absolute safety lower limit and lower than the statistical benchmark value of the first channel, the data item is migrated from the first channel to the second channel and the state of the data item is updated to the exploration state.

[0073] For example, if the absolute safety lower limit is set to 1.5%, and the upper limit confidence of a certain core state data item drops to 1.2% (below the absolute safety lower limit of 1.5%), and is also below the statistical benchmark value of 2.5% in the first channel, the system will migrate the data item from the first channel to the second channel and update its status to the exploration state, and reconfigure resources according to the resource allocation strategy of the second channel.

[0074] By using a dual-condition downgrade judgment (absolute safety lower limit and statistical benchmark value), we ensure that only data items with significantly declining conversion efficiency are downgraded, thus avoiding the accidental downgrade of high-quality data in the core channel due to short-term fluctuations.

[0075] In one embodiment of this application, the elimination process includes: for data items in the exploration state, when the confidence level of the upper limit of the conversion efficiency of the data item is lower than the preset elimination benchmark and lower than the preset fluctuation range of the statistical benchmark value of the second channel, updating the state of the data item to the negative elimination state and stopping the allocation of resources to the data item; adding the search terms corresponding to the data items in the negative elimination state to the search term blacklist, and setting the search term blacklist through the advertising platform interface so that the search terms no longer receive exposure.

[0076] For example, if the upper limit confidence level of a certain exploratory data item is 0.3% (lower than the preset elimination benchmark of 0.5%) and also lower than the preset fluctuation range of 0.8% for the second channel statistical benchmark, the status of this data item is updated to a negative elimination state, and no resources are allocated to it. Simultaneously, the text identifier corresponding to this data item is added to the negative identifier library through the distribution platform interface and set as a negative identifier, preventing this data item from receiving any display opportunities on the distribution platform, including in scenarios where it is associated with other data items. This automatically identifies extremely low-value data based on elimination conditions and cuts off resource allocation. The negative identifier library completely blocks the entry point for low-value data from the source of display, avoiding ineffective calculations and resource consumption of eliminated data in subsequent filtering, evaluation, and circulation stages.

[0077] In one embodiment of this application, a conversion delay distribution function is fitted according to the historical conversion delay distribution, and the future conversion amount of data items that have not yet generated complete conversion data within the delayed conversion window is estimated based on the conversion delay distribution function; the estimated conversion amount is integrated with the already returned conversion amount to obtain the compensated conversion data.

[0078] The system accumulates historical full conversion data and statistically analyzes the proportion of conversions to total conversions at different time intervals (approximately 20% return after 1 day, 50% after 3 days, 90% after 7 days, and 100% after 14 days after triggering a data item). A conversion delay distribution function is then fitted to this distribution. In a certain calculation period, if a data item has 10 returned conversions after 3 days online, the system inputs the returned conversions and the number of days online for this data item into the conversion delay distribution function. It estimates that there are approximately 10 delayed conversions that have not yet been returned. Integrating the estimated conversions with the returned conversions yields a compensated conversion count of 20, which is used for subsequent confidence calculations.

[0079] By compensating for conversion volume prediction based on historical delay distribution, the error in confidence calculation caused by the lag in conversion data is corrected, reducing the probability of data items being mistakenly downgraded or eliminated due to incomplete conversion data.

[0080] In one embodiment of this application, in step S14, the time when each data item enters the current channel is recorded, and the duration of each data item entering the current channel is determined according to the time; for data items whose duration is less than the preset protection period, the current state is maintained and the flow calculation is skipped, and for data items whose duration is not less than the preset protection period, the flow calculation is performed.

[0081] For example, if a preset protection period is set to 14 days, and a data item enters the second channel on June 1st, the system records its entry time. During the period from June 1st to June 14th (less than 14 days), the system skips the promotion, demotion, and elimination calculations for this data item, maintaining its exploratory state and allocating basic resources normally to accumulate interactive data. On June 15th (a period of at least 14 days), the system begins performing flow calculations for this data item, determining whether it meets the promotion conditions based on its upper and lower confidence levels.

[0082] The cooling-off period protection mechanism provides a buffer period for newly entered data items to accumulate interactive samples, avoiding statistical distortion due to insufficient sample size and reducing the probability of high-quality new data being eliminated prematurely or unstable data being promoted prematurely.

[0083] In one embodiment of this application, before calculating the upper and lower confidence levels of the conversion efficiency, abnormal interaction data generated by web crawlers or cheating bots is identified and removed, and the upper and lower confidence levels of the conversion efficiency are calculated based on the removed interaction data; wherein, the upper confidence level is taken as the upper bound of the conversion efficiency prediction interval, and the lower confidence level is taken as the lower bound of the conversion efficiency prediction interval.

[0084] Before calculating the confidence level, interactive behavior feature analysis is used to identify fake interactive data generated by suspected web crawlers or cheating bots (such as a single IP generating a large number of interactions in a short period of time but without conversion). After removing this abnormal interactive data, the conversion efficiency prediction interval for each data item is calculated based on the remaining normal interactive data. The upper confidence level is taken as the upper bound of the prediction interval, and the lower confidence level is taken as the lower bound of the prediction interval. The two values ​​together constitute the confidence interval for conversion efficiency.

[0085] By removing abnormal interaction data before confidence calculation, the interference of false data on conversion efficiency assessment is reduced, and the accuracy of confidence calculation is improved. By specifying that the upper and lower confidence limits are taken as the upper and lower bounds of the prediction interval, a clear and quantifiable evaluation basis is provided for subsequent state machine transition judgment.

[0086] In a specific embodiment of this application, such as Figure 2 As shown, step S110: Obtain the initial data set and perform initial screening and interception using multi-dimensional semantics and classification. After the system accesses the massive initial data set through an external interface, it first calls the preset semantic blacklist rule library to remove data items containing globally negative word roots; it parses the classification tags of the data items and blocks data items that do not conform to the target classification; it executes a competitor isolation strategy, extracts entity tags from the data items, and if it is determined to be a non-own core brand, it isolates it to an independent comparison database to prevent disruption of the main channel's resource allocation.

[0087] Step S210: Input the initially screened data into the pre-trained evaluation model and perform competition and resource boundary evaluation filtering. The system extracts the traffic characteristics (such as average search volume index) and competitive environment characteristics (such as competition index) of data items, inputs the above characteristics into the evaluation model to obtain a comprehensive score, and blocks data items with a comprehensive score lower than the dynamic threshold; at the same time, it extracts the historical highest consumption boundary and lowest consumption boundary of data items. If the boundary data is missing or the lower boundary deviates significantly from the industry benchmark value, the corresponding data item is filtered.

[0088] Step S310: The filtered data items are routed to the dual-channel architecture (first channel and second channel), and differentiated resource allocation is performed by the resource scheduling model. The system divides the data items into a first set and a second set according to the evaluation weights, and routes them to the first channel and the second channel respectively; different weights of system resource totals are configured for the two channels; the historical consumption boundaries, industry benchmark values ​​and channel attributes of each data item are processed by the resource scheduling model, and the allocation strategy biased towards the upper limit of resources for the first channel and the allocation strategy biased towards the lower limit of resources for the second channel are output.

[0089] Step S410: Start the finite state machine engine, calculate the confidence interval based on the time series data, and execute the dynamic closed-loop flow of data. The system establishes a finite state machine for all data items in progress, obtains feedback data according to the set calculation cycle, calculates the upper and lower confidence limits of the conversion efficiency of each data item based on the historical data window and the delayed conversion window, and triggers the state transition mechanism according to the calculation results to realize the promotion, demotion, and elimination flow of data items between the two channels.

[0090] like Figure 3 The specific implementation logic of the dynamic flow of the data state machine in step S410 shown includes the following steps: 1) Cooling-off Judgment: Determine if the time it takes for the currently processed data item to enter its current channel is less than the preset protection cooling-off period. If yes (i.e., the entry time is less than the protection cooling-off period), skip this flow calculation and maintain the current state of the data item; if no (i.e., the entry time is not less than the protection cooling-off period), proceed to the next calculation and begin the judgment of promotion, demotion, or elimination. The purpose of this step is to provide a buffer period for newly entering data items to accumulate interaction samples, avoiding statistical distortion due to insufficient sample size.

[0091] 2) Exploration Promotion: For data items in the exploration state (i.e., in the second channel), firstly, determine whether their interaction count (e.g., clicks) has reached the preset activation threshold. If so, further determine whether their upper limit confidence level of conversion efficiency is greater than the statistical benchmark value of the first channel, and whether their lower limit confidence level of conversion efficiency is greater than the preset fluctuation range of the statistical benchmark value of the first channel. If both conditions are met, change the data item's status identifier from the exploration state to the core state, and migrate it from the second channel to the first channel. Each promotion flow has a maximum single flow limit. When the number of data items meeting the promotion conditions exceeds the limit, select the data items within the aforementioned maximum limit according to the comprehensive evaluation factor ranking for promotion.

[0092] 3) Core Degradation: For data items in the core state (i.e., in the first channel), determine whether their upper confidence score of conversion efficiency (H-Score) is lower than the preset absolute safety lower limit (L-Score) and whether it is lower than the statistical benchmark value of the first channel. If both conditions are met, the data item is reverted from the core state to the exploration state, migrated from the first channel to the second channel, and resources are reconfigured according to the resource allocation strategy of the second channel.

[0093] 4) Negative Elimination: For data items in the exploratory state (i.e., in the second channel), determine whether their upper limit confidence level of conversion efficiency is lower than the preset elimination benchmark and whether it is lower than the preset fluctuation range of the statistical benchmark value of the second channel. If both conditions are met, update the data item's status to negative elimination state, completely stop allocating resources to the data item; at the same time, add the text identifier corresponding to the data item to the negative identifier library, and set the negative identifier through the distribution platform interface so that the data item will no longer receive resource allocation and display opportunities.

[0094] Figure 4The diagram illustrates a system architecture for dynamic data filtering and distribution according to another aspect of this application. The system includes: a multi-level preprocessing filtering module 10, used to acquire an initial data set and perform multi-level preprocessing filtering on the data set; a dual-channel scheduling module 20, used to route filtered data items to a first channel and a second channel respectively, the first channel and the second channel being independent parallel data processing queues, and performing differentiated resource allocation on the first channel and the second channel based on a resource scheduling model; and a state machine transition module 30, used to establish a finite state machine for each data item, acquire the interaction data and transformation data generated by each data item within a historical data window according to a preset period, and perform time alignment compensation on the transformation data; calculate the upper and lower confidence levels of the transformation efficiency of each data item in the first channel and the second channel based on the interaction data and the compensated transformation data; and execute the dynamic flow of data items between the two channels according to the upper and lower confidence levels, and skip the flow calculation for data items that have entered the first channel or the second channel before the preset protection period has expired.

[0095] The system is deployed on a cloud server. The multi-level preprocessing and filtering module obtains 100,000 data items from an external data platform daily through an interface and performs three levels of filtering. The dual-channel scheduling module routes the filtered 20,000 data items to two independent resource allocation units. The state machine transition module pulls the interaction and transformation data of each data item every morning and performs latency compensation. Based on the calculated confidence level, it performs promotion, demotion, and elimination operations of data items between the two resource allocation units.

[0096] In a specific embodiment of this application, such as Figure 5 As shown, the multi-level preprocessing filtering module includes a semantic interception unit, a classification blocking unit, and a competitor isolation unit. The semantic interception unit performs text matching and negative word root removal operations based on a semantic blacklist rule base; the classification blocking unit performs classification tag parsing and blocking operations for non-target category data items; and the competitor isolation unit performs entity tag extraction, non-own brand identification, and independent database isolation operations.

[0097] Feature Evaluation Network Module: This module performs competition evaluation and comprehensive score calculation. It receives retained data items from the multi-level preprocessing and filtering module, extracts their traffic and competition environment features, outputs a comprehensive score through a pre-trained evaluation model, and dynamically generates an interception threshold based on the score distribution, intercepting data items below the threshold. Simultaneously, it performs resource consumption boundary evaluation, filtering out data items with missing or abnormal boundary data.

[0098] The dual-channel scheduling module comprises a first allocation unit and a second allocation unit. The first allocation unit handles data routing and resource configuration operations for the first channel, outputting an allocation strategy biased towards the upper limit of resources. The second allocation unit handles data routing and resource configuration operations for the second channel, outputting an allocation strategy biased towards the lower limit of resources. Each allocation unit corresponds to an independent resource allocation unit, and dynamic adjustment of resources between channels is achieved by adjusting the resource quotas of each allocation unit.

[0099] The state machine transition module includes a confidence calculation unit, an upgrade / downgrade trigger unit, and a cooldown / freeze unit. The confidence calculation unit performs operations such as time alignment compensation, acquisition of interactive and conversion data within the historical data window, and calculation of the upper and lower confidence limits for conversion efficiency. The upgrade / downgrade trigger unit performs judgment and trigger operations for three types of state transitions: promotion, downgrade, and elimination. The cooldown / freeze unit performs operations such as recording the entry time of data items, comparing protection periods, and skipping transition calculations.

[0100] By using NLP semantics, system classification labels, and competitor entity recognition for multi-dimensional pre-blocking, compared to traditional single numerical threshold filtering, a large amount of "pseudo-high popularity" data with no commercial value can be eliminated at a very early stage, greatly reducing the computational load of subsequent neural network evaluation and resource allocation nodes, and improving the overall computational efficiency of the system.

[0101] It changed the flat allocation method of single channel. By forcibly dividing the data into two channels, "core protection" and "low-cost exploration", and using a resource scheduling model to provide a non-linear allocation strategy that approximates the upper and lower limits, it not only ensures the data processing priority and resource saturation of the top core data, but also completes the continuous exploration of long-tail unknown data with extremely low system trial and error costs.

[0102] This solves the problem of data hierarchy solidification. By introducing upper confidence scores (H-Score) and lower confidence scores (L-Score) that include delayed transformation features, and in conjunction with a "cooling-off period freeze protection" mechanism, the system can achieve self-iteration based on real feedback data without manual intervention. It not only possesses the ability to evolve through natural selection but also effectively prevents frequent system upgrades and downgrades caused by short-term data fluctuations.

[0103] Figure 6 The diagram illustrates an electronic device structure according to another aspect of this application, the device including at least a processor 601 and a memory 602.

[0104] Processor 601 may include one or more processing cores, such as a quad-core processor or an octa-core processor. Processor 601 may be implemented using at least one hardware form selected from DSP (Digital Signal Processing), FPGA (Field-Programmable Gate Array), and PLA (Programmable Logic Array). Processor 601 may also include a main processor and a coprocessor. The main processor, also known as a CPU (Central Processing Unit), is used to process data in the wake-up state; the coprocessor is a low-power processor used to process data in the standby state. In some embodiments, processor 601 may integrate a GPU (Graphics Processing Unit), which is responsible for rendering and drawing the content to be displayed on the screen. In some embodiments, processor 601 may also include an AI (Artificial Intelligence) processor, which is used to handle computational operations related to machine learning.

[0105] The memory 602 may include one or more computer-readable storage media, which may be non-transitory. The memory 602 may also include high-speed random access memory and non-volatile memory, such as one or more disk storage devices or flash memory devices. In some embodiments, the non-transitory computer-readable storage media in the memory 602 are used to store at least one instruction, which is executed by the processor 601 to implement a method for dynamic data filtering and distribution provided in the method embodiments of this application.

[0106] In some embodiments, the electronic device may also optionally include: a peripheral device interface and at least one peripheral device. The processor 601, memory 602, and peripheral device interface can be connected via a bus or signal line. Each peripheral device can be connected to the peripheral device interface via a bus, signal line, or circuit board. Indicatively, peripheral devices include, but are not limited to: radio frequency circuits, touch displays, audio circuits, and power supplies.

[0107] Of course, the electronic device may also include fewer or more components, and this embodiment does not limit this.

[0108] This application also provides a computer-readable medium having computer instructions stored thereon, which can be executed by a processor to implement a method for dynamic data filtering and distribution as described above.

[0109] When a method for dynamic data filtering and distribution is implemented as a computer program, it can also be stored as an article of manufacture in a computer-readable storage medium. For example, computer-readable storage media can include, but are not limited to, magnetic storage devices (e.g., hard disks, floppy disks, magnetic stripes), optical discs (e.g., compact discs (CDs), digital multifunction discs (DVDs)), smart cards, and flash memory devices (e.g., electrically erasable programmable read-only memory (EEPROM), cards, sticks, key drives). Furthermore, the various storage media described herein can represent one or more devices and / or other machine-readable media used for storing information. The term "machine-readable medium" can include, but is not limited to, wireless channels and various other media (and / or storage media) capable of storing, containing, and / or carrying code and / or instructions and / or data.

[0110] It should be understood that the embodiments described above are merely illustrative. The embodiments described herein may be implemented in hardware, software, firmware, middleware, microcode, or any combination thereof. For hardware implementation, the processor may be implemented within one or more application-specific integrated circuits (ASICs), digital signal processors (DSPs), digital signal processing devices (DSPDs), programmable logic devices (PLDs), field-programmable gate arrays (FPGAs), processors, controllers, microcontrollers, microprocessors, and / or other electronic units designed to perform the functions described herein, or combinations thereof.

[0111] Some aspects of this application can be executed entirely by hardware, entirely by software (including firmware, resident software, microcode, etc.), or by a combination of hardware and software. The aforementioned hardware or software may be referred to as a "data block," "module," "engine," "unit," "component," or "method." The processor may be one or more application-specific integrated circuits (ASICs), digital signal processors (DSPs), digital signal processing devices (DSPDs), programmable logic devices (PLDs), field-programmable gate arrays (FPGAs), processors, controllers, microcontrollers, microprocessors, or combinations thereof. Furthermore, aspects of this application may manifest as computer products residing in one or more computer-readable media, including computer-readable program code. For example, computer-readable media may include, but are not limited to, magnetic storage devices (e.g., hard disks, floppy disks, magnetic tapes, etc.), optical discs (e.g., compressed CDs, digital multifunction DVDs, etc.), smart cards, and flash memory devices (e.g., cards, sticks, key drives, etc.).

[0112] A computer-readable medium may contain a propagated data signal containing computer program code, for example, on baseband or as part of a carrier wave. This propagated signal may take various forms, including electromagnetic, optical, and so on, or suitable combinations thereof. A computer-readable medium can be any computer-readable medium other than a computer-readable storage medium, which can be connected to an instruction execution method, apparatus, or device to enable communication, propagation, or transmission of a program for use. The program code located on the computer-readable medium can be propagated through any suitable medium, including radio, cable, fiber optic cable, radio frequency signals, or similar media, or any combination of the above media.

[0113] The basic concepts have been described above. Obviously, for those skilled in the art, the above disclosure is merely illustrative and does not constitute a limitation of this application. Although not explicitly stated herein, those skilled in the art may make various modifications, improvements, and corrections to this application. Such modifications, improvements, and corrections are suggested in this application, and therefore remain within the spirit and scope of the exemplary embodiments of this application.

[0114] Furthermore, this application uses specific terms to describe embodiments of the application. For example, "an embodiment," "one embodiment," and / or "some embodiments" refer to a particular feature, structure, or characteristic related to at least one embodiment of the application. Therefore, it should be emphasized and noted that "an embodiment," "one embodiment," or "an alternative embodiment" mentioned twice or more in different locations in this specification do not necessarily refer to the same embodiment. In addition, certain features, structures, or characteristics in one or more embodiments of the application can be appropriately combined.

[0115] In some embodiments, numbers describing the quantity of components and attributes are used. It should be understood that such numbers used in the description of embodiments are modified in some examples with the terms "approximately," "approximately," or "generally." Unless otherwise stated, "approximately," "approximately," or "generally" indicates that the numbers are allowed to vary by ±20%. Accordingly, in some embodiments, the numerical parameters used in the specification and claims are approximate values, which may be changed depending on the characteristics required by individual embodiments. In some embodiments, numerical parameters should take into account specified significant digits and employ a general method of digit reservation. Although the numerical ranges and parameters used to confirm their breadth of scope in some embodiments of this application are approximate values, in specific embodiments, such values ​​are set as precisely as feasible.

Claims

1. A method for dynamic data filtering and distribution, characterized in that, The method includes: Obtain an initial data set and perform multi-level preprocessing filtering on the data set; The filtered data items are routed to the first channel and the second channel respectively. The first channel and the second channel are independent parallel data processing queues, and differentiated resource allocation is performed on the first channel and the second channel based on the resource scheduling model. A finite state machine is established for each data item, and the interaction data and transformation data generated by each data item within the historical data window are obtained at a preset period. Time alignment compensation is performed on the transformation data. Based on the interactive data and the compensated conversion data, the upper and lower confidence levels of the conversion efficiency of each data item in the first and second channels are calculated. According to the upper and lower confidence levels, the dynamic flow of data items between the two channels is executed, and the flow calculation is skipped for data items that have entered the first or second channel before the preset protection period has expired.

2. The method according to claim 1, characterized in that, The multi-level preprocessing and filtering of the data set includes: Based on a preset semantic blacklist rule base, the text content of each data item in the dataset is matched, and data items containing global negative word roots are removed. The classification labels of the remaining data items after the removal are analyzed, and the data items whose classification labels do not match the target classification are blocked to obtain the blocked data items.

3. The method according to claim 2, characterized in that, The multi-stage pretreatment filtration also includes: Extract the entity tags from the blocked data items. If the brand corresponding to the entity tag does not belong to the preset core brand set, then isolate the data item to an independent comparison database and do not participate in the data processing flow of the first channel and the second channel. The data items isolated to the comparison database are blacklisted, preventing them from receiving resource allocation and display opportunities. In related display scenarios, the triggering mechanism for the data items in the comparison database is disabled.

4. The method according to claim 2, characterized in that, The multi-stage pretreatment filtration also includes: Extract the traffic characteristics and competition environment characteristics of the blocked data items, and remove data items whose traffic characteristics are less than a preset first threshold to obtain the first filtered data items; Based on the traffic characteristics and competitive environment characteristics of the first screened data items, a comprehensive evaluation factor is calculated, and data items that do not meet the preset conditions are filtered out to obtain the second screened data items; Extract the historical highest consumption boundary and historical lowest consumption boundary of each data item in the second filtering data item. If the historical highest consumption boundary is missing, the historical lowest consumption boundary is missing, or the historical lowest consumption boundary is greater than the industry benchmark value, filter the corresponding data item in the second filtering data item.

5. The method according to claim 1, characterized in that, The differentiated resource allocation between the first channel and the second channel based on the resource scheduling model includes: The system resource totals with different weights are configured for the first channel and the second channel, wherein the resource total weight of the first channel is greater than the resource total weight of the second channel; A resource-cap-biased allocation strategy is adopted for data items in the first channel, and a resource-cap-biased allocation strategy is adopted for data items in the second channel.

6. The method according to claim 5, characterized in that, The method further includes: The allocation strategy for the upper limit of the biased resources and the allocation strategy for the lower limit of the biased resources are determined based on the nonlinear evaluation model, respectively. The nonlinear evaluation model takes the historical highest consumption boundary, historical lowest consumption boundary and industry benchmark value of the data item as input, and outputs the corresponding single resource allocation threshold according to the channel to which the data item belongs. Specifically, the single resource allocation threshold of the first channel is biased towards the side of the historical highest consumption boundary, while the single resource allocation threshold of the second channel is biased towards the side of the historical lowest consumption boundary.

7. The method according to claim 1, characterized in that, The step of routing the filtered data items to the first channel and the second channel respectively includes: The top N data items are selected based on the ranking of traffic characteristics and routed to the first channel, where N is a preset positive integer, and the remaining data items are routed to the second channel; The first channel and the second channel correspond to two independent resource allocation units. Differentiated allocation of system resources is achieved by adjusting the resource quota of each resource allocation unit. The resource allocation unit corresponding to the first channel has a single resource allocation threshold and an overall resource quota that are greater than those of the second channel, and contains fewer data items than the resource allocation unit corresponding to the second channel.

8. The method according to claim 5, characterized in that, The total amount of system resources configured with different weights includes: Based on the comprehensive evaluation result of the conversion efficiency of the data items in the first channel, the resource quota ratio of the first channel to the total system resources is adjusted so that the resource quota ratio is positively correlated with the comprehensive evaluation result of the conversion efficiency of the first channel, and a lower limit constraint value and an upper limit constraint value of the resource quota ratio are set.

9. The method according to claim 1, characterized in that, The finite state machine includes an exploratory state, a core state, and a negative elimination state; The dynamic flow includes the promotion flow of data items from the exploration state to the core state, the demotion flow of data items from the core state back to the exploration state, and the elimination flow of data items from the exploration state to the negative elimination state.

10. The method according to claim 9, characterized in that, The promotion process includes: For a data item in the exploration state, obtain the interaction feature quantity of the data item; When the interaction feature quantity reaches a preset activation threshold, the confidence level of the upper limit of the conversion efficiency of the data item is greater than the statistical benchmark value of the first channel, and the confidence level of the lower limit of the conversion efficiency of the data item is greater than the preset fluctuation range of the statistical benchmark value of the first channel, the data item is migrated from the second channel to the first channel and the state of the data item is updated to the core state. In each circulation cycle, the number of data items migrated to the first channel does not exceed the preset maximum circulation limit. When the number of data items that meet the promotion conditions exceeds the maximum circulation limit, the number of data items corresponding to the maximum circulation limit is selected according to the comprehensive evaluation factor and promoted.

11. The method according to claim 9, characterized in that, The downgrade process includes: For a data item in the core state, when the confidence level of the upper limit of the conversion efficiency of the data item is lower than the preset absolute safety lower limit and lower than the statistical benchmark value of the first channel, the data item is migrated from the first channel to the second channel and the state of the data item is updated to the exploration state.

12. The method according to claim 9, characterized in that, The elimination and turnover process includes: For a data item in the exploration state, when the confidence level of the upper limit of the conversion efficiency of the data item is lower than the preset elimination benchmark and lower than the preset fluctuation range of the statistical benchmark value of the second channel, the state of the data item is updated to the negative elimination state, and resources are stopped being allocated to the data item. Add the search terms corresponding to the data items in the negative elimination state to the search term blacklist, and set the search term blacklist through the distribution platform interface so that the search terms no longer receive resource allocation and display opportunities.

13. The method according to claim 1, characterized in that, The time alignment compensation for the transformed data includes: Based on the historical conversion delay distribution, a conversion delay distribution function is fitted, and the future conversion volume of data items that have not yet generated complete conversion data within the delayed conversion window is estimated based on the conversion delay distribution function. The estimated conversion rate is combined with the already returned conversion rate to obtain the compensated conversion data.

14. The method according to claim 1, characterized in that, The step of skipping the flow calculation for data items entering the first or second channel before the preset protection period has expired includes: Record the time when each data item enters the current channel, and determine the duration of each data item entering the current channel based on the time. For data items whose duration is less than the preset protection period, maintain the current state and skip the flow calculation; for data items whose duration is not less than the preset protection period, perform the flow calculation.

15. The method according to claim 1, characterized in that, Before calculating the upper and lower confidence levels of the conversion efficiency, the method further includes: Identify and remove abnormal interaction data generated by web crawlers or cheating bots, and calculate the upper and lower confidence levels of the conversion efficiency based on the removed interaction data; Wherein, the upper confidence level is taken as the upper bound of the conversion efficiency prediction interval, and the lower confidence level is taken as the lower bound of the conversion efficiency prediction interval.

16. A system for dynamic data filtering and distribution, characterized in that, The system includes: A multi-level preprocessing and filtering module is used to obtain an initial data set and perform multi-level preprocessing and filtering on the data set; The dual-channel scheduling module is used to route filtered data items to the first channel and the second channel respectively. The first channel and the second channel are independent parallel data processing queues, and differentiated resource allocation is performed on the first channel and the second channel based on the resource scheduling model. The state machine transition module is used to establish a finite state machine for each data item, acquire the interaction data and transformation data generated by each data item within the historical data window according to a preset period, and perform time alignment compensation on the transformation data; calculate the upper and lower confidence levels of the transformation efficiency of each data item in the first and second channels based on the interaction data and the compensated transformation data, and execute the dynamic transition of the data item between the two channels according to the upper and lower confidence levels, and skip the transition calculation for data items that have entered the first or second channel before the preset protection period has expired.

17. An electronic device, characterized in that, include: A memory, on which computer programs are stored; And a processor, which, when executing the computer program, implements the steps of the method for dynamic data filtering and distribution as claimed in any one of claims 1 to 15.

18. A computer-readable storage medium storing computer-readable instructions thereon, characterized in that, The computer-readable instructions can be executed by a processor to implement the method as described in any one of claims 1 to 15.

Images