A method for missing reconstruction and admission control of heterogeneous consumer behavior signals

Abstract

The present application relates to heterogeneous data processing and missing observation signal reconstruction technology. The method comprises: identifying original states such as valid values, valid zeros, page missing and acquisition anomalies; converting complete observation tracks with real interaction values into entity internal relative shares; learning the mapping of sequence to relative share only by using complete observation entities with real interaction values and current sequence information in the same acquisition batch; reconstructing the same semantic relative share without generating absolute interaction values for records only with sequence information; performing access judgment on the reconstruction result by excluding entity-level masking verification of entities participating in the training of the target entity; allowing the estimated track to enter the subsequent processing as a unified preference value only when the predetermined access conditions are met, otherwise performing failure shutdown. The present application can reduce data pollution caused by mixing of heterogeneous missing states and low-trust estimates propagating downstream.

Description

Technical Field

[0001] This invention relates to the fields of computer data processing, big data preprocessing, semantic alignment of heterogeneous observation signals, reconstruction of missing interactive signals, and data quality control. In particular, it relates to a computer implementation method for performing track-based processing on different states simultaneously in the same data source, including complete interactive values, ordered bit information only, missing pages, acquisition anomalies, and mixed states, and estimating signal propagation through verification gating control. Background Technology

[0002] In the scenario of collecting data on catering and other consumer behaviors, different entities on the same information platform may display different types of interaction signals: some records contain real interaction values, some records only retain the current sequence information, and there may also be valid zero values, empty values, pages not displayed, collection failures, and mixed states.

[0003] The statistical objects, time windows, and page display logic for different interaction fields may differ. If various missing states are uniformly imputed, or the current ordinal position is directly converted into an absolute interaction quantity, it's easy to forcibly mix the semantics of incomparable data. Valid zero values ​​also have different data meanings than page missing values ​​or data collection anomalies.

[0004] Furthermore, if estimation results are generated solely based on the current ordinal position without independent hold-out verification or entity-level occlusion verification, estimation errors may propagate directly to subsequent aggregation and computation processes. Therefore, a data processing method is needed that can distinguish between the original state, unify the semantics of complete orbit and ordinal orbit data, and control the admission of estimation results through independent verification. Summary of the Invention

[0005] The technical problem to be solved by this invention is to reduce the semantic confusion of different missing states in heterogeneous consumer behavior observation data, the inconsistency of the dimensions of complete observation tracks and ordinal estimation tracks, the self-verification of ordinal mapping training, and the data pollution caused by the propagation of low-reliability estimation results to subsequent data processing.

[0006] To address the aforementioned issues, this invention provides a method for reconstructing missing heterogeneous consumer behavior signals and implementing admission control. This method includes original state identification, complete observation trajectory relativization, batch complete entity mapping learning, ordinal estimation and trajectory semantic reconstruction, entity-level occlusion verification, admission control, and unified output.

[0007] Among them, the original state identification treats valid values ​​and valid zero values ​​as complete observations; for records with missing interactive values ​​but still having current sequence information, they are entered into the sequence estimation track; for abnormal states such as acquisition failure, estimation is not forced; for mixed states, only the real records that can be confirmed are retained and the abnormality marker is retained.

[0008] The true interaction values ​​in a complete observation track are first converted into relative shares within the current observable set of each entity, so that the absolute interaction scales are not directly compared between different entities. Then, using only complete observation entities within the same acquisition batch that have both true interaction values ​​and current order information, the mapping relationship from the current order to the statistical representative value of the true relative share is learned.

[0009] For records that retain only the current order information, the system generates candidate relative values ​​based on the mapping relationship and normalizes them within the current observable set of the corresponding entity to obtain an estimated relative share with the same data semantics as the true relative share, without restoring the unverifiable absolute interaction value.

[0010] To control the reliability of the mapping relationship, the system performs entity-level occlusion verification. The true interaction value of the target entity is hidden during verification, but its current order information is retained. The mapping relationship used for verification of this target entity is established by other complete observed entities and does not include the target entity. The system compares the reconstruction result with the hidden true relative share of the target entity and generates a verification admission state based on one or more predetermined indicators. Only when the verification meets the predetermined admission conditions is the estimated relative share allowed to proceed to subsequent processing; otherwise, execution fails and is closed.

[0011] By adopting the above scheme, we can distinguish between valid zero values, business missing values, and collection anomalies, and reduce data pollution caused by uniform imputation of different missing semantics; through real track relativization and batch mapping learning, the reconstructed values ​​of missing tracks are in the same data semantics as the real tracks; through entity-level occlusion verification, we reduce the self-verification of training samples; and through failure closure, we reduce the probability of low-confidence estimates entering the downstream processing chain. Attached Figure Description

[0012] Figure 1 The flowchart illustrates the overall process of the method for reconstructing and controlling the access of heterogeneous consumer behavior signals provided in this embodiment of the invention.

[0013] Figure 2 This is a schematic diagram of the entity-level occlusion verification process provided in an embodiment of the present invention. Detailed Implementation

[0014] The present invention will be further described below with reference to the accompanying drawings and one embodiment. This embodiment is used to illustrate the technical flow of the present invention and does not limit specific data sources, specific business platforms, specific grouping methods, or parameter values. The overall flow of this embodiment is as follows: Figure 1 As shown.

[0015] The first step is to identify the state of the original signal. For numerical states, including values ​​of 0, the signal is assigned to the complete observation track; for null values ​​or records not displayed on the page but still having current sequence information, the signal is assigned to the sequence estimation track; for records with failed acquisitions, the signal is marked as an abnormal state and no estimation is performed; for mixed states, the verifiable true numerical records are retained, and the unverifiable parts are retained as abnormal states.

[0016] The second step is to perform intra-entity relativization on the complete observation trajectory. For record i in entity s, the smoothed true relative share can be calculated using the following formula:

[0017] thumb_share_si = (thumb_si + C / M_s) / (Σ_j thumb_sj + C)

[0018] Where thumb_si represents the actual interaction value of record i, M_s represents the number of records in entity s that actually enter the calculation, and C represents the smoothing intensity parameter. This processing yields the relative share of a record within the current observable set of this entity, rather than the absolute interaction value compared directly across entities.

[0019] The third step is to establish the mapping relationship between the current rank and the true relative share. Training samples are selected only from complete observation entities within the same collection batch that simultaneously possess both true interaction values ​​and current rank information. The system first calculates the true relative share of each training entity, then aggregates the true relative shares corresponding to the same current rank r to form rank_profile_g(r). The aggregated representative value can be the mean, median, quantile statistics, or a smoothed statistical representative value.

[0020] The mapping relationship can be established according to a predetermined group g. When the subgroup samples are insufficient, the mapping relationship can be backed to a coarser group or the entire complete observation sample, and at least some of the following items, profile_version, group_key, fallback_level, training_batch and sample_count, can be recorded for auditing and reproduction.

[0021] The fourth step is to perform semantic reconstruction on the ordinal estimation track. For record i that retains only the current ordinal position, let q_si = rank_profile_g(rank_si), and then normalize it within the current observable set of the corresponding entity according to the following formula:

[0022] estimated_share_si = q_si / Σ_j q_sj

[0023] After normalization, the sum of estimated_share for all records within an entity is 1. This restores the relative shares within the entity, rather than the missing absolute interaction values.

[0024] The fifth step is to unify the representation of the true and estimated tracks. The unified preference value for the complete observation track is the true relative share; after passing the validation admission, the unified preference value for the ordinal estimated track is the estimated relative share. At least some of the following items are saved: signal_source, raw_state, profile_version, fallback_level, and validation_status, so that the two types of values ​​remain auditable under the unified field.

[0025] Step 6: Perform entity-level occlusion verification, the process of which is as follows: Figure 2 As shown. For example, select brand A, which originally has real interaction values, as the validation object: hide brand A's real interaction values, but retain its current order; when establishing the mapping relationship for validation, only use other complete observed brands; reconstruct its estimated relative share based on brand A's retained current order; and then compare the estimated relative share with brand A's hidden real relative share. Repeat this process for multiple brands to form brand-level masked cross-validation.

[0026] Step 7: Perform verification gating. Verification can employ one or more of the following indicators: sorting consistency, head record overlap, preference concentration error, and systematic overestimation or underestimation bias. The system sets predetermined admission criteria for the selected indicators; only mapping relationships or mapping groups that meet the admission criteria are granted admission status.

[0027] Step 8: Close if execution fails. When validation passes, the estimated relative share can be used as the estimation track for the unified preference value in subsequent processing; when validation fails, the estimated track must not be merged with the complete observation track, and can be output separately, left missing, or sent for manual review; when the sample is insufficient, it can fall back to a coarser-level mapping relationship and be revalidated. If the revalidation still fails, the process will continue to be blocked.

[0028] In an exemplary data structure, an output record can be represented as Preference_Record={entity_id,item_id,preference_value,signal_source,raw_state,profile_version,group_key,fallback_level,validation_status,audit_flag}. Field names can be replaced as long as the traceability between relative values, source track, and validation status is maintained.

[0029] For example, within the same data collection batch, there are several complete observed brands with actual interaction values ​​and current menu positions. The system first calculates the actual relative share within each brand's representative entity, and then statistically analyzes typical shares according to the current position to form a mapping relationship. For a certain brand Z, its actual interaction value is not displayed but its current position is retained. The system reconstructs and estimates the relative share based on the mapping relationship. Subsequently, using a complete observed brand A as the masking object, brand A is excluded when training the mapping relationship, and only other brands are used to reconstruct brand A. The reconstruction result is then compared with the hidden actual relative share of brand A. Only when predetermined verification conditions are met is the estimated relative share of brand Z allowed to proceed to subsequent processing.

Claims

1. A method for reconstructing missing heterogeneous consumer behavior signals and implementing admission control, characterized in that, include: Acquire observation data on the consumption behavior of multiple entities within the same collection batch, wherein the observation data includes at least one of the interaction numerical status and current sequence information; The original state identification is performed on the consumer behavior observation data. Valid values ​​and valid zero values ​​are assigned to the complete observation track. Records with missing interaction values ​​but current sequence information are assigned to the sequence estimation track. Abnormal collection records are marked as abnormal states and are not subject to forced estimation. For the complete observation track, a true relative share with a sum of 1 is generated within the current observable set of the corresponding entity based on the true interaction values ​​of each record. Only complete observation entities within the same acquisition batch that simultaneously possess true interaction values ​​and current order information are used. The true relative shares are aggregated according to the current order, establishing a mapping relationship from the current order to the statistical representative value of the true relative share. For the order estimation track, candidate relative values ​​are obtained from the mapping relationship based on the current order of the corresponding record, and normalized to an estimated relative share with a sum of 1 within the current observable set of the corresponding entity. This ensures that the estimated relative share has the same data semantics as the true relative share, without generating missing absolute interaction values. Entity-level occlusion verification is performed on the mapping relationship, where the true interaction values ​​of the target complete observation entity are hidden while its current order is preserved. The target entity is excluded when establishing the mapping relationship for verification of the target entity, thus reconstructing the estimated relative share of the target entity. The estimated relative share of the target entity is compared with its hidden true relative share based on one or more predetermined verification indicators, generating a verification admission status. And only when the verification admission status meets the predetermined admission conditions, the estimated relative share is allowed to be used as the unified preference value of the estimated orbit for subsequent data processing; when the verification admission status does not meet the predetermined admission conditions, the corresponding estimated orbit is closed due to failure, preventing its merging with the unified preference value of the complete observation orbit.

2. The method according to claim 1, characterized in that, Valid zero values ​​are treated as true observations and not as missing values; the original status identifiers are saved for the page not displayed, null value status, and acquisition failure status.

3. The method according to claim 1, characterized in that, For record i in entity s, the true relative share is calculated according to the following formula: thumb_share_si=(thumb_si+C / M_s) / (Σ_j thumb_sj+C), where thumb_si is the true interaction value, M_s is the number of records entering the calculation, and C is the smoothing intensity parameter.

4. The method according to claim 1, characterized in that, The mapping relationship is formed by aggregating the true relative shares of complete observation entities in the same collection batch according to the current sequence r. The aggregated representative value adopts at least one of the mean, median, quantile statistics or smoothed statistical representative value.

5. The method according to claim 1, characterized in that, The mapping relationship is established according to a predetermined group; when the sample size of the subgroup is insufficient, it is back to a coarser group or all complete observation samples according to a predetermined level, and at least three of the following are recorded: mapping version, group identifier, back-down level, training batch, and sample size.

6. The method according to claim 1, characterized in that, For record i in the ordinal estimation track, the candidate value q_si is obtained from the mapping relationship of the corresponding group according to its current ordinal position, and normalized in the current observable set of the corresponding entity according to estimated_share_si=q_si / (Σ_j q_sj) so that the sum of the estimated relative shares is 1.

7. The method according to claim 1, characterized in that, The true relative share of the complete observation track and the estimated relative share of the ordinal estimated track are stored together as a unified preference value. At the same time, at least three of the following are saved: signal source, original state, mapping version, fallback level, and verification state, to maintain the auditable distinction between the true track and the estimated track.

8. The method according to claim 1, characterized in that, The entity-level occlusion verification is a brand-level occlusion cross-validation: brands with complete observation data are selected sequentially as target brands, the target brands are excluded during the mapping relationship training phase, and the relative share is reconstructed and estimated based on the current order of the brand.

9. The method according to claim 1, characterized in that, The predetermined verification metrics include at least one of sorting consistency, head record overlap, preference concentration error, and systematic overestimation or underestimation bias.

10. The method according to claim 1, characterized in that, When there are insufficient samples to represent the state, the verification is re-executed using a coarser-level mapping relationship; if the re-verification still does not meet the admission criteria, the estimation track is kept isolated, the uncalculable state is output, or it enters manual review.

11. A heterogeneous consumer behavior signal processing system, characterized in that, The device includes a processor and a memory, wherein the memory stores a computer program that, when executed by the processor, causes the processor to perform the method according to any one of claims 1 to 10.

12. An electronic device comprising a processor and a memory, characterized in that, The memory stores a computer program, which, when executed by the processor, causes the electronic device to perform the method described in any one of claims 1 to 10.

13. A computer-readable storage medium having a computer program stored thereon, characterized in that, When the computer program is executed by a processor, it implements the method according to any one of claims 1 to 10.

Images