E-commerce homogeneous advertisement material cleaning method and device based on multi-modal learning

Abstract

The invention discloses an e-commerce homogeneous advertisement material cleaning method and device based on multi-modal learning, and relates to the technical field of e-commerce advertisements. The method comprises the steps of obtaining a to-be-processed advertisement material, extracting image, text and audio data of the to-be-processed advertisement material, and obtaining a multi-modal feature vector by utilizing a visual language model and an audio fingerprint model which are pre-trained through comparative learning; inputting the multi-modal feature vector into a deep multi-modal attitude measurement learning model to calculate a homogenized similarity score, and judging a homogenized material group by adopting a hierarchical density clustering algorithm; in each homogenized material group, based on a performance threshold dynamically determined by statistical analysis of historical delivery data, screening out to-be-cleaned materials with poor performance; responding to the cleaning instruction, determining the priority according to the last updating timestamp of the cleaning rule, and deleting the screened materials. According to the invention, accurate and automatic cleaning of homogenized and low-efficiency materials can be realized, and the advertisement management efficiency and the putting benefit are improved.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of e-commerce advertising, and particularly relates to an e-commerce homogenization advertisement material cleaning method and device based on multi-modal learning. BACKGROUND

[0002] With the rapid development of e-commerce, digital advertising has become the core link of enterprise marketing. Advertisers often need to manage a large amount of advertising materials, including images, videos, scripts, etc., in multiple e-commerce platforms and advertising accounts. In order to improve the advertising delivery effect and budget utilization efficiency, it is crucial to effectively manage these materials, especially to clean up redundant and inefficient materials in a timely manner.

[0003] In actual operation, in order to cover different delivery scenarios and test effects, advertisers will create a large number of homogenized materials that are similar or identical in content. Over time, these materials accumulate rapidly, leading to a bloated material library and increasing management difficulty. Currently, the cleaning of homogenized materials mainly relies on manual operation. The operator needs to rely on experience to judge the similarity of the materials by comparing pictures with the naked eye and reading scripts, and manually selects and deletes the materials with poor performance in combination with the background data report. This method is not only inefficient and consumes a lot of manpower, but also is subjective and prone to incorrect deletion or omission, making it difficult to adapt to large-scale and fast-paced advertising delivery needs.

[0004] To solve the drawbacks of manual cleaning, some automatic cleaning tools have appeared in the prior art. However, these tools have obvious technical limitations. First, they mostly make similarity judgments based on a single modality, such as using only image hash values for comparison or keyword matching. This method cannot comprehensively evaluate the overall semantics of advertising materials composed of multiple modalities such as images, scripts, and audio, resulting in inaccurate judgments of similar content. Second, the existing similarity measurement methods are relatively primitive and cannot identify materials with consistent core content after slight modifications such as cropping, color adjustment, or changing background music, resulting in unsatisfactory recall rate and accuracy rate. In addition, when selecting objects to be cleaned, these tools usually use fixed and static performance thresholds (such as fixed return on investment or consumption thresholds), which cannot be adjusted adaptively according to market environment, account historical performance, and other dynamic factors, making the cleaning strategy rigid and possibly deleting potential materials or retaining truly inefficient materials.

[0005] In summary, the existing e-commerce advertising material cleaning method relies too much on manual work, the automatic tool has single function and low intelligent degree, which leads to the inability to accurately and efficiently identify and process homogenized and inefficient materials. This not only increases the operation burden, but also may cause internal competition among homogenized materials, resulting in waste of advertising budget and ultimately affecting the overall advertising effectiveness. Therefore, there is an urgent need for a technical solution that can intelligently, dynamically and comprehensively manage and clean e-commerce advertising materials. SUMMARY

[0006] The present application provides an e-commerce homogenized advertising material cleaning method and device based on multi-modal learning, which solves the problem of low efficiency of manual cleaning and insufficient accuracy of automatic recognition in the prior art.

[0007] In a first aspect, the present application provides an e-commerce homogenized advertising material cleaning method based on multi-modal learning, which comprises:

[0008] Obtaining a plurality of to-be-processed advertising materials belonging to at least one advertising account;

[0009] For each advertising material, extracting the image data, text data and audio data contained therein; using a visual language model pre-trained by contrast learning to map the image data and text data to a unified semantic embedding space to generate a joint visual text feature vector, and using an audio fingerprint model to extract an audio feature vector, thereby obtaining a multi-modal feature vector of each advertising material;

[0010] Inputting the multi-modal feature vector into a pre-trained deep multi-modal metric learning model to calculate the homogenization similarity score between any two advertising materials in a metric space optimized for measuring material similarity; and using a hierarchical density clustering algorithm that can automatically determine the number of clusters and identify noise points to cluster all advertising materials, and determining the materials belonging to the same cluster in the clustering result as a group of homogenized materials;

[0011] In each homogenized material group, obtaining the performance data of each material in the homogenized material group within a preset data period, the performance data including the consumption amount and the return on investment; based on the statistical analysis of the historical performance data of the same category of advertising materials of at least one advertising account within a historical time window, dynamically determining a dynamic performance threshold for filtering; from each homogenized material group, filtering out the materials with performance data lower than the dynamic performance threshold as to-be-cleaned materials;

[0012] In response to the cleaning instruction, when there are multiple cleaning rules under the same advertising account, the priority is determined according to the last update timestamp of the cleaning rule, and the to-be-cleaned materials filtered out by the rule with the highest priority are automatically executed.

[0013] Preferably, the step of generating joint visual-textual feature vectors using a vision-language model pre-trained with contrastive learning and extracting audio feature vectors using an audio fingerprint model comprises:

[0014] The vision-language model comprises an image encoder and a text encoder, which are pre-trained with a contrastive learning loss function to maximize cosine similarity of feature vectors of matched image-text pairs and minimize cosine similarity of feature vectors of mismatched image-text pairs.

[0015] The audio fingerprint model generates audio feature vectors by extracting Mel-frequency cepstral coefficients from audio data of each advertising material.

[0016] Preferably, the step of clustering all advertising materials using a hierarchical density-based clustering algorithm capable of automatically determining the number of clusters and identifying noise points comprises:

[0017] Constructing an all-pairs distance graph of all advertising materials based on homogenization similarity scores;

[0018] Constructing a minimum spanning tree from the all-pairs distance graph;

[0019] Constructing a clustering hierarchy containing all data points based on the minimum spanning tree;

[0020] Pruning the clustering hierarchy based on a pre-set minimum cluster size to obtain a simplified cluster tree;

[0021] Extracting final homogenized material groups based on the stability of each cluster in the cluster tree, and identifying materials not grouped into any cluster as non-homogenized noise points.

[0022] Preferably, after the step of obtaining multi-modal feature vectors for each advertising material, the method further comprises:

[0023] Using a no-reference image quality assessment model based on a convolutional neural network to score the quality of images or video key frames in the advertising materials;

[0024] Quantifying the blurriness of the image or video key frame by applying a Laplacian operator to the image or video key frame and calculating the variance of its response;

[0025] Marking advertising materials with a quality score below a first pre-set quality threshold or a blurriness variance below a second pre-set quality threshold as low-quality materials and including them in the range of materials to be cleaned up.

[0026] Preferably, after the step of obtaining multi-modal feature vectors for each advertising material, the method further comprises:

[0027] The multimodal feature vectors of the advertising creatives, along with user profile features and contextual features, are input into a pre-trained multimodal click-through rate prediction model to obtain the predicted click-through rate value for each advertising creative.

[0028] Creatives with predicted click-through rates below a dynamic efficiency threshold are marked as inefficient creatives and included in the list of creatives to be cleaned up.

[0029] Preferably, after obtaining the multimodal feature vector of each advertising creative, the method further includes:

[0030] Extract keyframes from images or videos of advertising materials;

[0031] Keyframes are retrieved from a public network media asset library through a robust reverse image search application programming interface that performs image transformations, including cropping, rotation, scaling, brightness adjustment, contrast adjustment, and watermarking.

[0032] If the search results contain third-party materials that were published earlier than the current ad creative and whose perceptual hash similarity to the keyframe is higher than a reposting threshold, then the current ad creative will be marked as a reposted material and included in the list of materials to be cleaned up.

[0033] Preferably, the step of dynamically determining a dynamic performance threshold for filtering, based on statistical analysis of historical performance data of at least one advertising account's advertising creatives within a historical time window, includes:

[0034] Collect historical performance data of advertising creatives of the same category under at least one advertising account within the past N preset time windows, and construct the time series distribution of the return on investment (ROI) indicator.

[0035] An adaptive threshold algorithm based on historical data distribution is adopted, which sets the preset quantile of the time series distribution or the value that is lower than the preset multiple standard deviation of the mean as the current dynamic performance threshold. The preset quantile ranges from 10 to 30 quantiles.

[0036] Preferably, the step of calculating the homogeneity similarity score between any two advertising creatives is implemented using the following weighted fusion formula:

[0037]

[0038] in:

[0039] For material and The final homogeneity similarity score between them;

[0040] The materials are respectively The visual feature vector output by the image encoder;

[0041] The materials are respectively The text feature vector output by the text encoder;

[0042] The materials are respectively The audio feature vector output by the audio fingerprint model;

[0043] This is the function for calculating cosine similarity.

[0044] The weighting coefficients for each mode satisfy the following conditions: ,and The weights are determined by a pre-trained attention network based on the type of advertising campaign to which the content belongs. Dynamically generated:

[0045] ,in This is an embedding function used to map advertising campaign types to embedding vectors.

[0046] Preferably, the step of determining priority based on the last update timestamp of the cleanup rule includes:

[0047] Before performing the deletion operation, iterate through all enabled cleanup rules applicable to the advertising account;

[0048] Extract the last modified timestamp of each enabled cleanup rule;

[0049] Set the cleanup rule with the latest timestamp as the only valid rule to be executed at present, and ignore the rest of the rules until the next cleanup task is triggered.

[0050] Secondly, embodiments of the present invention provide an e-commerce homogenized advertising material cleaning device based on multimodal learning, comprising:

[0051] The acquisition unit is used to acquire multiple ad creatives to be processed, belonging to at least one ad account;

[0052] The feature extraction unit is used to extract the image data, text data and audio data contained in each advertising material; using a visual language model pre-trained by contrastive learning, the image data and text data are mapped to a unified semantic embedding space to generate a joint visual text feature vector, and the audio fingerprint model is used to extract the audio feature vector, thereby obtaining the multimodal feature vector of each advertising material.

[0053] The homogenization grouping unit is used to input multimodal feature vectors into a pre-trained deep multimodal attitude quantization learning model to calculate the homogenization similarity score between any two ad creatives in a metric space optimized for measuring creative similarity. It also employs a hierarchical density clustering algorithm that can automatically determine the number of clusters and identify noise points to cluster all ad creatives, and determines the creatives belonging to the same cluster in the clustering results as a group of homogenized creatives.

[0054] The creative filtering unit is used to obtain the performance data of each creative in each homogeneous creative group within a preset data period. The performance data includes spending amount and return on investment. Based on the statistical analysis of the historical performance data of the same category of advertising creatives of at least one advertising account within a historical time window, a dynamic performance threshold for filtering is dynamically determined. From each homogeneous creative group, creatives with performance data lower than the dynamic performance threshold are selected as creatives to be cleaned up.

[0055] The execution unit is used to determine the priority of cleanup rules based on the last update timestamp of the cleanup rules when there are multiple cleanup rules under the same advertising account, and automatically perform deletion operations on the cleanup materials selected by the rule with the highest priority.

[0056] Beneficial effects:

[0057] By leveraging a multimodal learning approach combining visual language models and audio fingerprint models, deep cross-modal semantic understanding of advertising creatives is achieved, enabling high-precision identification of homogeneous creatives. This overcomes the limitations of existing technologies that rely on single modalities or simple feature comparisons, resulting in low accuracy and an inability to handle subtle variations. A dynamic performance threshold determination mechanism based on historical campaign performance data is introduced, replacing traditional fixed, static filtering rules. This makes the selection of inefficient creatives more intelligent and adaptive, accurately retaining potential creatives and eliminating inferior ones, thus avoiding budget waste. Furthermore, through automated clustering, filtering, and timestamp-priority-based rule execution, an end-to-end intelligent cleanup process is implemented, significantly improving the efficiency of creative management for large-scale advertising accounts, reducing manual operating costs, and effectively preventing a decline in advertising effectiveness caused by internal competition among homogeneous creatives. Attached Figure Description

[0058] Various other advantages and benefits will become apparent to those skilled in the art upon reading the following detailed description of preferred embodiments. The accompanying drawings are for illustrative purposes only and are not intended to limit the invention. Furthermore, the same reference numerals denote the same parts throughout the drawings. In the drawings:

[0059] Figure 1This is a schematic diagram of the electronic device structure of the hardware operating environment involved in the embodiments of the present invention;

[0060] Figure 2 This is a flowchart illustrating the steps of a method for cleaning up homogeneous advertising materials in e-commerce based on multimodal learning, as provided in an embodiment of the present invention.

[0061] Figure 3 This is a schematic diagram of the functional units of an e-commerce homogenized advertising material cleaning device based on multimodal learning provided in an embodiment of the present invention. Detailed Implementation

[0062] To more clearly illustrate the technical solutions in the embodiments of the invention or the prior art, the invention will be briefly introduced below in conjunction with the accompanying drawings and descriptions of the embodiments or the prior art. Obviously, the following description of the structure of the drawings is merely some embodiments of the invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort. It should be noted that the description of these embodiments is for the purpose of aiding understanding the invention, but does not constitute a limitation on the invention. The solution of the present invention will be further described below in conjunction with the accompanying drawings.

[0063] Reference Figure 1 This is a schematic diagram of the electronic device structure of the hardware operating environment involved in the embodiments of the present invention.

[0064] like Figure 1 As shown, the electronic device may include: a processor 1001, such as a central processing unit (CPU), a communication bus 1002, a user interface 1003, a network interface 1004, and a memory 1005. The communication bus 1002 is used to enable communication between these components. The user interface 1003 may include a display screen or an input unit such as a keyboard; optionally, the user interface 1003 may also include a standard wired interface or a wireless interface. The network interface 1004 may optionally include a standard wired interface or a wireless interface (such as a Wi-Fi interface). The memory 1005 may be a high-speed random access memory (RAM) or a stable non-volatile memory (NVM), such as a disk drive. The memory 1005 may also optionally be a storage device independent of the aforementioned processor 1001.

[0065] Those skilled in the art will understand that Figure 1The structure shown does not constitute a limitation on the electronic device and may include more or fewer components than shown, or combine certain components, or have different component arrangements.

[0066] like Figure 1 As shown, the memory 1005, as a storage medium, may include an operating device, a data storage module, a network communication module, a user interface module, and an electronic program. In the illustrated electronic device, the network interface 1004 is mainly used for data communication with a network server; the user interface 1003 is mainly used for data interaction with the user; the processor 1001 and memory 1005 in the electronic device of this invention can be disposed within the electronic device. The electronic device uses the processor 1001 to call the e-commerce homogenized advertising material cleaning device based on multimodal learning stored in the memory 1005, and executes the e-commerce homogenized advertising material cleaning method based on multimodal learning provided in this embodiment of the invention.

[0067] Reference Figure 2 This invention provides a method for cleaning up homogeneous advertising materials in e-commerce based on multimodal learning. This method aims to accurately identify and clean up homogeneous and inefficient advertising materials on e-commerce advertising platforms through deep learning models and automated processes, thereby improving advertising efficiency and account management. The specific process of this method includes the following steps:

[0068] S101, Obtain multiple pending ad creatives belonging to at least one ad account;

[0069] This step aims to collect advertising creatives from one or more advertisers' advertising accounts that require analysis and cleansing. These creatives are the basic units that make up advertising campaigns and typically exist in the form of images, videos, or a combination of text and images.

[0070] In a specific implementation, this method interacts with the backend server of the e-commerce advertising platform through a data interface module.

[0071] S1011 first receives at least one advertising account ID specified by the user through the management interface. This management interface supports multiple account selection, allowing operations personnel to manage and clean up materials under multiple accounts in a unified manner at one time.

[0072] S1012: Based on the selected account ID, a data request is initiated to the platform server by calling the Application Programming Interface (API) provided by the advertising platform to obtain a list of ad creatives currently running under the specified account. To ensure timely processing, preliminary filtering can be performed based on the creative creation time range set in the cleanup rules. For example, users can select creatives created within the "previous 30 days," "previous 60 days," or "previous 90 days" as the processing targets, and the corresponding timestamp filtering parameter will be added to the API request.

[0073] S1013 receives the media data returned by the platform server. This data typically includes a unique identifier for each media (media ID), media file (such as image URLs, video files), associated text information (such as advertising copy, titles), and other metadata (such as media format, resolution, duration, etc.). This raw media data is cached in local storage or memory for subsequent processing.

[0074] S102. For each advertising material, extract the image data, text data and audio data contained therein; use a visual language model pre-trained by contrastive learning to map the image data and text data to a unified semantic embedding space to generate a joint visual text feature vector, and use an audio fingerprint model to extract audio feature vectors, thereby obtaining the multimodal feature vector of each advertising material.

[0075] The core of this step lies in transforming unstructured multimedia materials into structured, computable numerical feature vectors, laying the foundation for subsequent similarity calculations and cluster analysis. This process comprehensively utilizes information from visual, textual, and audio modalities to achieve a holistic understanding of the advertising material's content.

[0076] S1021, for each advertising creative to be processed, modal separation is performed first. If the creative is video, a series of keyframe images are extracted at a preset frame rate (e.g., 1 frame per second) using video decoding technology, and the complete audio track file is separated. If the creative is a still image, the image data is directly obtained. At the same time, text data such as advertising copy and title associated with the creative are extracted.

[0077] S1022 employs a Vision-Language Model (VLM) pre-trained using Contrastive Learning to process image and text data. Models like CLIP (Contrastive Language-Image Pre-training) have the core advantage of learning deep semantic relationships between images and text and mapping them to a unified, high-dimensional semantic embedding space.

[0078] In a preferred embodiment, the visual language model includes an image encoder and a text encoder.

[0079] Image encoders can employ a Vision Transformer (ViT) architecture. ViT first segments the input image into a series of fixed-size patches, then linearly embeds these patches into a vector sequence, adding positional encoding information. This sequence is then fed into a standard Transformer encoder, which uses multi-head self-attention to capture the global dependencies between the patches, ultimately outputting a feature vector that represents the semantics of the entire image—the visual feature vector. Text encoders can employ a Transformer-based BERT (Bidirectional Encoder Representations from Transformers) architecture. BERT tokenizes the input text sequence (advertising copy) and converts it into a combination of word embeddings, positional embeddings, and paragraph embeddings. This embedding sequence is processed by a multi-layer bidirectional Transformer encoder, enabling a deep understanding of the text's context and ultimately generating a feature vector representing the overall semantics of the text—the text feature vector.

[0080] S1023, the pre-training process of this visual language model is crucial. The model is trained on a large-scale dataset containing hundreds of millions of image-text pairs. The image encoder and the text encoder are pre-trained using a contrastive learning loss function to maximize the cosine similarity of the feature vectors of matching image-text pairs and minimize the cosine similarity of the feature vectors of mismatched image-text pairs. Through this contrastive learning approach, the model learns to bring semantically related images and texts closer together in the embedding space and push unrelated ones further apart, thereby constructing a unified semantic space across modalities. In this step, the extracted image data is input into the image encoder to obtain visual feature vectors, and the text data is input into the text encoder to obtain text feature vectors. Since both are in the same semantic space, they together constitute a joint visual-text feature vector.

[0081] S1024 utilizes an audio fingerprint model to process audio data extracted from video footage. Audio fingerprinting technology aims to generate a compact and robust digital signature for audio segments, ensuring that the fingerprint can still be effectively identified even after audio has undergone compression, noise interference, or slight speed changes.

[0082] In a preferred embodiment, the audio fingerprint model generates the audio feature vector by extracting Mel-Frequency Cepstral Coefficients (MFCCs) from the audio data of each advertising creative. MFCCs are a widely used feature in speech recognition and audio processing, effectively simulating the nonlinear perceptual characteristics of the human ear to sound frequencies. The extraction process typically includes the following steps:

[0083] Pre-emphasis is achieved by using a high-pass filter to boost the energy of the high-frequency components in order to balance the spectrum.

[0084] Framing and windowing divide the audio signal into a series of short frames (e.g., 20-40 milliseconds) and apply a window function (e.g., Hamming window) to each frame to reduce spectral leakage.

[0085] The Fast Fourier Transform (FFT) performs an FFT on each frame, transforming it from the time domain to the frequency domain to obtain the spectrum.

[0086] Mel filtering passes the spectrum through a set of triangular filters (Mel filter bank) whose center frequencies are evenly distributed on the Mel scale, thus simulating the frequency perception of the human ear.

[0087] Logarithmic energy calculation: Calculate the logarithmic energy of the output of each filter.

[0088] The Discrete Cosine Transform (DCT) is applied to the logarithmic energy spectrum to obtain a set of coefficients, known as the MFCC. Typically, the first 12-13 coefficients are selected as the features of that frame. Finally, the MFCC vectors of all frames are aggregated (e.g., through averaging or concatenation) to form an audio feature vector representing the entire audio content.

[0089] S1025, the joint visual text feature vector generated in the previous steps is concatenated or weighted and fused with the audio feature vector to form the final multimodal feature vector of each advertising creative. This vector comprehensively represents the core content of the advertising creative in the three dimensions of visual, textual, and auditory.

[0090] S103 inputs the multimodal feature vectors into a pre-trained deep multimodal attitude quantization learning model to calculate the homogeneity similarity score between any two ad creatives in a metric space optimized specifically for measuring creative similarity; and uses a hierarchical density clustering algorithm that can automatically determine the number of clusters and identify noise points to cluster all ad creatives, and determines the creatives belonging to the same cluster in the clustering results as a group of homogeneous creatives;

[0091] This step aims to measure and group all the advertising creatives to be processed based on the multimodal feature vectors extracted in S102, in order to find a set of homogeneous creatives that are highly similar in content.

[0092] S1031 first utilizes a pre-trained Deep Multimodal Metric Learning model to compute the similarity between materials. Unlike directly using general metrics such as cosine similarity, deep metric learning aims to learn a specific non-linear mapping that projects the original feature vectors into a new metric space. In this space, semantically similar materials are closer in Euclidean distance, while semantically dissimilar materials are farther apart. This model typically employs architectures such as Siamese Networks or Triplet Networks and is trained using contrastive loss or triplet loss to optimize this metric space.

[0093] In a preferred embodiment, calculate any two advertising creatives and Homogeneity similarity score between The steps are implemented using the following weighted fusion formula:

[0094]

[0095] in: For material and The final homogeneity similarity score between them; The materials are respectively The visual feature vector output by the image encoder described in S102; The materials are respectively The text feature vector output by the text encoder described in S102; The materials are respectively The audio feature vector output by the audio fingerprint model described in S102; This is the function for calculating cosine similarity. The weighting coefficients for each mode satisfy the following conditions: ,and The weighting coefficients are determined by a pre-trained attention network based on the advertising campaign type to which the material belongs. Dynamically generated: ,in This is an embedding function used to map advertising campaign types to embedding vectors. This attention mechanism can dynamically adjust the importance of visual, textual, and audio information in similarity calculation based on the specific type of advertising campaign (e.g., brand promotion, sales promotion, etc.), thereby achieving more accurate homogeneity judgment.

[0096] S1032, after obtaining the homogeneity similarity scores among all material pairs, employs a hierarchical density clustering algorithm that can automatically determine the number of clusters and identify noise points, specifically HDBSCAN (Hierarchical Density-Based Spatial Clustering of Applications with Noise). Compared to traditional clustering algorithms, HDBSCAN exhibits better robustness to clusters and noise points of varying densities.

[0097] In a preferred embodiment, the execution flow of the hierarchical density clustering algorithm is as follows:

[0098] Based on the homogeneity similarity scores, a mutual reachability graph is constructed for all ad creatives. First, based on the homogeneity similarity scores (which can be converted into distances) between creatives, the mutual reachability distance between all data points (i.e., creatives) is calculated. This distance considers not only the direct distance between two points but also incorporates the concept of local density, making the algorithm insensitive to noise. A minimum spanning tree (MST) is then constructed based on the mutual reachability graph. Treating all creatives as nodes in the graph and the mutual reachability distance as the edge weights, a minimum spanning tree connecting all nodes is built. The MST effectively captures the inherent density structure of the data.

[0099] Based on the minimum spanning tree, a clustering hierarchy containing all data points can be constructed: by sorting the edges of the MST in ascending order of weight and removing the edges in sequence, a clustering hierarchy containing all data points can be constructed, also known as a dendrogram.

[0100] The clustering hierarchy is pruned according to a preset minimum cluster size to obtain a simplified cluster tree: the hierarchy is pruned according to a preset minimum cluster size parameter. If, after a cluster splits, the size of its sub-clusters is smaller than the preset value, then these points are considered to have "fallen" from the original cluster, rather than forming new independent clusters.

[0101] The final homogeneous material group is extracted based on the stability of each cluster in the cluster tree, and materials not assigned to any cluster are identified as non-homogeneous noise points. Finally, the final homogeneous material group is extracted based on the "stability" of each cluster in the cluster tree (i.e., the duration of cluster existence under different density thresholds). Clusters with high stability are considered valid clustering results. Materials not included in any stable cluster are identified as non-homogeneous noise points.

[0102] S1033, After clustering, all ad creatives belonging to the same cluster are determined to be a group of homogeneous creatives. A unique group ID is assigned to each group, and the results are stored for use in subsequent filtering steps.

[0103] S104, within each homogeneous creative group, obtain the performance data of each creative within the homogeneous creative group during a preset data period. The performance data includes spending amount and return on investment. Based on the statistical analysis of the historical performance data of advertising creatives of the same category for at least one advertising account within a historical time window, dynamically determine a dynamic performance threshold for filtering. From each homogeneous creative group, select creatives with performance data lower than the dynamic performance threshold as creatives to be cleaned up.

[0104] This step aims to filter out underperforming creatives from the identified homogeneous creative groups based on actual ad performance, targeting them for cleanup. This process employs dynamic thresholds to adapt to performance fluctuations across different product categories and market environments.

[0105] S1041, for each homogeneous creative group identified in S103, iterate through each creative within the group. Using the creative ID, call the advertising platform's API again to query the performance data of these creatives within a preset data period. This data period can be configured by the user in the cleanup rules, such as "first 7 days," "first 30 days," etc. Key performance metrics include:

[0106] Cost: refers to the total advertising budget spent on this material within a specified period.

[0107] Return on Investment (ROI): This refers to the ratio of transaction amount to expenditure amount, and is a core indicator for measuring the effectiveness of advertising.

[0108] S1042, dynamically determine a performance threshold for filtering inefficient content. Traditional methods using fixed thresholds for filtering cannot adapt to differences in different product categories and different campaign stages. Therefore, this embodiment employs an adaptive threshold algorithm based on historical data distribution.

[0109] In a preferred embodiment, the steps for determining the dynamic performance threshold are as follows:

[0110] Collect historical performance data of advertising creatives of the same category under at least one advertising account within the past N preset time windows, and construct the time series distribution of the return on investment indicator;

[0111] An adaptive thresholding algorithm based on historical data distribution is employed. A preset quantile of the time series distribution, or a value lower than a preset multiple of the mean standard deviation, is set as the current dynamic performance threshold. The preset quantile ranges from the 10th to the 30th quantile. This method ensures that the threshold automatically adjusts with fluctuations in the overall performance of the category; categories with generally better performance will have relatively higher thresholds, and vice versa.

[0112] S1043 compares the actual performance data (such as ROI) of each material within each homogeneous material group with the dynamic performance threshold calculated in S1042. If the ROI of a material is lower than the dynamic threshold, the material is marked as "material to be cleaned up".

[0113] S1044, in a preferred embodiment, after obtaining the multimodal feature vector of each advertising creative, the quality, efficiency and source of a single creative can also be evaluated in parallel to expand the scope of creatives to be cleaned.

[0114] Low-quality material identification:

[0115] S10441 utilizes a No-Reference Image Quality Assessment (NR-IQA) model based on a Convolutional Neural Network (CNN) to score the quality of images or video keyframes in the advertising material. This model is trained on a large amount of image data with labeled quality levels and can automatically evaluate the aesthetic and technical quality of images.

[0116] S10442, the blurriness of the image or video keyframe is quantified by applying the Laplacian operator to the keyframe and calculating the variance of its response. As a second-derivative operator, the Laplacian operator effectively highlights edges and areas of rapid intensity change in an image. The lower the variance of its response, the less edge information is in the image, and the more blurred the image.

[0117] S10443, mark advertising creatives with a quality score lower than the first preset quality threshold or a fuzziness variance lower than the second preset quality threshold as low-quality creatives and include them in the scope of creatives to be cleaned.

[0118] Inefficient material identification:

[0119] S10444, the multimodal feature vector of the advertising creative, along with user profile features and contextual features, are input into a pre-trained multimodal click-through rate (CTR) prediction model to obtain the predicted CTR value for each advertising creative. User profile features may include age, gender, region, etc.; contextual features may include delivery time, device type, etc.

[0120] S10445, Ad creatives with predicted click-through rates below a dynamic efficiency threshold are marked as inefficient creatives and included in the list of creatives to be cleaned up.

[0121] Material Removal Recognition:

[0122] S10446, Extract the image or video keyframes of the advertising material;

[0123] S10447, the keyframe is retrieved from a public network media asset library using a reverse image search application programming interface (API) that is robust to image transformations, wherein the image transformations include cropping, rotation, scaling, brightness adjustment, contrast adjustment, and watermarking. This retrieval technique is typically based on the Perceptual Hashing (pHash) algorithm, which is robust to common image modifications.

[0124] S10448, if the search results contain third-party materials whose publication time is earlier than the current advertising material and whose perceptual hash similarity with the keyframe is higher than a reposting judgment threshold, then the current advertising material is marked as a reposting material and included in the scope of materials to be cleaned up.

[0125] S105, responding to cleanup commands, when there are multiple cleanup rules under the same advertising account, determines the priority based on the last update timestamp of the cleanup rules, and automatically performs deletion operations on the cleanup materials selected by the rule with the highest priority.

[0126] This step is the end point of the entire process and is responsible for performing the actual cleanup actions.

[0127] S1051, the cleanup operation can be initiated manually by the user or automatically triggered by a preset scheduled task (such as every morning). When the cleanup command is responded to, the cleanup process begins.

[0128] S1052 addresses potential rule conflicts. An ad account may have multiple cleanup rules configured. To ensure that the executed strategy aligns with the latest intentions of the operations team, a timestamp-based prioritization mechanism is employed.

[0129] In a preferred embodiment, the priority determination step is as follows:

[0130] Before performing the deletion operation, all enabled cleanup rules applicable to the advertising account are iterated through;

[0131] Extract the last modification timestamp of each of the enabled cleanup rules;

[0132] Set the cleanup rule with the latest timestamp as the only valid rule to be executed at present, and ignore the rest of the rules until the next cleanup task is triggered.

[0133] S1053, based on the list of creatives to be cleaned determined by the highest priority rule, a deletion command is sent to the platform server via the advertising platform's API. This command typically contains a list of creative IDs to be deleted. After receiving the command and verifying permissions, the platform server will perform the deletion operation, such as removing the creative from the creative library and automatically unlinking it from all advertising campaigns. This concludes a complete process of cleaning up homogenized advertising creatives.

[0134] Through the above embodiments, the method provided by the present invention can automate and intelligently complete the entire process from material acquisition, multimodal feature extraction, homogeneous clustering, dynamic threshold screening to final cleanup, significantly improving the efficiency and accuracy of e-commerce advertising material management.

[0135] This invention also provides a device for cleaning up homogeneous e-commerce advertising materials based on multimodal learning, referring to... Figure 3 The diagram shows a functional unit diagram of an e-commerce homogeneous advertising material cleanup device 300 based on multimodal learning according to the present invention. The device may include the following units:

[0136] The acquisition unit 301 is used to acquire multiple unprocessed advertising materials belonging to at least one advertising account;

[0137] The feature extraction unit 302 is used to extract the image data, text data and audio data contained in each advertising material; using a visual language model pre-trained by contrastive learning, the image data and text data are mapped to a unified semantic embedding space to generate a joint visual text feature vector, and the audio fingerprint model is used to extract the audio feature vector, thereby obtaining the multimodal feature vector of each advertising material.

[0138] The homogenization grouping unit 303 is used to input multimodal feature vectors into a pre-trained deep multimodal attitude quantization learning model to calculate the homogenization similarity score between any two advertising materials in a metric space optimized for measuring material similarity; and to use a hierarchical density clustering algorithm that can automatically determine the number of clusters and identify noise points to cluster all advertising materials, and to determine materials belonging to the same cluster in the clustering results as a group of homogenized materials;

[0139] The creative material filtering unit 304 is used to obtain the performance data of each creative material in each homogeneous creative material group within a preset data period. The performance data includes spending amount and return on investment. Based on the statistical analysis of the historical performance data of advertising creative materials of the same category for at least one advertising account within a historical time window, a dynamic performance threshold for filtering is dynamically determined. From each homogeneous creative material group, creative materials with performance data lower than the dynamic performance threshold are selected as creative materials to be cleaned up.

[0140] The execution unit 305 is used to determine the priority based on the last update timestamp of the cleanup rules when there are multiple cleanup rules under the same advertising account, and automatically perform the deletion operation on the cleanup materials selected by the rule with the highest priority.

[0141] Based on the same inventive concept, another embodiment of the present invention provides an electronic device, including a processor, a communication interface, a memory, and a communication bus, wherein the processor, the communication interface, and the memory communicate with each other through the communication bus.

[0142] Memory, used to store computer programs;

[0143] When a processor executes a program stored in memory, it implements the multimodal learning-based method for cleaning up homogeneous e-commerce advertising materials of the present invention.

[0144] The communication bus mentioned above can be a Peripheral Component Interconnect (PCI) bus or an Extended Industry Standard Architecture (EISA) bus, etc. This communication bus can be divided into address bus, data bus, control bus, etc. For ease of illustration, only one thick line is used in the diagram, but this does not indicate that there is only one bus or one type of bus. The communication interface is used for communication between the aforementioned terminal and other devices. The memory can include Random Access Memory (RAM), or non-volatile memory, such as at least one disk storage device. Optionally, the memory can also be at least one storage device located remotely from the aforementioned processor.

[0145] The processors mentioned above can be general-purpose processors, including central processing units (CPUs), network processors (NPs), etc.; they can also be digital signal processors (DSPs), application-specific integrated circuits (ASICs), field-programmable gate arrays (FPGAs), or other programmable logic devices, discrete gate or transistor logic devices, or discrete hardware components.

[0146] Furthermore, to achieve the above objectives, embodiments of the present invention also propose a computer-readable storage medium storing a computer program, which, when executed by a processor, implements the e-commerce homogenized advertising material cleanup method based on multimodal learning, as described in the embodiments of the present invention.

[0147] Those skilled in the art will understand that embodiments of the present invention can be provided as methods, apparatus, or computer program products. Therefore, embodiments of the present invention can take the form of entirely hardware embodiments, entirely software embodiments, or embodiments combining software and hardware aspects. Furthermore, embodiments of the present invention can take the form of computer program products implemented on one or more computer-usable hardware devices (including, but not limited to, disk storage, CD-ROM, optical storage, etc.) containing computer-usable program code.

[0148] The embodiments of the present invention are described with reference to flowchart illustrations and / or block diagrams of methods, terminal devices (apparatus), and computer program products according to embodiments of the invention. It should be understood that each block of the flowchart illustrations and / or block diagrams, and combinations of blocks in the flowchart illustrations and / or block diagrams, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, special-purpose computer, embedded processor, or other programmable data processing terminal device to produce a machine, such that the instructions, which execute via the processor of the computer or other programmable data processing terminal device, generate instructions for implementing the flowchart illustrations and / or block diagrams. Figure 1 One or more processes and / or boxes Figure 1 A device that provides the functions specified in one or more boxes.

[0149] These computer program instructions may also be stored in a computer-readable storage medium that can direct a computer or other programmable data processing terminal device to operate in a particular manner, such that the instructions stored in the computer-readable storage medium produce an article of manufacture including instruction means, which are implemented in a process Figure 1 One or more processes and / or boxes Figure 1 The function specified in one or more boxes.

[0150] These computer program instructions can also be loaded onto a computer or other programmable data processing terminal equipment, causing a series of operational steps to be performed on the computer or other programmable terminal equipment to produce a computer-implemented process, thereby providing instructions that execute on the computer or other programmable terminal equipment for implementing the process. Figure 1 One or more processes and / or boxes Figure 1 The steps of the function specified in one or more boxes.

[0151] Finally, it should be noted that in this document, relational terms such as "first" and "second" are used only to distinguish one entity or operation from another, and do not necessarily require or imply any such actual relationship or order between these entities or operations. "And / or" indicates that either one or both can be chosen. Furthermore, the terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or terminal device that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such a process, method, article, or terminal device. Without further limitations, an element defined by the phrase "comprising one..." does not exclude the presence of other identical elements in the process, method, article, or terminal device that includes the element.

[0152] The above are merely specific embodiments of the present invention, but the scope of protection of the present invention is not limited thereto. Any person skilled in the art can easily conceive of various equivalent modifications or substitutions within the technical scope disclosed in the present invention, and these modifications or substitutions should all be covered within the scope of protection of the present invention. Therefore, the scope of protection of the present invention should be determined by the scope of the claims.

Claims

1. A method for cleaning up homogeneous advertising materials in e-commerce based on multimodal learning, characterized in that, The method includes: Retrieve multiple pending ad creatives belonging to at least one ad account; For each of the aforementioned advertising materials, the image data, text data, and audio data contained therein are extracted; using a visual language model pre-trained by contrastive learning, the image data and the text data are mapped to a unified semantic embedding space to generate a joint visual text feature vector, and the audio feature vector is extracted using an audio fingerprint model, thereby obtaining the multimodal feature vector of each advertising material. The multimodal feature vectors are input into a pre-trained deep multimodal attitude quantization learning model to calculate the homogeneity similarity score between any two advertising materials in a metric space optimized for measuring material similarity. A hierarchical density clustering algorithm that can automatically determine the number of clusters and identify noise points is used to cluster all advertising materials, and materials belonging to the same cluster in the clustering results are judged as a group of homogeneous materials. Within each homogenized creative group, the performance data of each creative within the homogenized creative group is obtained within a preset data period, including spending amount and return on investment; based on statistical analysis of the historical performance data of the at least one advertising account's advertising creatives of the same category within a historical time window, a dynamic performance threshold for filtering is dynamically determined; from each homogenized creative group, creatives with performance data lower than the dynamic performance threshold are selected as creatives to be cleaned up; In response to a cleanup command, when multiple cleanup rules exist under the same advertising account, the priority is determined based on the last update timestamp of the cleanup rules, and the deletion operation is automatically performed on the materials to be cleaned selected by the rule with the highest priority.

2. The method according to claim 1, characterized in that, The steps of generating joint visual text feature vectors using a contrastive learning pre-trained visual language model and extracting audio feature vectors using an audio fingerprint model include: The visual language model includes an image encoder and a text encoder, which are pre-trained using a contrastive learning loss function to maximize the cosine similarity of the feature vectors of matching image-text pairs and minimize the cosine similarity of the feature vectors of mismatched image-text pairs. The audio fingerprint model generates the audio feature vector by extracting the Mel frequency cepstral coefficients from the audio data of each advertising material.

3. The method according to claim 1, characterized in that, The step of clustering all advertising materials using a hierarchical density clustering algorithm that can automatically determine the number of clusters and identify noise points includes: Based on the homogeneity similarity score, construct a distance map of all advertising creatives; Construct a minimum spanning tree based on the interconnection distance graph; Construct a clustering hierarchy containing all data points based on the minimum spanning tree; The clustering hierarchy is pruned according to the preset minimum cluster size to obtain a simplified cluster tree; Based on the stability of each cluster in the cluster tree, the final homogenized material group is extracted, and the material that is not assigned to any cluster is identified as non-homogeneous noise points.

4. The method according to claim 1, characterized in that, After the step of obtaining the multimodal feature vector of each advertising creative, the method further includes: A no-reference image quality assessment model based on a convolutional neural network is used to score the quality of images or video keyframes in the advertising material. The blurriness of the image or video keyframe is quantified by applying the Laplacian operator to the keyframe and calculating the variance of its response. Ad creatives with a quality score lower than the first preset quality threshold or a fuzziness variance lower than the second preset quality threshold will be marked as low-quality creatives and included in the list of creatives to be cleaned up.

5. The method according to claim 1, characterized in that, After the step of obtaining the multimodal feature vector of each advertising creative, the method further includes: The multimodal feature vector of the advertising material, along with user profile features and contextual environment features, are input into a pre-trained multimodal click-through rate prediction model to obtain the predicted click-through rate value for each advertising material. Ad creatives with predicted click-through rates below a dynamic efficiency threshold are marked as inefficient creatives and included in the list of creatives to be cleaned up.

6. The method according to claim 1, characterized in that, After the step of obtaining the multimodal feature vector of each advertising creative, the method further includes: Extract keyframes from the images or videos of the advertising material; The keyframes are retrieved from a public network media asset library through a reverse image search application programming interface that is robust to image transformations, wherein the image transformations include cropping, rotation, scaling, brightness adjustment, contrast adjustment, and watermarking. If the search results contain third-party materials whose publication time is earlier than the current advertising material and whose perceptual hash similarity with the keyframe is higher than a reposting threshold, then the current advertising material will be marked as a reposted material and included in the scope of materials to be cleaned up.

7. The method according to claim 1, characterized in that, The step of dynamically determining a dynamic performance threshold for filtering based on statistical analysis of historical performance data of the at least one advertising account for the same category of advertising creatives within a historical time window includes: Collect historical performance data of advertising creatives of the same category under at least one advertising account within the past N preset time windows, and construct the time series distribution of the return on investment indicator; An adaptive threshold algorithm based on historical data distribution is adopted to set the preset quantile of the time series distribution or the value that is lower than the preset multiple standard deviation of the mean as the current dynamic performance threshold, wherein the preset quantile ranges from 10 to 30 quantiles.

8. The method according to claim 1, characterized in that, The step of calculating the homogeneity similarity score between any two advertising creatives is implemented using the following weighted fusion formula: in: For material and The final homogeneity similarity score between them; The materials are respectively The visual feature vector output by the image encoder; The materials are respectively The text feature vector output by the text encoder; The materials are respectively The audio feature vector output by the audio fingerprint model; This is the function for calculating cosine similarity. The weighting coefficients for each mode satisfy the following conditions: ,and The weighting coefficients are determined by a pre-trained attention network based on the advertising campaign type to which the material belongs. Dynamically generated: ,in This is an embedding function used to map advertising campaign types to embedding vectors.

9. The method according to claim 1, characterized in that, The step of determining priority based on the last update timestamp of the cleanup rule includes: Before performing the deletion operation, all enabled cleanup rules applicable to the advertising account are iterated through; Extract the last modification timestamp of each of the enabled cleanup rules; Set the cleanup rule with the latest timestamp as the only valid rule to be executed at present, and ignore the rest of the rules until the next cleanup task is triggered.

10. A device for cleaning up homogeneous advertising materials in e-commerce based on multimodal learning, characterized in that, include: The acquisition unit is used to acquire multiple ad creatives to be processed, belonging to at least one ad account; The feature extraction unit is used to extract image data, text data and audio data contained in each of the advertising materials; using a visual language model pre-trained by contrastive learning, the image data and the text data are mapped to a unified semantic embedding space to generate a joint visual text feature vector, and the audio feature vector is extracted using an audio fingerprint model, thereby obtaining the multimodal feature vector of each advertising material. The homogenization grouping unit is used to input the multimodal feature vector into a pre-trained deep multimodal attitude quantization learning model to calculate the homogenization similarity score between any two ad creatives in a metric space optimized for measuring creative similarity. A hierarchical density clustering algorithm that can automatically determine the number of clusters and identify noise points is used to cluster all advertising materials, and materials belonging to the same cluster in the clustering results are judged as a group of homogeneous materials. The material filtering unit is used to obtain the performance data of each material in each homogeneous material group within a preset data period, including spending amount and return on investment; based on statistical analysis of the historical performance data of the same category of advertising materials of the at least one advertising account within a historical time window, dynamically determine a dynamic performance threshold for filtering; and filter out materials with performance data lower than the dynamic performance threshold from each homogeneous material group as materials to be cleaned up. The execution unit is used to determine the priority of the cleanup rules based on the last update timestamp of the cleanup rules when there are multiple cleanup rules under the same advertising account, and automatically perform the deletion operation on the cleanup materials selected by the rule with the highest priority.

Images