A marking method and system based on advertising material data

Abstract

The present invention relates to the technical field of advertising material annotation, and in particular to a method and system for annotation based on advertising material data. The method comprises the following steps: acquiring advertising material video data; performing frame extraction processing according to the advertising material video data to obtain advertising material frame extraction data; performing texture feature extraction on the advertising material frame extraction data to obtain frame extraction texture feature data; performing information entropy calculation on the frame extraction texture feature data to obtain texture feature information entropy data; screening the advertising material frame extraction data according to the texture feature information entropy data to obtain advertising material frame extraction screening data; performing advertising element feature extraction on the advertising material frame extraction screening data to obtain advertising element feature data, so as to perform advertising material annotation operations. The present invention effectively improves the processing efficiency and annotation accuracy of advertising materials, and greatly reduces the time and cost required for manual intervention.

Description

Technical Field

[0001] The present invention relates to the technical field of advertising material annotation, and in particular to a method and system for annotating advertising material data. Background Art

[0002] With the rapid development of Internet technology, video advertising has been widely used in various digital media platforms as an important means of promotion. Advertising materials are usually displayed in the form of videos. Video ads attract users' attention through rich visual and auditory elements, thereby increasing brand awareness and product sales. However, existing advertising material annotation methods mostly rely on manual operations, which are inefficient and prone to inaccurate annotation. Especially when processing large-scale advertising video data, the workload of manual annotation is extremely large, which is difficult to meet the needs of real-time advertising delivery and intelligent advertising analysis. Summary of the invention

[0003] In order to solve the above technical problems, the present invention proposes a marking method and system based on advertising material data to solve at least one of the above technical problems.

[0004] The present application provides a labeling method based on advertising material data, the method comprising:

[0005] S1. Acquire video data of the advertising material; perform frame extraction processing according to the video data of the advertising material to obtain frame extraction data of the advertising material;

[0006] S2, extracting texture features from the frame-by-frame data of the advertising material to obtain frame-by-frame texture feature data; calculating information entropy on the frame-by-frame texture feature data to obtain texture feature information entropy data;

[0007] S3, filtering the advertising material frame extraction data according to the texture feature information entropy data to obtain the advertising material frame extraction filtering data;

[0008] S4. Extract advertising element features from the frame-selected data of the advertising material to obtain advertising element feature data for advertising material labeling.

[0009] In the present invention, long video content is decomposed into key frames through frame extraction processing, which can effectively reduce the amount of data processing and save storage and computing resources. The subtle visual features in the advertising material are captured by texture feature extraction technology, and the information complexity in the image is quantified by information entropy calculation. According to the entropy value of the texture feature, the key frames are screened out to ensure that only high-value, information-intensive frames are retained for subsequent processing. Through screening, the proportion of important content in the advertising material is increased, which helps to improve the accuracy and pertinence of annotation and reduce unnecessary data processing. Extract key elements in the advertising material, such as brand logos, products, text, etc., and improve the efficiency and accuracy of annotation through accurate identification of advertising element features. By extracting key elements in the advertisement, more accurate automatic annotation can be achieved, manual intervention can be reduced, annotation efficiency can be improved, and the degree of automation of advertising data analysis and management can be improved.

[0010] Optionally, the advertising material frame extraction data includes first advertising material frame extraction data, second advertising material frame extraction data and third advertising material frame extraction data, and S1 includes:

[0011] Get the video data of the creative;

[0012] Performing frame extraction processing at equal intervals according to the video data of the advertising material to obtain first advertising material frame extraction data;

[0013] Perform motion detection frame extraction according to the advertisement material video data to obtain second advertisement material frame extraction data;

[0014] Key frames are extracted according to the advertisement material video data to obtain third advertisement material frame data.

[0015] In the present invention, different types of frame information can be extracted from the advertising material video by combining equal-interval frame extraction, motion detection frame extraction and key frame extraction. Equal-interval frame extraction ensures the overall coverage of the video, motion detection frame extraction ensures the capture of dynamic changes, and key frame extraction focuses on the important content in the video. Equal-interval frame extraction can effectively avoid missing content due to too fast or too slow changes, and is suitable for those who need to analyze the entire video evenly, such as video timing analysis or overall coverage review of advertising materials. Motion detection frame extraction focuses on identifying important dynamic parts in advertisements, and is suitable for detecting rapidly changing advertising content, such as product demonstrations, character movements, etc. It can better capture scenes with greater visual impact and improve the pertinence of advertising analysis. Key frame extraction can reduce the number of frames processed while retaining the most important content. It is suitable for summary extraction or key point analysis of advertising videos, ensuring that the most informative parts can be highlighted in the material, such as brand logos, product close-ups, etc.

[0016] Optionally, the motion detection frame extraction is specifically:

[0017] Calculating the video content complexity of the advertisement material video data to obtain advertisement video content complexity data;

[0018] Performing motion detection on the advertisement material video data to obtain advertisement video content motion detection data;

[0019] Performing scene switching detection on the advertisement material video data to obtain scene switching detection data;

[0020] Calculate the inter-frame difference of the video data of the advertisement material according to the scene switching detection data to obtain the inter-frame difference data;

[0021] Calculate the frame extraction interval according to the advertisement video content complexity data, the advertisement video content motion detection data, the frame difference data and the scene switching detection data to obtain the frame extraction interval data;

[0022] The advertisement material video data is subjected to frame extraction processing according to the frame extraction interval data to obtain second advertisement material frame extraction data.

[0023] In the present invention, by calculating the complexity of the video content, the complexity of different areas in the video (such as multi-character scenes, fast switching, etc.) can be identified. Scenes with high content complexity often contain more information, so more detailed frame extraction is required. By detecting motion changes in the video, such as the movement of objects, the movements of characters, etc., key dynamic scenes can be identified. This helps to extract frames more accurately at important moments of motion and retain important details of the action. By detecting scene switching in the video, the starting and ending positions of different scenes can be identified. Scene switching often marks important turning points in advertising content, such as product display switching, character interaction, etc. By analyzing the changes between adjacent frames, subtle image or content changes in the frame can be identified, which helps to prioritize extraction between frames with large content changes and eliminate redundant frames with little change. Inter-frame difference calculation can accurately filter out frames with large amounts of information to ensure that subtle changes in the advertisement are not missed. The frame extraction interval calculation ensures the accuracy of frame selection, and automatically adjusts the frame extraction interval according to the content and motion conditions in different scenes, so that the density of the frame adapts to different video content, which not only reduces data redundancy, but also improves the efficiency of capturing key content.

[0024] Optionally, the motion detection includes:

[0025] Performing three-dimensional convolution calculation on the advertising material video data to obtain the advertising material video motion feature data;

[0026] Extracting local motion features from the video motion feature data of the advertisement material to obtain local motion feature data;

[0027] Perform motion feature clustering calculation on the local waybill feature data to obtain local motion feature clustering data;

[0028] Perform difference screening on the local motion feature data according to the local waybill feature clustering data to obtain local motion feature difference screening data;

[0029] Motion behavior labels are generated based on the local motion feature difference screening data and the local waybill feature clustering data corresponding to the local motion feature difference screening data to obtain the advertising video content motion detection data.

[0030] The three-dimensional convolution in the present invention not only captures the static features in the advertising material in space, but also captures the motion changes of objects in the time dimension, and extracts the motion features containing time and space information. By extracting motion features from the local area in the advertising video, the motion behavior of the specific area can be analyzed more finely, rather than just the average motion in the global range. Clustering calculation is performed on the extracted local motion features, and the regions with similar motion patterns can be grouped to identify different motion behavior categories. By screening the differences of the local motion features, those regions with significant motion changes can be identified, while small, irrelevant motions are ignored. By marking the features after the difference screening, generating motion behavior tags, different motion behaviors in the advertisement can be identified and classified. By screening the important change areas in the motion features and generating corresponding behavior tags, the system can carry out structured understanding and annotation of the motion content in the advertising material, which is suitable for the automatic processing and labeling management of the advertising material.

[0031] Optionally, the scene switching detection includes:

[0032] Perform frame convolution calculation on the advertising material video data to obtain advertising material frame convolution layer data;

[0033] Performing pooling calculation on the convolutional layer data of the advertising material frame to obtain the pooling layer data of the advertising material frame;

[0034] Perform clustering calculation on the advertising material frame pooling layer data to obtain advertising material frame clustering data;

[0035] Performing Gaussian distribution calculation according to the advertising material frame clustering data to obtain the advertising material frame Gaussian clustering data;

[0036] Performing frame structure similarity association on the advertising material frame pooling layer data according to the advertising material frame Gaussian clustering data to obtain advertising material frame structure similarity data;

[0037] Calculate pixel motion information based on the advertising material frame structure similarity data to obtain pixel motion information data;

[0038] Performing motion information clustering calculation according to pixel information motion data to obtain motion information clustering data;

[0039] Calculate the inter-frame motion difference according to the motion information clustering data to obtain the inter-frame motion difference data;

[0040] The scene switching is generated according to the inter-frame motion difference data to obtain the scene switching detection data.

[0041] In the present invention, frame convolution calculation is performed on the advertising material video, and high-level features such as image structure information such as edges and textures can be extracted from the original video frame. The generation of pooling layer data helps to reduce unnecessary redundant information, compress feature maps, and enhance the robustness of features, especially in the rapidly changing scenes in advertisements, it can more stably capture key change points. Clustering calculation is performed on the frame pooling layer data, which can effectively group similar frames and identify a series of frames with similar image features. Gaussian distribution calculation is performed on the clustered frames, which can more accurately describe the similarity or difference between frames, help distinguish frames of different categories, and at the same time, provide a precondition for the next step of structural similarity association. By similarity association based on frame Gaussian clustering data, the structural similarity of different frames is detected, and the relationship between frames is further analyzed. Based on the frame structure similarity, the pixel motion information in each frame is further calculated to detect motion changes in the advertising material, such as moving objects, scene conversion, etc. Motion information clustering ensures that different motion modes in advertisements are distinguished, such as distinguishing static product display scenes from fast-switching dynamic scenes, thereby improving the effect of scene switching detection. By calculating the motion differences between adjacent frames, frames with larger motion amplitudes can be identified.

[0042] Optionally, the frame difference calculation includes:

[0043] Calculate the difference of scene switching content and extract the feature of scene switching frequency according to the scene switching detection data and the video data of the advertising material, and obtain the scene switching content difference data and the scene switching frequency feature data;

[0044] Constructing a video frame scale according to the scene switching content difference data and the scene switching frequency feature data to obtain video frame scale data;

[0045] Extract frames at equal intervals according to the video data of the advertising material to obtain initial advertising material frame data;

[0046] According to the video frame scale data, an image pyramid is constructed for the initial advertising material frame data to obtain the advertising material image pyramid data;

[0047] Constructing a temporal dependency relationship according to the pyramid data of the advertising material image to obtain the temporal dependency relationship data of the advertising material image;

[0048] Extract color-texture fusion features based on the temporal dependency relationship data of the advertising material images to obtain color-texture fusion feature data;

[0049] The difference calculation is performed based on the color and texture fusion feature data to obtain the inter-frame difference data.

[0050] In the present invention, by performing differential analysis on the scene switching content in the advertising material, significant changes during scene switching can be captured, and these changes can be quantified to generate differential data. Feature extraction is performed on the scene switching frequency in the advertising video to help analyze the dynamic information such as the speed and rhythm of the switching in the advertisement. Through different scale analysis, the detailed changes from global to local in the advertising material can be captured to ensure the accuracy of the difference between frames. The image pyramid provides a multi-level feature expression from high resolution to low resolution, which not only retains the global information but also captures the local detail changes. The temporal dependency analysis can capture the continuous changes in the advertising material, ensure the temporal coherence between frames, help distinguish the gradual or sudden changes in different scenes, and improve the accurate detection of the difference between frames, which is particularly suitable for the gradual changes or transition scenes in the advertisement. By fusing and extracting the color information and texture information of the image, the changes in the visual appearance and structural characteristics of the image can be captured. The differential calculation can integrate the multi-dimensional information (such as color, texture, and timing) extracted previously, accurately detect the changes between frames, and improve the sensitivity to scene switching, dynamic changes, and subtle visual differences in the advertising material.

[0051] Optionally, S2 includes:

[0052] Divide the frame extraction data of the advertisement material into local areas to obtain frame extraction local area data;

[0053] Calculate the texture direction according to the local area data of the extracted frame to obtain the texture direction data of the local area of ​​the extracted frame;

[0054] Calculate the texture gradient according to the texture direction data of the local area of ​​the frame extraction to obtain the frame extraction texture feature data;

[0055] The information entropy of the extracted frame texture feature data is calculated to obtain texture feature information entropy data.

[0056] The present invention refines the processing of image content by dividing each frame of the advertising material into multiple local areas. The main direction of the texture is calculated in each local area, and the structural features of different parts of the image can be captured. By detecting the texture direction of the local area, the shape and boundary information of the object in the image can be better understood. Gradient calculation based on texture direction data can detect the degree of change of the texture in the image, including the clarity and direction change of the edge and contour. Gradient calculation provides quantitative data for the complexity of the texture in the image. By calculating the information entropy of the texture feature data, the information complexity in the local area can be quantified. Areas with high entropy values ​​usually represent complex image content and rich information; while areas with low entropy values ​​represent simple images or more repeated information. By comprehensively calculating the texture direction, gradient and information entropy of the local area, the changes between frames in the advertising material can be more accurately identified, including subtle scene transitions, movement of advertising elements, etc. Redundant frames and areas can be effectively ignored through entropy screening, thereby improving processing efficiency. The system can focus on high-value advertising content, reduce the waste of processing resources, and thus accelerate the overall analysis and annotation process.

[0057] Optionally, S3 includes:

[0058] According to the texture feature information entropy data, the advertisement material frame data is separated into high and low entropy values ​​to obtain the advertisement material high entropy frame data and the advertisement material low entropy frame data;

[0059] Sorting the high entropy frame data and the low entropy frame data of the advertising material by frame entropy values ​​to obtain high entropy frame sorting data and low entropy frame sorting data respectively;

[0060] Perform local entropy value fluctuation detection based on high entropy frame sorting data and low entropy frame sorting data to obtain high entropy frame local entropy value fluctuation data and low entropy frame local entropy value fluctuation data respectively;

[0061] The local entropy value fluctuation gradient is calculated for the local entropy value fluctuation data of the high entropy extraction frame and the local entropy value fluctuation data of the low entropy extraction frame, and the local entropy value fluctuation gradient data of the high entropy extraction frame and the local entropy value fluctuation gradient data of the low entropy extraction frame are obtained respectively;

[0062] The advertising material frame extraction data is screened using the high entropy frame extraction local entropy value fluctuation gradient data and the low entropy frame extraction local entropy value fluctuation gradient data to obtain first advertising material frame extraction screening data and second advertising material frame extraction screening data respectively;

[0063] The overlap degree is screened according to the first advertising material frame extraction and screening data and the second advertising material frame extraction and screening data to obtain the advertising material frame extraction and screening data.

[0064] In the present invention, the high and low entropy values ​​of the frames are separated according to the texture feature information entropy, and the frames with high information content (high entropy) can be distinguished from the frames with low information content (low entropy), ensuring that the high entropy and low entropy frames are analyzed in a targeted manner during processing. Frame entropy value sorting can help distinguish important frames from redundant frames, ensure that those frames with large information content can be processed first, improve the accuracy of frame selection, and are particularly suitable for capturing key scenes in advertising materials. The local detection method can capture the changes in the amount of information at the detail level of the frame, not only focusing on the overall features, but also identifying important local changes, ensuring a more detailed analysis of changes in advertising materials, and is suitable for detailed analysis of complex advertising scenes. Gradient calculation can more accurately identify subtle changes between frames, improve the detection ability of gradual scenes in advertising materials, and enable the system to capture changes that are not visually noticeable but are crucial to advertising content, such as the visual effect of fading in and out. Using the local gradient data of high entropy and low entropy frames to screen advertising materials, key information frames can be screened out from different angles to ensure that valuable frames can be extracted under different amounts of information. Overlap screening can effectively remove redundant frames with high similarity and repetitiveness, thereby reducing the interference of useless information, further optimizing the frame selection results, and improving the efficiency and quality of data processing.

[0065] Optionally, S4 includes:

[0066] Perform object detection on the frame-selected data of the advertising material to obtain preliminary object detection data;

[0067] Classify the advertising material type according to the preliminary object detection data to obtain advertising material type data;

[0068] Constructing spatial layout relationships according to the advertising material type data to obtain advertising material type spatial relationship data;

[0069] Feature vector conversion is performed according to the spatial relationship data of the advertising material type to obtain the advertising element feature data for the advertising material labeling operation.

[0070] In the present invention, object detection is performed on frame-by-frame screening data of advertising materials, and core objects in videos or images, such as brand logos, products, people, etc., can be identified. According to the object detection results, the types of advertising materials are classified to distinguish different advertising contents, such as product display ads, brand promotion ads, service ads, etc. The construction of spatial layout relationships can help the system understand the arrangement and interaction of different elements in the advertising scene, identify the key visual structures in the advertisements, and ensure a better understanding of the relative importance of elements during the advertisement annotation process. It is particularly suitable for the comprehensive display of brand logos, product locations and text descriptions in advertisements. Feature vector conversion can convert complex information in advertising materials into unified structured data, which is convenient for automated processing and analysis.

[0071] Optionally, the present application further provides a labeling system based on advertising material data, which is used to execute the labeling method based on advertising material data as described above. The labeling system based on advertising material data includes:

[0072] The advertising material video frame extraction module is used to obtain advertising material video data; perform frame extraction processing according to the advertising material video data to obtain advertising material frame extraction data;

[0073] The frame extraction texture feature extraction module is used to extract texture features from the frame extraction data of the advertising material to obtain the frame extraction texture feature data;

[0074] The texture feature information entropy calculation module is used to calculate the information entropy of the extracted frame texture feature data to obtain the texture feature information entropy data;

[0075] The advertising material frame extraction and screening module is used to screen the advertising material frame extraction data according to the texture feature information entropy data to obtain the advertising material frame extraction and screening data;

[0076] The advertisement element feature extraction module is used to extract advertisement element features from advertisement material frame extraction and screening data to obtain advertisement element feature data for advertisement material labeling.

[0077] The purpose of the present invention is that the advertising material is effectively divided into multiple key frames through frame extraction processing, which significantly reduces the redundant data processing requirements for the entire video. The calculation method of texture features and information entropy is combined, especially in step S2, through texture feature extraction and information entropy calculation, the complexity and information content of the frame can be accurately measured, the difference of the frame can be quantified, and the data basis can be provided for the subsequent screening step, which not only significantly improves the screening efficiency in large-scale material processing, but also ensures that the frames with key advertising elements are retained, and the waste of computing resources for processing useless frames is reduced. The frame screening mechanism in step S3 uses texture feature information entropy data to screen the frame data, and through high and low entropy value separation and sorting, the frames with large information content are screened out to ensure that key scenes and elements are retained. At the same time, local entropy value fluctuation detection and gradient calculation further refine the screening process, making the change trend of the frame clearer and eliminating redundant frames. In step S4, by extracting advertising element features from the screened advertising material frames, key advertising elements such as products, brand logos, text, characters, etc. are identified, and feature vectors are constructed according to spatial layout relationships. The feature extraction and spatial layout analysis of advertising elements can enhance the depth of annotation work and ensure that the relationship between advertising elements is accurately reflected. The present invention improves the processing efficiency and quality of advertising materials and is suitable for the automatic processing and intelligent analysis of large-scale advertising data. BRIEF DESCRIPTION OF THE DRAWINGS

[0078] Other features, objects and advantages of the present application will become more apparent by reading the detailed description of non-limiting implementations made with reference to the following drawings:

[0079] Figure 1 A flowchart showing a method for marking advertisement material data according to an embodiment of the present invention is provided;

[0080] Figure 2 A flowchart showing a method for extracting frames from a video of an advertisement material according to an embodiment of the present invention is provided;

[0081] Figure 3 A flowchart of a method for extracting texture features from a frame is shown in one embodiment;

[0082] Figure 4 A flowchart showing a method for selecting frames of advertisement materials according to an embodiment of the present invention is provided;

[0083] Figure 5 A flowchart showing a method for extracting features of advertising elements according to an embodiment of the present invention is provided;

[0084] The realization of the purpose, functional features and advantages of the present invention will be further explained in conjunction with embodiments and with reference to the accompanying drawings. DETAILED DESCRIPTION

[0085] The technical method of the present invention is described clearly and completely below in conjunction with the accompanying drawings. Obviously, the described embodiments are part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by technicians in this field without creative work are within the scope of protection of the present invention.

[0086] In addition, the drawings are only schematic illustrations of the present invention and are not necessarily drawn to scale. The functional entities can be implemented in software form, or in one or more hardware modules or integrated circuits, or in different network and / or processor methods and / or microcontroller methods.

[0087] It should be understood that, although the terms "first", "second", etc. may be used herein to describe various units, these units should not be limited by these terms. These terms are used only to distinguish one unit from another unit. For example, without departing from the scope of the exemplary embodiments, the first unit may be referred to as the second unit, and similarly the second unit may be referred to as the first unit. The term "and / or" used herein includes any and all combinations of one or more of the listed associated items.

[0088] 1. Technical implementation content

[0089] The advertising material automatically extracts video frames to generate advertising material frame data. Frame extraction is to reduce the amount of data while retaining key frame information, thereby avoiding the processing of redundant data. In practical applications, the combination of equal-interval frame extraction and scene switching detection has the best effect. Based on the frame extraction data, the texture features of each frame are extracted through advanced texture analysis technology (such as gray-level co-occurrence matrix, etc.), and the image complexity and information content of each frame are further evaluated through information entropy calculation. This technology ensures that frames with high information content can be retained in the subsequent frame screening process to avoid the interference of redundant data. Frame screening and sorting are performed based on information entropy data, and combined with the local entropy value fluctuation detection method, frames with small changes or irrelevant frames are eliminated, and the differences between frames are identified through gradient calculation. Practice has proved that the use of frame data after high and low entropy values ​​reduces the amount of processed data by about 50% while ensuring information density. Utilizing automated object detection and feature extraction technology, combined with spatial layout analysis, the core elements in the advertising material (such as brand logos, product close-ups, character images, etc.) are identified and labeled. Through feature vector conversion technology, the system can convert advertising elements into structured data for analysis.

[0090] It is recommended to use 30 FPS (frames per second) video material for processing, which can ensure that most dynamic changes in advertising materials are captured. The typical frame extraction interval is 1-2 frames per second (1 / 30 to 1 / 15 seconds), and the specific interval can be adjusted according to the dynamic degree of the advertising content. For advertisements with more dynamic changes, the frame extraction interval should be appropriately reduced to ensure full coverage of the frame. When using the image difference method for scene switching detection, the recommended difference threshold is 0.5-0.6. This parameter is based on the color change of pixels and the difference in image features, and is used to determine the scene switching point.

[0091] Use gray-level co-occurrence matrix (GLCM) to extract texture features. Set to 1-2 pixels to calculate the grayscale relationship between pixel pairs in the local area. The value can be set to 0°, 45°, 90°, 135°, representing the extraction of texture features in different directions. Mainly extract contrast, energy, homogeneity, etc. Use a 5×5 or 7×7 window for local texture calculation, which can be adjusted according to the resolution of the material.

[0092] The information entropy calculation is based on the local area of ​​texture features, and the recommended window size is 5×5 or 10×10 pixel area. When separating high and low entropy values ​​of frames, the recommended entropy threshold is between 0.7 and 0.9, and the threshold can be adjusted dynamically according to the complexity of the material. The fluctuation detection window is recommended to be 3×3 or 5×5 pixels in the local area to ensure that changes in the local area are captured. The fluctuation threshold in the local fluctuation detection between frames is recommended to be set to 0.1 to 0.3 to distinguish subtle changes from irrelevant changes.

[0093] The recommended number of pyramid layers is 3-4 for image processing and feature extraction at multiple resolutions, with the minimum layer being 1 / 4 of the original resolution. A 3×3 Gaussian filter is used to reduce the resolution.

[0094] Use deep learning object detection models, such as YOLOv5 or Faster R-CNN, to identify key elements in advertisements, such as brand logos, characters, and products. (Normalize images that do not meet the requirements.) The input image size is 640×640 pixels to balance accuracy and processing speed. Set the detection confidence threshold to 0.6-0.8 to ensure the accuracy of the detection results.

[0095] Use relative distance and size ratio to analyze the spatial layout of advertising elements. Perform layout analysis based on the distance between objects in the image, and the threshold is recommended to be set to 1 / 5 to 1 / 3 of the image width.

[0096] 2. Hardware and device configuration parameters

[0097] To ensure efficient operation of the automatic labeling system for creatives, the following are the device configuration parameters required by the system:

[0098] Server Hardware

[0099] CPU: It is recommended to use a high-performance processor, at least an Intel Xeon or AMD EPYC processor with 16 cores or more and a main frequency of 3.0 GHz or more to ensure parallel processing capabilities.

[0100] GPU: Deep learning models require a high-performance graphics processor. We recommend using a high-performance GPU such as NVIDIA Tesla V100 or NVIDIA A100, with at least 16 GB of video memory. For large-scale ad creative processing, we recommend configuring a multi-GPU cluster.

[0101] Memory: At least 128 GB RAM is recommended to support large-scale creative processing and running convolutional neural network models.

[0102] Storage device: NVMe SSD solid-state drives are recommended. The storage capacity is configured according to the scale of the creatives. At least 4 TB of storage space is recommended to ensure fast reading and writing of creatives and efficiency of model calculations.

[0103] Software Environment

[0104] Operating system: It is recommended to use a Linux system, such as Ubuntu 20.04 or CentOS, to efficiently manage and deploy large-scale parallel tasks.

[0105] Deep learning framework: Use TensorFlow or PyTorch framework, which is suitable for neural network training and reasoning. It is recommended to install the latest version of the deep learning library and ensure compatibility with CUDA and cuDNN to support GPU accelerated computing.

[0106] Image processing library: Use OpenCV library to perform video processing, frame extraction, image preprocessing and other operations for advertising materials.

[0107] Network requirements

[0108] Bandwidth: If you need to obtain creatives from an external server or perform real-time analysis, a network bandwidth of at least 1 Gbps is recommended.

[0109] Storage access: The storage system should support high throughput and low latency. You can consider using a distributed storage system such as Ceph or GlusterFS to ensure the collaborative processing capabilities of multiple nodes.

[0110] With the support of high-performance hardware, the system should be able to process more than 30 frames of advertising materials per second. The specific processing time will fluctuate depending on the complexity of the material content and the number of frames.

[0111] Through object detection models such as YOLOv5 or Faster R-CNN, the system's advertising element recognition accuracy should be maintained above 90%, and the annotation accuracy varies according to the complexity of the advertising scene. The processing time of a single advertising material from data loading to final annotation should be controlled within 5 minutes to meet the needs of rapid processing of a large number of advertising materials.

[0112] The system should be designed with good scalability: support large-scale distributed computing, so that processing power can be increased by adding GPU nodes when the amount of advertising material data increases dramatically. Support multiple advertising material formats, including MP4, MOV, JPEG, PNG, etc., to ensure that different types of advertising materials can be processed efficiently. Support flexible configuration of model parameters and data processing processes to meet the needs of different industries or advertising types.

[0113] See also Figures 1 to 5 The present application provides a method for labeling based on advertising material data, the method comprising:

[0114] S1. Acquire video data of the advertising material; perform frame extraction processing according to the video data of the advertising material to obtain frame extraction data of the advertising material;

[0115] Specifically, first extract the advertising video data from the advertising platform, video library or other advertising data sources to ensure that the video data format is compatible (such as MP4, AVI, etc.). By analyzing the frame rate per second of the video, extract static frame images from different time periods such as the middle frame to the key frame. The appropriate frame extraction frequency can be determined by calculating the frame time interval. Each frame after frame extraction is saved in a separate data set folder to form the frame extraction data of the advertising material.

[0116] S2, extracting texture features from the frame-by-frame data of the advertising material to obtain frame-by-frame texture feature data; calculating information entropy on the frame-by-frame texture feature data to obtain texture feature information entropy data;

[0117] Specifically, the pixel value analysis of the local area is performed for each frame of the image, the main texture area in the image is segmented, the gray level, pixel gradient and other information in the area are calculated, and the local texture features of the image are extracted. Each texture feature can be described by a set of numerical values, and the generated feature data contains multi-dimensional information such as position, size, and texture intensity;

[0118] After texture feature extraction, the information complexity of each local texture area is further analyzed. The specific process includes: normalizing the pixel values ​​of the texture area, calculating the probability distribution of each pixel value, and then generating the information entropy value of the area by summing the cumulative probability logarithm. Finally, a set of information entropy data corresponding to each local texture area is obtained to measure the complexity and significance of the texture.

[0119] S3, filtering the advertising material frame extraction data according to the texture feature information entropy data to obtain the advertising material frame extraction filtering data;

[0120] Specifically, a screening criterion is set based on the generated texture feature information entropy data, for example, a higher information entropy value represents a complex texture area, while a lower information entropy value represents a simple area. Therefore, a threshold can be set to screen frames with high information entropy as the focus of analysis.

[0121] The texture feature information entropy of each frame is compared according to the screening criteria, and those frames with information entropy exceeding the threshold are retained, and frames with lower information entropy are eliminated. The filtered advertising material frame extraction and screening data only include those frames with significant features, providing valid data for subsequent analysis.

[0122] S4. Extract advertising element features from the frame-selected data of the advertising material to obtain advertising element feature data for advertising material labeling.

[0123] Specifically, for each frame after screening, the advertising elements are detected and analyzed. First, the parts in the frame are segmented to identify possible advertising element areas, such as brand logos, texts, image bodies, etc. Then, the visual features of these advertising elements are further extracted, such as color distribution, shape contours, edge strength, etc., to generate feature data of the advertising elements.

[0124] According to the extracted advertising element features, matching the existing advertising element library or annotation rules, each advertising element is automatically labeled. These labels may include information such as the element category (such as logo, product, text), location (such as upper left corner, center), etc. After the annotation is completed, these data are combined with the frame extraction and screening data to generate an advertising material annotation data set containing annotation information.

[0125] In the present invention, long video content is decomposed into key frames through frame extraction processing, which can effectively reduce the amount of data processing and save storage and computing resources. The subtle visual features in the advertising material are captured by texture feature extraction technology, and the information complexity in the image is quantified by information entropy calculation. According to the entropy value of the texture feature, the key frames are screened out to ensure that only high-value, information-intensive frames are retained for subsequent processing. Through screening, the proportion of important content in the advertising material is increased, which helps to improve the accuracy and pertinence of annotation and reduce unnecessary data processing. Extract key elements in the advertising material, such as brand logos, products, text, etc., and improve the efficiency and accuracy of annotation through accurate identification of advertising element features. By extracting key elements in the advertisement, more accurate automatic annotation can be achieved, manual intervention can be reduced, annotation efficiency can be improved, and the degree of automation of advertising data analysis and management can be improved.

[0126] Optionally, the advertising material frame extraction data includes first advertising material frame extraction data, second advertising material frame extraction data and third advertising material frame extraction data, and S1 includes:

[0127] S11, obtaining advertising material video data;

[0128] Specifically, collect advertising videos from advertising databases, media libraries or network resources. Ensure the uniformity of video formats (such as MP4, AVI, etc.) and load them into the processing system. Each video material needs to be assigned a unique ID for subsequent tagging and tracking.

[0129] S12, performing frame extraction processing at equal intervals according to the advertising material video data to obtain first advertising material frame extraction data;

[0130] Specifically, based on the frame rate of the video (such as 30 frames per second), the timestamps at fixed intervals are calculated. For example, one frame is extracted every 1 second, or other intervals such as 0.5 seconds or 2 seconds are set to extract frames within a fixed time interval.

[0131] At each set timestamp, the corresponding frames are extracted from the video data. These frames are saved in the form of static images to form the first advertising material frame extraction data set. The extracted frames are numbered in sequence and associated with the original video file.

[0132] S13, performing motion detection frame extraction according to the advertisement material video data to obtain second advertisement material frame extraction data;

[0133] Specifically, the video is first analyzed frame by frame to identify the motion changes between frames. The difference between each frame and the frames before and after it can be calculated by pixel difference, and frames with large displacement or changes can be detected. This motion detection is used to capture the parts of the video that are moving significantly;

[0134] Based on the detected motion area, frames with significant motion changes are retained, and frames with static or less motion are eliminated. The extracted frames constitute the second creative frame data, which is mainly used to analyze dynamic elements in the video.

[0135] S14. Extract key frames according to the advertisement material video data to obtain third advertisement material frame data.

[0136] Specifically, by analyzing the video content, we extract frames that represent the main plot or content turning points of the video. We calculate the visual difference and content importance of each frame, and select those frames that are significantly different from surrounding frames or represent scene transitions.

[0137] Based on the changes in visual content, such as scene switching, significant changes in color histograms, or frames where text appears, key frames are determined. These extracted frames form the third creative frame data, focusing on important scenes and information in the video.

[0138] In the present invention, different types of frame information can be extracted from the advertising material video by combining equal-interval frame extraction, motion detection frame extraction and key frame extraction. Equal-interval frame extraction ensures the overall coverage of the video, motion detection frame extraction ensures the capture of dynamic changes, and key frame extraction focuses on the important content in the video. Equal-interval frame extraction can effectively avoid missing content due to too fast or too slow changes, and is suitable for those who need to analyze the entire video evenly, such as video timing analysis or overall coverage review of advertising materials. Motion detection frame extraction focuses on identifying important dynamic parts in advertisements, and is suitable for detecting rapidly changing advertising content, such as product demonstrations, character movements, etc. It can better capture scenes with greater visual impact and improve the pertinence of advertising analysis. Key frame extraction can reduce the number of frames processed while retaining the most important content. It is suitable for summary extraction or key point analysis of advertising videos, ensuring that the most informative parts can be highlighted in the material, such as brand logos, product close-ups, etc.

[0139] Optionally, the motion detection frame extraction is specifically:

[0140] Calculating the video content complexity of the advertisement material video data to obtain advertisement video content complexity data;

[0141] Specifically, for each frame in the advertising video, its color, texture, brightness and other characteristics are analyzed. First, the frame image is divided into multiple regions, and the pixel values ​​of each region are counted to calculate its grayscale value, texture change and edge strength. Then, using this local information, the complexity of the entire frame is accumulated to form the advertising video content complexity data. Frames with higher complexity have more image details and changes.

[0142] Performing motion detection on the advertisement material video data to obtain advertisement video content motion detection data;

[0143] Specifically, each frame of the video is compared with the previous frame at the pixel level to detect the pixel change rate between the two frames. For example, by calculating the pixel difference between the previous and next frames and analyzing the area with significant changes, the moving parts of the video can be identified;

[0144] Through frame-by-frame motion analysis, areas with changes exceeding a certain threshold are marked, which may contain moving objects or significant visual changes. This motion detection process generates motion detection data for advertising video content, marking the time and spatial location of the moving parts.

[0145] Performing scene switching detection on the advertisement material video data to obtain scene switching detection data;

[0146] Specifically, the global visual features between video frames are analyzed, such as color histogram, texture distribution, etc. These features tend to change significantly when the video scene switches. For example, when the frames of two different scenes differ greatly in image color or structure, the system can identify the frame as a scene switching point; once a scene switch is detected, the time point when the switch occurs and the corresponding frame position are recorded to generate scene switch detection data for subsequent frame difference calculations.

[0147] Calculate the inter-frame difference of the video data of the advertisement material according to the scene switching detection data to obtain the inter-frame difference data;

[0148] Specifically, based on the scene switching detection data, adjacent frame pairs are selected to perform frame-by-frame difference calculations on the image content. First, image features are extracted for each pair of frames to compare their differences in color, texture, shape, etc. For each pair of adjacent frames, the magnitude of the changes in these features is calculated to generate inter-frame difference data to reflect the content changes between frames.

[0149] Calculate the frame extraction interval according to the advertisement video content complexity data, the advertisement video content motion detection data, the frame difference data and the scene switching detection data to obtain the frame extraction interval data;

[0150] Specifically, the advertising video content complexity data, advertising video content motion detection data, frame difference data and scene switching detection data are integrated. According to the motion intensity, content complexity and scene switching of each time period, the frame extraction interval of each paragraph is calculated. For example, when the scene changes more or the motion detection is stronger, the frame extraction interval is reduced; and in the part with simpler content and less motion, the frame extraction interval is appropriately increased. Based on the comprehensive analysis results, the frame extraction interval of different time periods is calculated to generate frame extraction interval data. This data guides the system on how to efficiently extract frames in the video.

[0151] The advertisement material video data is subjected to frame extraction processing according to the frame extraction interval data to obtain second advertisement material frame extraction data.

[0152] Specifically, according to the frame extraction interval data, frames are extracted from different parts of the advertising video. A corresponding number of frames are extracted from each part according to the corresponding interval to obtain the frame extraction data of the second advertising material, which includes key frames with significant motion and large changes. These extracted frames are stored in chronological order and associated with the timestamp of the advertising video.

[0153] In the present invention, by calculating the complexity of the video content, the complexity of different areas in the video (such as multi-character scenes, fast switching, etc.) can be identified. Scenes with high content complexity often contain more information, so more detailed frame extraction is required. By detecting motion changes in the video, such as the movement of objects, the movements of characters, etc., key dynamic scenes can be identified. This helps to extract frames more accurately at important moments of motion and retain important details of the action. By detecting scene switching in the video, the starting and ending positions of different scenes can be identified. Scene switching often marks important turning points in advertising content, such as product display switching, character interaction, etc. By analyzing the changes between adjacent frames, subtle image or content changes in the frame can be identified, which helps to prioritize extraction between frames with large content changes and eliminate redundant frames with little change. Inter-frame difference calculation can accurately filter out frames with large amounts of information to ensure that subtle changes in the advertisement are not missed. The frame extraction interval calculation ensures the accuracy of frame selection, and automatically adjusts the frame extraction interval according to the content and motion conditions in different scenes, so that the density of the frame adapts to different video content, which not only reduces data redundancy, but also improves the efficiency of capturing key content.

[0154] Optionally, the motion detection includes:

[0155] Performing three-dimensional convolution calculation on the advertising material video data to obtain the advertising material video motion feature data;

[0156] Specifically, the advertisement video data contains spatial and temporal dimensions. First, the video data is preprocessed to segment each frame in the video into pixel blocks to form three-dimensional input data (including height, width and time dimensions).

[0157] By performing synchronous convolution operations on the spatial and temporal dimensions of the video frames, the motion changes of each pixel block are extracted frame by frame. The convolution calculation captures the change information of the image content between multiple frames and generates the preliminary motion features of the video. The result is a collection of motion features in the spatial and temporal dimensions of each time period, which constitutes the motion feature data of the advertising material video.

[0158] Extracting local motion features from the video motion feature data of the advertisement material to obtain local motion feature data;

[0159] Specifically, based on the motion feature data generated by three-dimensional convolution, local areas with large changes in the video frame are identified. These areas often represent significant moving parts, such as moving objects or rapidly changing backgrounds; for the identified local areas, fine-grained motion information is extracted. For example, parameters such as direction changes and speed changes in these local areas are analyzed. By decomposing and calculating these parameters, more detailed local motion feature data is obtained, which can describe the motion characteristics of each local area.

[0160] Perform motion feature clustering calculation on the local waybill feature data to obtain local motion feature clustering data;

[0161] Specifically, the extracted local motion feature data is organized according to the time and space dimensions to form a high-dimensional data set containing multiple features, such as the direction, speed, amplitude, etc. of the motion. Clustering is performed based on the similarity of the local motion features. For example, regions with similar motion directions and speeds are grouped into the same category to form a series of motion patterns. Each clustering result represents a motion type, and the generated local motion feature clustering data describes the distribution of motion behaviors in the advertising video.

[0162] Perform difference screening on the local motion feature data according to the local waybill feature clustering data to obtain local motion feature difference screening data;

[0163] Specifically, a motion feature difference threshold is set to identify motion regions that are different from most local motion features. The motion feature difference can be determined by calculating the distance between the local feature and its cluster center. The larger the distance, the more significant the difference. The local motion feature data is filtered according to the set threshold. Those regions that are significantly different from their cluster centers are retained. These regions represent special motion behaviors or unique visual changes. The filtered data is called local motion feature difference filtered data.

[0164] Motion behavior labels are generated based on the local motion feature difference screening data and the local waybill feature clustering data corresponding to the local motion feature difference screening data to obtain the advertising video content motion detection data.

[0165] Specifically, according to the difference of the local motion characteristics after screening, the data is screened and compared with the local motion characteristics clustering data corresponding to it. According to the category and difference of the motion pattern, the corresponding motion behavior label is generated. For example, specific labels such as "fast motion", "slow motion", "direction change" etc. can be assigned to different types of motion behaviors. The feature of each local motion area is matched with its clustering result, and a label is generated for each motion behavior to form the motion detection data of the advertising video content. This data is used to describe the type and position of the motion behavior in the video.

[0166] The three-dimensional convolution in the present invention not only captures the static features in the advertising material in space, but also captures the motion changes of objects in the time dimension, and extracts the motion features containing time and space information. By extracting motion features from the local area in the advertising video, the motion behavior of the specific area can be analyzed more finely, rather than just the average motion in the global range. Clustering calculation is performed on the extracted local motion features, and the regions with similar motion patterns can be grouped to identify different motion behavior categories. By screening the differences of the local motion features, those regions with significant motion changes can be identified, while small, irrelevant motions are ignored. By marking the features after the difference screening, generating motion behavior tags, different motion behaviors in the advertisement can be identified and classified. By screening the important change areas in the motion features and generating corresponding behavior tags, the system can carry out structured understanding and annotation of the motion content in the advertising material, which is suitable for the automatic processing and labeling management of the advertising material.

[0167] Optionally, the scene switching detection includes:

[0168] Perform frame convolution calculation on the advertising material video data to obtain advertising material frame convolution layer data;

[0169] Specifically, a pixel-by-pixel convolution calculation is performed on each frame of the video to extract local features of the frame, such as edges, contours, color changes, etc. The convolution operation extracts spatial features within the frame through filters to form a series of convolution feature maps, which are called creative frame convolution layer data. These feature data reflect the spatial structure information and texture features in the image.

[0170] Performing pooling calculation on the convolutional layer data of the advertising material frame to obtain the pooling layer data of the advertising material frame;

[0171] Specifically, the convolutional layer data of the frame is processed for dimensionality reduction, retaining important features while reducing data redundancy. The pooling operation reduces the size of the feature map by taking the maximum or average value of the local area to generate more representative features. These pooling layer data retain the most significant features in the image while reducing the computational complexity.

[0172] Perform clustering calculation on the advertising material frame pooling layer data to obtain advertising material frame clustering data;

[0173] Specifically, similar frame features in the pooling layer data are grouped. The clustering process assigns similar frames to the same category based on the similarity of frame features. By calculating the distance or similarity of frame features, the clustering algorithm divides the frames into several groups, each of which represents a specific scene or content change pattern in the video, and obtains the advertising material frame clustering data.

[0174] Performing Gaussian distribution calculation according to the advertising material frame clustering data to obtain the advertising material frame Gaussian clustering data;

[0175] Specifically, the probability distribution of frame clustering data is estimated, assuming that the frame features conform to Gaussian distribution. By calculating the Gaussian distribution parameters (mean, variance, etc.) of each cluster, the probability distribution model of frame clustering is obtained. These Gaussian clustering data reflect the variation range and distribution characteristics of frame features in each cluster, helping to identify the transition points between different scenes.

[0176] Performing frame structure similarity association on the advertising material frame pooling layer data according to the advertising material frame Gaussian clustering data to obtain advertising material frame structure similarity data;

[0177] Specifically, we use Gaussian clustering data to perform similarity analysis on each frame and calculate the similarity between adjacent frames in terms of spatial structure and feature distribution. By comparing the feature differences between each frame and its adjacent frames, we generate frame structure similarity data, which reflects the visual continuity or mutation between frames.

[0178] Calculate pixel motion information based on the advertising material frame structure similarity data to obtain pixel motion information data;

[0179] Specifically, based on the frame structure similarity data, the pixel-level motion changes between frames are further analyzed. The movement amount of each pixel between adjacent frames is calculated, and information such as the motion direction and speed are extracted. The pixel motion information data reflects the movement pattern of objects in the video and the dynamic changes of the scene.

[0180] Performing motion information clustering calculation according to pixel information motion data to obtain motion information clustering data;

[0181] Specifically, pixel motion information is clustered to group pixels with similar motion patterns into the same group. This clustering process can classify similar motion behaviors (such as object movement, camera switching, etc.) into the same category, helping to identify the main motion patterns in the scene. The generated motion information clustering data describes the motion behavior characteristics of different regions in the video.

[0182] Calculate the inter-frame motion difference according to the motion information clustering data to obtain the inter-frame motion difference data;

[0183] Specifically, the motion information between adjacent frames is compared to calculate the difference in motion between frames. For example, by comparing the motion vector, direction, and speed differences between two frames, significant motion change areas between frames are identified. The generated inter-frame motion difference data is used to measure the possibility of scene switching.

[0184] The scene switching is generated according to the inter-frame motion difference data to obtain the scene switching detection data.

[0185] Specifically, scene switching points are identified based on inter-frame motion difference data. Scene switching is usually manifested as areas with large inter-frame motion differences and significant changes in visual content. By setting a threshold, frame changes exceeding the threshold are detected and marked as scene switching. The identified scene switching points are recorded as scene switching detection data to help analyze different scenes in the video and their transition moments.

[0186] In the present invention, frame convolution calculation is performed on the advertising material video, and high-level features such as image structure information such as edges and textures can be extracted from the original video frame. The generation of pooling layer data helps to reduce unnecessary redundant information, compress feature maps, and enhance the robustness of features, especially in the rapidly changing scenes in advertisements, it can more stably capture key change points. Clustering calculation is performed on the frame pooling layer data, which can effectively group similar frames and identify a series of frames with similar image features. Gaussian distribution calculation is performed on the clustered frames, which can more accurately describe the similarity or difference between frames, help distinguish frames of different categories, and at the same time, provide a precondition for the next step of structural similarity association. By similarity association based on frame Gaussian clustering data, the structural similarity of different frames is detected, and the relationship between frames is further analyzed. Based on the frame structure similarity, the pixel motion information in each frame is further calculated to detect motion changes in the advertising material, such as moving objects, scene conversion, etc. Motion information clustering ensures that different motion modes in advertisements are distinguished, such as distinguishing static product display scenes from fast-switching dynamic scenes, thereby improving the effect of scene switching detection. By calculating the motion differences between adjacent frames, frames with larger motion amplitudes can be identified.

[0187] Optionally, the frame difference calculation includes:

[0188] Calculate the difference of scene switching content and extract the feature of scene switching frequency according to the scene switching detection data and the video data of the advertising material, and obtain the scene switching content difference data and the scene switching frequency feature data;

[0189] Specifically, by analyzing the frame contents before and after the adjacent scene switches, the degree of change in the visual features (such as color, texture, shape, etc.) of the frames before and after the scene switch is calculated. The differences are quantified by comparing the global image histograms and local area features of these frames to generate scene switch content difference data. According to the frequency of scene switching in the video, the distribution and statistical characteristics of the scene switching time interval are extracted. By counting the time interval between each scene switch, the number of switches, and other information, the scene switching frequency feature data is extracted.

[0190] Constructing a video frame scale according to the scene switching content difference data and the scene switching frequency feature data to obtain video frame scale data;

[0191] Specifically, based on the scene switching content difference data and scene switching frequency feature data, frame scale models for different time periods are constructed. For high-frequency switching parts, a smaller frame scale is used to capture finer-grained changes; while for low-frequency switching parts, a larger frame scale is used to reduce the amount of calculation. In this way, the corresponding video frame scale data is generated.

[0192] Extract frames at equal intervals according to the video data of the advertising material to obtain initial advertising material frame data;

[0193] Specifically, according to the total duration of the video and the set frame extraction interval, frame data is extracted from the advertising video at equal time intervals. The initial frame extraction time interval can be adjusted according to the content density of the advertisement and the video frame scale to obtain the initial advertising material frame extraction data. These frames will serve as the basis for subsequent processing.

[0194] According to the video frame scale data, an image pyramid is constructed for the initial advertising material frame data to obtain the advertising material image pyramid data;

[0195] Specifically, the initial advertising material frame data is extracted to construct a pyramid structure according to the video frame scale data. In the image pyramid, each layer represents a resolution level, the higher layer is a low-resolution version, and the lower layer is a high-resolution image. By reducing the image resolution, multi-scale feature analysis can be achieved to obtain the advertising material image pyramid data.

[0196] Constructing a temporal dependency relationship according to the pyramid data of the advertising material image to obtain the temporal dependency relationship data of the advertising material image;

[0197] Specifically, based on the continuity between frames at each level in the image pyramid, the temporal dependency in the frame sequence is analyzed. By analyzing the content changes between adjacent frames, a temporal dependency model for each layer of frames is constructed. This model is used to capture the gradual changes in the frame sequence and generate the temporal dependency relationship data of the advertising material image.

[0198] Extract color-texture fusion features based on the temporal dependency relationship data of the advertising material images to obtain color-texture fusion feature data;

[0199] Specifically, the color and texture information of each frame of the image are fused and extracted. The color space of the image is segmented and the texture features of the local area, such as edges, gradients, etc., are extracted at the same time. These color and texture information are weightedly fused to generate color and texture fusion feature data with multi-dimensional features.

[0200] The difference calculation is performed based on the color and texture fusion feature data to obtain the inter-frame difference data.

[0201] Specifically, based on the color and texture fusion feature data, the difference between adjacent frames is calculated. By comparing the color and texture fusion features of the two frames, the difference score of each pixel or region is calculated. Finally, the difference of the entire image is accumulated and summarized to generate inter-frame difference data, which is used to measure the content changes between the two frames.

[0202] In the present invention, by performing differential analysis on the scene switching content in the advertising material, significant changes during scene switching can be captured, and these changes can be quantified to generate differential data. Feature extraction is performed on the scene switching frequency in the advertising video to help analyze the dynamic information such as the speed and rhythm of the switching in the advertisement. Through different scale analysis, the detailed changes from global to local in the advertising material can be captured to ensure the accuracy of the difference between frames. The image pyramid provides a multi-level feature expression from high resolution to low resolution, which not only retains the global information but also captures the local detail changes. The temporal dependency analysis can capture the continuous changes in the advertising material, ensure the temporal coherence between frames, help distinguish the gradual or sudden changes in different scenes, and improve the accurate detection of the difference between frames, which is particularly suitable for the gradual changes or transition scenes in the advertisement. By fusing and extracting the color information and texture information of the image, the changes in the visual appearance and structural characteristics of the image can be captured. The differential calculation can integrate the multi-dimensional information (such as color, texture, and timing) extracted previously, accurately detect the changes between frames, and improve the sensitivity to scene switching, dynamic changes, and subtle visual differences in the advertising material.

[0203] Optionally, S2 includes:

[0204] S21, dividing the frame extraction data of the advertising material into local areas to obtain frame extraction local area data;

[0205] Specifically, each frame of the image is spatially segmented to divide the frame into multiple small local regions. The image can be divided into blocks of equal or different sizes using a fixed-size grid or a segmentation algorithm based on the image content. These local area data reflect the detailed information of each part of the image, which is convenient for subsequent feature extraction.

[0206] S22, calculating the texture direction according to the local area data of the extracted frame to obtain the texture direction data of the local area of ​​the extracted frame;

[0207] Specifically, in each local area, the grayscale gradient change direction of the pixel is analyzed to extract the main texture direction of the local area. The specific process includes: calculating the pixel gradient of the local area, identifying key structures such as edges and corners in the image, and determining the direction of these structures. By calculating the distribution of the direction vector in each area, the texture direction data of the area is generated.

[0208] S23, performing texture gradient calculation according to the texture direction data of the local area of ​​the extracted frame to obtain the extracted frame texture feature data;

[0209] Specifically, based on the texture direction data, the gradient strength of each local area is further calculated to reflect the severity of the texture change. Gradient calculation can be achieved by calculating the brightness difference or direction change rate between adjacent pixels. The texture features of each local area include information such as the size of the gradient and the frequency of direction changes, forming detailed texture feature data.

[0210] S24, performing information entropy calculation on the extracted frame texture feature data to obtain texture feature information entropy data.

[0211] Specifically, based on the obtained texture feature data, the texture information entropy of each local area is calculated to measure the complexity of the information in the area. The specific steps are: first, the texture features in the local area are normalized and the probability distribution of each texture feature value is calculated. Then, the information entropy is calculated based on the probability distribution. The higher the information entropy value, the more complex the texture change in the area. The generated texture feature information entropy data is used to describe the texture complexity of different areas in the frame.

[0212] The present invention refines the processing of image content by dividing each frame of the advertising material into multiple local areas. The main direction of the texture is calculated in each local area, and the structural features of different parts of the image can be captured. By detecting the texture direction of the local area, the shape and boundary information of the object in the image can be better understood. Gradient calculation based on texture direction data can detect the degree of change of the texture in the image, including the clarity and direction change of the edge and contour. Gradient calculation provides quantitative data for the complexity of the texture in the image. By calculating the information entropy of the texture feature data, the information complexity in the local area can be quantified. Areas with high entropy values ​​usually represent complex image content and rich information; while areas with low entropy values ​​represent simple images or more repeated information. By comprehensively calculating the texture direction, gradient and information entropy of the local area, the changes between frames in the advertising material can be more accurately identified, including subtle scene transitions, movement of advertising elements, etc. Redundant frames and areas can be effectively ignored through entropy screening, thereby improving processing efficiency. The system can focus on high-value advertising content, reduce the waste of processing resources, and thus accelerate the overall analysis and annotation process.

[0213] Optionally, S3 includes:

[0214] S31, separating the advertisement material frame data into high and low entropy values ​​according to the texture feature information entropy data, to obtain advertisement material high entropy frame data and advertisement material low entropy frame data;

[0215] Specifically, according to the previously calculated texture feature information entropy data, the advertising material frame data is divided into two categories: high entropy value and low entropy value. A threshold can be set to classify according to the size of the information entropy value. High entropy value frames represent frames with complex textures and rich information, while low entropy value frames represent frames with simple textures. The generated high entropy frame data and low entropy frame data are stored separately.

[0216] S32, sorting the high entropy frame extraction data of the advertising material and the low entropy frame extraction data of the advertising material by frame entropy values ​​to obtain high entropy frame extraction sorting data and low entropy frame extraction sorting data respectively;

[0217] Specifically, each frame in the high entropy and low entropy frame extraction data is sorted according to its corresponding information entropy value. The purpose of sorting is to arrange the information entropy values ​​from high to low or from low to high to generate high entropy frame extraction sorting data and low entropy frame extraction sorting data. The sorted data set helps to identify the frames with the largest amount of information or the frames with relatively less information.

[0218] S33, performing local entropy value fluctuation detection according to the high entropy frame extraction sorting data and the low entropy frame extraction sorting data, and obtaining high entropy frame extraction local entropy value fluctuation data and low entropy frame extraction local entropy value fluctuation data respectively;

[0219] Specifically, local fluctuation detection is performed on the sorted high entropy and low entropy frame data. First, several adjacent frames are selected and the entropy value change amplitude between these frames is calculated. Local fluctuation reflects the difference in texture information between frames, and generates local entropy value fluctuation data of high entropy frame extraction and local entropy value fluctuation data of low entropy frame extraction.

[0220] S34, performing local entropy value fluctuation gradient calculation on the local entropy value fluctuation data of the high entropy extraction frame and the local entropy value fluctuation data of the low entropy extraction frame, to obtain the local entropy value fluctuation gradient data of the high entropy extraction frame and the local entropy value fluctuation gradient data of the low entropy extraction frame respectively;

[0221] Specifically, by calculating the gradient of the entropy change in the local entropy fluctuation data, the speed of change of the amount of information between frames is evaluated. The gradient reflects the direction and magnitude of the change in information entropy, and generates high-entropy frame local entropy fluctuation gradient data and low-entropy frame local entropy fluctuation gradient data. This process helps identify the areas where information changes most significantly.

[0222] S35, using the high entropy frame extraction local entropy value fluctuation gradient data and the low entropy frame extraction local entropy value fluctuation gradient data to filter the advertising material frame extraction data, and obtain first advertising material frame extraction filtering data and second advertising material frame extraction filtering data respectively;

[0223] Specifically, based on the local entropy value fluctuation gradient data of high-entropy and low-entropy frames, the frames with the most significant or stable information changes are screened out. A gradient threshold can be set to screen out frames whose gradients exceed the threshold, forming the first advertising material frame extraction screening data and the second advertising material frame extraction screening data, which correspond to the screening results of high-entropy and low-entropy frames, respectively.

[0224] S36, performing overlap screening according to the first advertising material frame extraction and screening data and the second advertising material frame extraction and screening data to obtain advertising material frame extraction and screening data.

[0225] Specifically, by comparing the first advertising material frame extraction and screening data with the second advertising material frame extraction and screening data, the similarity or overlap of the frames in the two data sets is analyzed. For identical or similar frames, feature similarity or spatial position overlap is calculated to screen and retain them. Finally, the advertising material frame extraction and screening data is obtained, which includes frames with representative information and variation in high entropy and low entropy frames.

[0226] In the present invention, the high and low entropy values ​​of the frames are separated according to the texture feature information entropy, and the frames with high information content (high entropy) can be distinguished from the frames with low information content (low entropy), ensuring that the high entropy and low entropy frames are analyzed in a targeted manner during processing. Frame entropy value sorting can help distinguish important frames from redundant frames, ensure that those frames with large information content can be processed first, improve the accuracy of frame selection, and are particularly suitable for capturing key scenes in advertising materials. The local detection method can capture the changes in the amount of information at the detail level of the frame, not only focusing on the overall features, but also identifying important local changes, ensuring a more detailed analysis of changes in advertising materials, and is suitable for detailed analysis of complex advertising scenes. Gradient calculation can more accurately identify subtle changes between frames, improve the detection ability of gradual scenes in advertising materials, and enable the system to capture changes that are not visually noticeable but are crucial to advertising content, such as the visual effect of fading in and out. Using the local gradient data of high entropy and low entropy frames to screen advertising materials, key information frames can be screened out from different angles to ensure that valuable frames can be extracted under different amounts of information. Overlap screening can effectively remove redundant frames with high similarity and repetitiveness, thereby reducing the interference of useless information, further optimizing the frame selection results, and improving the efficiency and quality of data processing.

[0227] Optionally, S4 includes:

[0228] S41, performing object detection on the frame-selected data of the advertising material to obtain preliminary object detection data;

[0229] Specifically, object detection is performed on each frame of the filtered advertising material. By detecting key objects in the image, such as brand logos, products, text, etc., the main visual elements in the advertisement are identified. The process includes locating different objects in the image and marking them into predefined categories to generate preliminary object detection data. This data contains information such as the location, size, and category of the object.

[0230] S42, classifying the advertising material type according to the preliminary object detection data to obtain advertising material type data;

[0231] Specifically, the advertising material is classified according to the type and number of detected objects. By analyzing the combination of objects in the frame, it is determined which type of advertisement it belongs to, such as promotional advertisement, brand promotion advertisement or product display advertisement. The classification basis may include the visual features of the object, layout style and text information. The final output advertising material type data describes the advertising type attributes of each frame.

[0232] S43, constructing a spatial layout relationship according to the advertising material type data to obtain advertising material type spatial relationship data;

[0233] Specifically, a spatial layout relationship model between objects is constructed based on the position, size and relative distance of objects in the advertising material. This model describes the arrangement and distribution of advertising elements in the image. By calculating the distance, angle and relative position between objects, spatial relationship data of the advertising material type is generated to reflect the layout characteristics of the advertising elements.

[0234] S44. Perform feature vector conversion according to the spatial relationship data of the advertising material type to obtain advertising element feature data for advertising material labeling.

[0235] Specifically, the spatial layout relationship data of the advertising material is converted into a high-dimensional feature vector. The position, size, category and relative relationship of each object with other objects are represented as a numerical vector. These feature vectors are used to further analyze the visual elements in the advertisement and generate advertising element feature data. The feature data can describe the core visual information in the advertisement to support subsequent advertisement labeling operations.

[0236] In the present invention, object detection is performed on frame-by-frame screening data of advertising materials, and core objects in videos or images, such as brand logos, products, people, etc., can be identified. According to the object detection results, the types of advertising materials are classified to distinguish different advertising contents, such as product display ads, brand promotion ads, service ads, etc. The construction of spatial layout relationships can help the system understand the arrangement and interaction of different elements in the advertising scene, identify the key visual structures in the advertisements, and ensure a better understanding of the relative importance of elements during the advertisement annotation process. It is particularly suitable for the comprehensive display of brand logos, product locations and text descriptions in advertisements. Feature vector conversion can convert complex information in advertising materials into unified structured data, which is convenient for automated processing and analysis.

[0237] Optionally, the present application further provides a labeling system based on advertising material data, which is used to execute the labeling method based on advertising material data as described above. The labeling system based on advertising material data includes:

[0238] The advertising material video frame extraction module is used to obtain advertising material video data; perform frame extraction processing according to the advertising material video data to obtain advertising material frame extraction data;

[0239] The frame extraction texture feature extraction module is used to extract texture features from the frame extraction data of the advertising material to obtain the frame extraction texture feature data;

[0240] The texture feature information entropy calculation module is used to calculate the information entropy of the extracted frame texture feature data to obtain the texture feature information entropy data;

[0241] The advertising material frame extraction and screening module is used to screen the advertising material frame extraction data according to the texture feature information entropy data to obtain the advertising material frame extraction and screening data;

[0242] The advertisement element feature extraction module is used to extract advertisement element features from advertisement material frame extraction and screening data to obtain advertisement element feature data for advertisement material labeling.

[0243] The purpose of the present invention is that the advertising material is effectively divided into multiple key frames through frame extraction processing, which significantly reduces the redundant data processing requirements for the entire video. The calculation method of texture features and information entropy is combined, especially in step S2, through texture feature extraction and information entropy calculation, the complexity and information content of the frame can be accurately measured, the difference of the frame can be quantified, and the data basis can be provided for the subsequent screening step, which not only significantly improves the screening efficiency in large-scale material processing, but also ensures that the frames with key advertising elements are retained, and the waste of computing resources for processing useless frames is reduced. The frame screening mechanism in step S3 uses texture feature information entropy data to screen the frame data, and through high and low entropy value separation and sorting, the frames with large information content are screened out to ensure that key scenes and elements are retained. At the same time, local entropy value fluctuation detection and gradient calculation further refine the screening process, making the change trend of the frame clearer and eliminating redundant frames. In step S4, by extracting advertising element features from the screened advertising material frames, key advertising elements such as products, brand logos, text, characters, etc. are identified, and feature vectors are constructed according to spatial layout relationships. The feature extraction and spatial layout analysis of advertising elements can enhance the depth of annotation work and ensure that the relationship between advertising elements is accurately reflected. The present invention improves the processing efficiency and quality of advertising materials and is suitable for the automatic processing and intelligent analysis of large-scale advertising data.

[0244] Therefore, from any point of view, the embodiments should be regarded as illustrative and non-restrictive, and the scope of the present invention is limited by the attached application documents rather than the above description, and it is intended that all changes falling within the meaning and scope of equivalent elements of the application documents are included in the present invention.

[0245] The above description is only a specific embodiment of the present invention, so that those skilled in the art can understand or implement the present invention. Various modifications to these embodiments will be apparent to those skilled in the art, and the general principles defined herein may be implemented in other embodiments without departing from the spirit or scope of the present invention. Therefore, the present invention will not be limited to the embodiments shown herein, but should conform to the widest scope consistent with the principles and novel features invented herein.

Claims

1. A marking method based on advertising material data, characterized in that: The method comprises: S1. Acquire video data of the advertising material; perform frame extraction processing according to the video data of the advertising material to obtain frame extraction data of the advertising material; S2, extracting texture features from the frame-by-frame data of the advertising material to obtain frame-by-frame texture feature data; calculating information entropy on the frame-by-frame texture feature data to obtain texture feature information entropy data; S3, filtering the advertising material frame extraction data according to the texture feature information entropy data to obtain the advertising material frame extraction filtering data; S4, extracting advertising element features from the advertising material frame extraction and screening data to obtain advertising element feature data for advertising material labeling; S3 includes: According to the texture feature information entropy data, the advertising material frame data is separated into high and low entropy values ​​to obtain the advertising material high entropy frame data and the advertising material low entropy frame data; Sorting the high entropy frame data and the low entropy frame data of the advertising material by frame entropy values ​​to obtain high entropy frame sorting data and low entropy frame sorting data respectively; Perform local entropy fluctuation detection based on high entropy frame sorting data and low entropy frame sorting data to obtain high entropy frame local entropy fluctuation data and low entropy frame local entropy fluctuation data respectively; The local entropy value fluctuation gradient is calculated for the local entropy value fluctuation data of the high entropy extraction frame and the local entropy value fluctuation data of the low entropy extraction frame, and the local entropy value fluctuation gradient data of the high entropy extraction frame and the local entropy value fluctuation gradient data of the low entropy extraction frame are obtained respectively; The advertising material frame extraction data is screened using the high entropy frame extraction local entropy value fluctuation gradient data and the low entropy frame extraction local entropy value fluctuation gradient data to obtain first advertising material frame extraction screening data and second advertising material frame extraction screening data respectively; Perform overlap screening according to the first advertising material frame extraction screening data and the second advertising material frame extraction screening data to obtain advertising material frame extraction screening data; S4 includes: Perform object detection on the frame-selected data of the advertising material to obtain preliminary object detection data; Classify the advertising material type according to the preliminary object detection data to obtain advertising material type data; Constructing spatial layout relationships according to the advertising material type data to obtain advertising material type spatial relationship data; Feature vector conversion is performed according to the spatial relationship data of the advertising material type to obtain the advertising element feature data for the advertising material labeling operation.

2. The method according to claim 1, characterized in that The advertising material frame extraction data includes first advertising material frame extraction data, second advertising material frame extraction data and third advertising material frame extraction data, and S1 includes: Get the video data of the creative; Performing frame extraction processing at equal intervals according to the video data of the advertising material to obtain first advertising material frame extraction data; Perform motion detection frame extraction according to the advertisement material video data to obtain second advertisement material frame extraction data; Key frames are extracted according to the advertisement material video data to obtain third advertisement material frame data.

3. The method according to claim 2, characterized in that The motion detection frame extraction is as follows: Calculating the video content complexity of the advertisement material video data to obtain advertisement video content complexity data; Performing motion detection on the advertisement material video data to obtain advertisement video content motion detection data; Performing scene switching detection on the video data of the advertising material to obtain scene switching detection data; Calculate the inter-frame difference of the video data of the advertisement material according to the scene switching detection data to obtain the inter-frame difference data; Calculate the frame extraction interval according to the advertisement video content complexity data, the advertisement video content motion detection data, the frame difference data and the scene switching detection data to obtain the frame extraction interval data; The advertisement material video data is subjected to frame extraction processing according to the frame extraction interval data to obtain second advertisement material frame extraction data.

4. The method according to claim 3, characterized in that Motion detection includes: Performing three-dimensional convolution calculation on the advertising material video data to obtain the advertising material video motion feature data; Extracting local motion features from the video motion feature data of the advertisement material to obtain local motion feature data; Perform motion feature clustering calculation on the local waybill feature data to obtain local motion feature clustering data; Perform difference screening on the local motion feature data according to the local waybill feature clustering data to obtain local motion feature difference screening data; Motion behavior labels are generated based on the local motion feature difference screening data and the local waybill feature clustering data corresponding to the local motion feature difference screening data to obtain the advertising video content motion detection data.

5. The method according to claim 3, characterized in that: The scene switching detection includes: Perform frame convolution calculation on the advertising material video data to obtain advertising material frame convolution layer data; Performing pooling calculation on the convolutional layer data of the advertising material frame to obtain the pooling layer data of the advertising material frame; Perform clustering calculation on the advertising material frame pooling layer data to obtain advertising material frame clustering data; Performing Gaussian distribution calculation according to the advertising material frame clustering data to obtain the advertising material frame Gaussian clustering data; Performing frame structure similarity association on the advertising material frame pooling layer data according to the advertising material frame Gaussian clustering data to obtain advertising material frame structure similarity data; Calculate pixel motion information based on the advertising material frame structure similarity data to obtain pixel motion information data; Performing motion information clustering calculation according to pixel information motion data to obtain motion information clustering data; Calculate the inter-frame motion difference according to the motion information clustering data to obtain the inter-frame motion difference data; The scene switching is generated according to the inter-frame motion difference data to obtain the scene switching detection data.

6. The method according to claim 3, characterized in that The frame difference calculation includes: Calculate the difference of scene switching content and extract scene switching frequency features based on the scene switching detection data and the advertising material video data to obtain scene switching content difference data and scene switching frequency feature data; Constructing a video frame scale according to the scene switching content difference data and the scene switching frequency feature data to obtain video frame scale data; Extract frames at equal intervals according to the video data of the advertising material to obtain initial advertising material frame data; According to the video frame scale data, an image pyramid is constructed for the initial advertising material frame data to obtain the advertising material image pyramid data; Constructing a temporal dependency relationship according to the pyramid data of the advertising material image to obtain the temporal dependency relationship data of the advertising material image; Extract color-texture fusion features based on the temporal dependency relationship data of the advertising material images to obtain color-texture fusion feature data; The difference calculation is performed based on the color and texture fusion feature data to obtain the inter-frame difference data.

7. The method according to claim 1, characterized in that S2 include: Divide the frame extraction data of the advertisement material into local areas to obtain frame extraction local area data; Calculate the texture direction according to the local area data of the extracted frame to obtain the texture direction data of the local area of ​​the extracted frame; Calculate the texture gradient according to the texture direction data of the local area of ​​the frame extraction to obtain the frame extraction texture feature data; The information entropy of the extracted frame texture feature data is calculated to obtain texture feature information entropy data.

8. A labeling system based on advertising material data, characterized in that: Used to execute the labeling method based on advertising material data according to claim 1, the labeling system based on advertising material data comprises: The advertising material video frame extraction module is used to obtain advertising material video data; perform frame extraction processing according to the advertising material video data to obtain advertising material frame extraction data; The frame extraction texture feature extraction module is used to extract texture features from the frame extraction data of the advertising material to obtain the frame extraction texture feature data; The texture feature information entropy calculation module is used to calculate the information entropy of the extracted frame texture feature data to obtain the texture feature information entropy data; The advertising material frame extraction and screening module is used to filter the advertising material frame extraction data according to the texture feature information entropy data to obtain the advertising material frame extraction and screening data; wherein the screening includes: separating the advertising material frame extraction data from high and low entropy values ​​according to the texture feature information entropy data to obtain the advertising material high entropy frame extraction data and the advertising material low entropy frame extraction data; sorting the advertising material high entropy frame extraction data and the advertising material low entropy frame extraction data by frame entropy values ​​to obtain high entropy frame extraction sorting data and low entropy frame extraction sorting data respectively; performing local entropy value fluctuation detection according to the high entropy frame extraction sorting data and the low entropy frame extraction sorting data to obtain high entropy frame extraction local entropy value fluctuation data respectively. and low entropy frame extraction local entropy value fluctuation data; perform local entropy value fluctuation gradient calculation on high entropy frame extraction local entropy value fluctuation data and low entropy frame extraction local entropy value fluctuation gradient data, respectively obtaining high entropy frame extraction local entropy value fluctuation gradient data and low entropy frame extraction local entropy value fluctuation gradient data; use high entropy frame extraction local entropy value fluctuation gradient data and low entropy frame extraction local entropy value fluctuation gradient data to filter the advertising material frame extraction data, respectively obtaining the first advertising material frame extraction screening data and the second advertising material frame extraction screening data; perform overlap screening according to the first advertising material frame extraction screening data and the second advertising material frame extraction screening data, and obtain the advertising material frame extraction screening data; The advertising element feature extraction module is used to extract advertising element features from the advertising material frame extraction and screening data to obtain advertising element feature data for advertising material labeling operations; wherein the advertising element feature extraction includes: performing object detection on the advertising material frame extraction and screening data to obtain preliminary object detection data; classifying the advertising material type according to the preliminary object detection data to obtain advertising material type data; constructing spatial layout relationships according to the advertising material type data to obtain advertising material type spatial relationship data; performing feature vector conversion according to the advertising material type spatial relationship data to obtain advertising element feature data for advertising material labeling operations.

Images