Advertisement recognition method and device, computing equipment, storage medium and program product

Abstract

The invention discloses an advertisement recognition method and device, computing equipment, a storage medium and a program product, and the method comprises the steps: obtaining a to-be-recognized video, the to-be-recognized video comprises at least one scene segment, and the scene segment comprises an image frame and audio data; obtaining text content contained in the image frame of the scene segment; obtaining audio text information corresponding to the audio data of the scene segment; obtaining a text abstract representing the picture content of the scene segment; and inputting the text content, the audio text information and the text abstract which are subjected to time alignment into a preset learning model, and determining whether the scene segment comprises advertisement information or not. According to the invention, the accuracy and the recognition efficiency of advertisement recognition can be improved.

Description

Technical Field

[0001] This application relates to the field of video playback technology, and in particular to an advertising recognition method, apparatus, computing device, storage medium, and program product. Background Technology

[0002] With the rapid development of short video platforms, video portals, and live streaming platforms, user-generated content (UGC) videos account for a large proportion of the internet.

[0003] Video advertising, especially soft product placement (i.e., products or brands embedded in program content in a non-direct promotional way), is widespread in various types of videos. To improve user experience and achieve advertising compliance with regulations, it is necessary to accurately identify and extract advertising segments from videos.

[0004] The information disclosed in this background section is included only to enhance the understanding of the context of this disclosure, and therefore may contain information that does not constitute relevant technology currently known to those skilled in the art. Summary of the Invention

[0005] This application provides an advertising recognition method, apparatus, computing device, storage medium, and program product to solve the problem of low accuracy in advertising recognition in related technologies.

[0006] The technical solution adopted in this application is as follows.

[0007] In a first aspect, this application provides an advertisement identification method, including: Acquire a video to be identified, the video to be identified including at least one scene segment, the scene segment including image frames and audio data; Obtain the text content contained in the image frames of the scene fragment; Obtain the audio text information corresponding to the audio data of the scene segment; Obtain a text summary representing the image content of the scene segment; The time-aligned text content, the audio text information, and the text summary are input into a preset learning model to determine whether the scene segment includes advertising information.

[0008] Thus, this application can achieve accurate and comprehensive identification of advertisements by combining text content, audio text information, and text summaries.

[0009] In conjunction with the first aspect, in one possible implementation, the method further includes: The scene switching point is determined based on the inter-frame similarity of the image frames in the video to be identified, wherein the next frame in the adjacent video frames with an inter-frame similarity lower than a preset threshold is taken as the scene switching point. Based on the scene switching point, the video to be identified is segmented into scenes to obtain at least one scene segment.

[0010] Thus, this application is able to accurately segment video scene segments.

[0011] In conjunction with the first aspect, in one possible implementation, the text content contained in the image frame of the scene fragment includes: Frames of the scene segment are extracted; Text recognition is performed on the image frames obtained by frame extraction to obtain the text content.

[0012] Thus, this application is able to accurately identify the text content in scene fragments.

[0013] In conjunction with the first aspect, in one possible implementation, obtaining the audio text information corresponding to the audio data of the scene segment includes: Based on the start and end times of the scene segment, obtain the audio data corresponding to the scene segment; The audio data is processed by an automatic speech recognition system to convert it into text, thereby obtaining the audio text information.

[0014] Thus, this application is able to accurately identify audio text information in scene segments.

[0015] In conjunction with the first aspect, in one possible implementation, obtaining a text summary representing the image content of the scene fragment includes: Frames of the scene segment are extracted; For each extracted image frame, image features are extracted using a preset image encoder; The image features of all extracted image frames are averaged and pooled to obtain the visual feature vector. The text summary is generated based on the visual feature vector using a preset text decoder.

[0016] Thus, this application is able to accurately identify the text summary of scene fragments.

[0017] In conjunction with the first aspect, in one possible implementation, the method further includes: The prompt words, along with the text content, the audio text information, and the text summary, are input into the preset learning model. The prompt words are used to provide recognition instructions, guiding the preset learning model to output corresponding recognition results based on the input text content, the audio text information, and the text summary. Based on the prompt words, the text content, the audio text information, and the text summary, the recognition result is output, and the recognition result includes at least whether the scene segment contains advertising information.

[0018] In this way, this application can obtain more recognition results and acquire advertising information contained in the video.

[0019] In conjunction with the first aspect, in one possible implementation, the method further includes: When the scene segment contains advertising information, the text content, the audio text information, the text summary, and the recognition result of the scene segment containing advertising information are used as training data for the preset learning model.

[0020] In this way, the accuracy of the pre-set learning model can be continuously improved.

[0021] In conjunction with the first aspect, in one possible implementation, the method further includes: When the scene segment includes advertising information, the duration for which the user watches the scene segment is obtained; When the overlap between the duration of the advertisement and the duration of the scene segment exceeds a preset duration, the advertisement information is determined to be a valid advertisement.

[0022] Thus, this application is able to identify valid advertisements, providing a basis for further processing.

[0023] In conjunction with the first aspect, in one possible implementation, the method further includes: When advertising information is included in the scene segment, the advertising information is compared with a preset database to determine whether the advertising information has been declared, wherein the preset database includes advertising information that has already been declared.

[0024] Thus, this application is able to identify undeclared advertising information.

[0025] In conjunction with the first aspect, in one possible implementation, the method further includes: When the video to be identified is played, and the scene segment includes advertising information, the scene segment is skipped according to its start and end times.

[0026] Thus, this application is able to provide an ad-skipping function.

[0027] In conjunction with the first aspect, in one possible implementation, the preset learning model is one of the following: a large language model, a chat generation pre-trained transformation model, a generative pre-trained transformation model, and a general language model.

[0028] Thus, this application can achieve accurate identification of advertisements by utilizing different preset learning models.

[0029] Secondly, this application also provides an advertising recognition device, comprising: A video acquisition module is used to acquire a video to be identified, the video to be identified including at least one scene segment, the scene segment including image frames and audio data; The text content acquisition module is used to acquire the text content contained in the image frames of the scene fragment; The audio text information acquisition module is used to acquire the audio text information corresponding to the audio data of the scene segment; The text summary acquisition module is used to acquire a text summary representing the image content of the scene segment; The recognition module is used to input the time-aligned text content, the audio text information, and the text summary into a preset learning model to determine whether the scene segment includes advertising information.

[0030] Thirdly, this application provides a computing device, characterized in that it includes a memory and a processor, wherein the memory is used to store computer programs or instructions; when the computer programs or instructions are executed by the processor, they implement the method in the first aspect or any possible implementation of the first aspect.

[0031] Fourthly, this application also provides a computer-readable storage medium storing a computer program or instructions that, when executed by a processor, implement the method in the first aspect or any possible implementation of the first aspect.

[0032] Fifthly, this application also provides a computer program product, including a computer program, characterized in that, when the computer program is executed by a processor, it implements the method in the first aspect or any possible implementation of the first aspect.

[0033] The beneficial effects of aspects two through five above can be referenced to aspect one or any possible implementation thereof, and will not be elaborated upon here. Based on the implementations provided in the above aspects, this application can also be further combined to provide more implementations.

[0034] Other advantages, objectives and features of this application will be partly apparent from the description below, and partly understood by those skilled in the art through study and practice of this application. Attached Figure Description

[0035] To more clearly illustrate the technical solutions in the embodiments or related technologies of this application, the accompanying drawings used in the description of the embodiments or related technologies will be briefly introduced below. Obviously, the accompanying drawings described below are only some embodiments of this application. For those skilled in the art, other drawings can be obtained based on the structures shown in these drawings without creative effort.

[0036] Figure 1 This is a schematic diagram of an exemplary operating environment according to an embodiment of this application; Figure 2 This is a flowchart illustrating an embodiment of an advertising recognition method according to this application; Figure 3 This is a flowchart illustrating another embodiment of the advertising recognition method of this application; Figure 4 This is a schematic diagram of the training process of the preset learning model in an embodiment of this application; Figure 5 This is a structural block diagram of an advertising recognition device according to an embodiment of this application; Figure 6 This is a structural block diagram of a computing device according to an embodiment of this application. Detailed Implementation

[0037] It should be noted that, in this application, the terms "exemplary" or "for example" are used to indicate that something is being described as an example, illustration, or illustration. Any embodiment or design described as "exemplary" or "for example" in this application should not be construed as being more preferred or advantageous than other embodiments or design solutions. Specifically, the use of terms such as "exemplary" or "for example" is intended to present the relevant concepts in a concrete manner.

[0038] The term "and / or" as used in this application includes any and all combinations of one or more of the associated listed items. In this application, "at least one" means one or more, and "more than one" means two or more. The terms "first," "second," and other ordinal terms used in this application may be used to describe various constituent elements, but these constituent elements are not limited by these terms. The purpose of using these terms is solely to distinguish one constituent element from others and should not be construed as indicating or implying relative importance. For example, without departing from the scope of this application, a first constituent element may be named a second constituent element, and similarly, a second constituent element may be named a first constituent element.

[0039] The execution order of the steps in this application is not unique and can be adjusted according to actual needs. Those skilled in the art will understand that the steps can be executed in the order described in the embodiments of this application, or in parallel, interleaved, or other suitable orders without departing from the spirit and substance of this application. This application does not limit the order of the method steps, and any adjustment of the order of the steps should be considered as falling within the protection scope of this application.

[0040] Before introducing the embodiments of this application, the technical terms and background technology involved in this application will be introduced first.

[0041] Video ads are widely present in various types of videos. In order to improve the user's video viewing experience, it is necessary to accurately identify the ad segments in the video so as to supervise the compliance, appropriateness and legality of the ads.

[0042] In related technologies, there are various methods for identifying advertising segments in videos, including those based on visual content, audio content, and manual labeling. For example, image recognition methods based on video content identify advertisements by recognizing product logos, matching image similarity, and detecting scene abrupt changes; audio content analysis methods identify advertisements through audio fingerprinting and keyword recognition; manual labeling methods identify advertisements through program guides or author-generated labels; and deep learning-based unimodal models use CNNs or LSTMs to extract features from images or audio and then classify them for advertising identification.

[0043] In related technologies, information from a single modality cannot fully express advertising intent. For example, detecting logos solely through images cannot identify subtle advertising placements (such as soft ads in spoken text); relying solely on voice analysis can also easily miss advertisements that combine text and images. Especially in UGC videos, due to the diverse types of advertisements, even if identification is achieved by matching audio features with the audio features of complete advertisements in a database, the database is incomplete, which can still lead to missed detections. Therefore, the accuracy of ad recognition in videos based on fixed rules or static models drops significantly in unstructured UGC content. This is especially true in scenarios with missing subtitles, high audio noise, and low image quality, where the accuracy is even lower. Furthermore, ad recognition technologies in this area often focus on extracting "commercial ad segments," lacking support for recognizing soft ads and undeclared ads, leading to regulatory gaps and a decline in user experience.

[0044] The advertising recognition method of this application can adapt to different types of content styles and advertising formats; it can be used in scenarios such as advertising duration statistics, identification of undeclared advertisements, and review of illegal content; it improves the automation level of advertising supervision and reduces the cost of manual review. Simultaneously, by introducing time-aligned subtitles or embedded text, audio recognition text, and scene fragment summaries, semantic-level advertising recognition is achieved through a large language model, improving the accuracy of advertising recognition.

[0045] First, one or more exemplary operating environments for embodiments of this application are described. Figure 1 A schematic diagram of an implementation environment is provided, which includes a server 101 and a client 102.

[0046] Server 101 and client 102 can communicate via a network connection. The network can be wired or wireless. The network includes various network devices such as routers, switches, multiplexers, hubs, modems, bridges, repeaters, firewalls, proxy devices, and / or similar devices. The network can include physical links, such as coaxial cable links, twisted-pair cable links, fiber optic links, or combinations thereof, or wireless links, such as cellular links, satellite links, and Wi-Fi links.

[0047] Client 102 may include various types of computer devices, such as portable handheld devices, general-purpose computers (such as personal computers and laptops), workstation computers, wearable devices, smart screen devices, self-service terminal devices, service robots, gaming systems, various messaging devices, or other electronic devices. These computer devices can run various types and versions of software applications and operating systems. Portable handheld devices may include cellular phones, smartphones, tablets, personal digital assistants, etc. Wearable devices may include head-mounted displays (such as smart glasses). Client 102 may include input / output interfaces. Input interfaces may include touchpads, touchscreens, mice, keyboards, or other sensing elements. Input interfaces can be configured to receive user commands that enable the client to perform various operations, such as uploading or browsing data. Output interfaces are used to output information to the user, such as displaying information.

[0048] Client 102 can run client software with video playback capabilities, such as an application (App). This application can be a standalone application or a subroutine within an application.

[0049] Server 101 should be interpreted broadly, referring to an entity capable of responding to external requests and providing data, resources, or services. Server 101 can be a standalone physical server, a server cluster or distributed system composed of multiple physical servers, or a cloud server providing basic cloud computing services such as cloud services, cloud databases, cloud computing, cloud functions, cloud storage, network services, cloud communication, middleware services, domain name services, security services, CDN (Content Delivery Network), and big data and artificial intelligence platforms.

[0050] Server 101 may consist of one or more computing devices. These computing devices may include virtualized computing instances. Virtualized computing instances may include virtual machines, such as emulations of computer systems, operating systems, etc. The computing devices may load virtual machines based on virtual images and / or other data defining specific software (e.g., operating systems, dedicated applications, servers) used for emulation. As the demand for different types of processing services changes, different virtual machines may be loaded and / or terminated on one or more computing devices. A hypervisor may be implemented to manage the use of different virtual machines on the same computing device. Server 101 may run one or more services or software applications that enable the execution of the methods according to this application. Server 101 may also provide other services or software applications, which may include non-virtualized environments and virtualized environments.

[0051] Server 101 may include one or more components that implement the functions performed by server 101. These components may include software components, hardware components, or combinations thereof that can be executed by one or more processors. In some embodiments, server 101 may provide services such as storage, reading, writing, querying, and deleting, such as providing data processing services to clients. In other embodiments, a client user may sequentially use one or more client applications to interact with server 101 to utilize the services provided by these components.

[0052] The technical solutions of this application are described below through several embodiments. It should be understood that these embodiments can be implemented in many different forms and should not be construed as being limited to the embodiments set forth herein.

[0053] In this embodiment, the video to be identified may include various video data transmitted over a network or stored locally. In live streaming scenarios, video data is acquired, encoded, and transmitted in real time. In on-demand or other video playback scenarios, video data is pre-recorded content stored on a server or local storage, which users can select and play as needed.

[0054] See Figure 2This is a flowchart illustrating an embodiment of an advertisement recognition method according to this application. The advertisement recognition method includes: S21: Obtain the video to be identified, which includes at least one scene segment, and the scene segment includes image frames and audio data.

[0055] After acquiring the video to be identified, it can be decoded to obtain image frames and audio data. As mentioned above, the video to be identified can be live or stored on a server. The video to be identified includes at least one scene segment. By analyzing each frame of the video to be identified, the video segments included can be identified.

[0056] In one embodiment of this application, scene switching points can be determined by calculating the inter-frame similarity of each image frame, thereby identifying scene segments.

[0057] In one embodiment, a scene switching point is determined based on the inter-frame similarity of image frames in the video to be identified, wherein the next frame in an adjacent video frame with an inter-frame similarity lower than a preset threshold is taken as the scene switching point; based on the scene switching point, the video to be identified is segmented into scenes to obtain at least one scene segment.

[0058] Inter-frame similarity can be obtained based on pixel differences, histogram differences, or visual feature differences extracted by deep learning models between image frames. For example, to obtain the inter-frame similarity of image frames, features of the image frames can be extracted first, such as color histograms, perceptual hashes, and depth features. Then, histogram similarity, mean squared error, and perceptual hash distance are calculated to obtain the inter-frame similarity. When the similarity between two adjacent frames is lower than a preset threshold, the latter frame is taken as the scene transition point. For example, if the similarity between frame i and frame i+1 is lower than the preset threshold, a scene transition has occurred between these two frames, and frame i+1 is the first frame of the new scene, which is taken as the scene transition point.

[0059] For each scene segment, steps S22-S25 are executed to determine whether the scene segment contains advertising information. This confirms whether the video to be identified contains advertising information and the specific time when the advertising information appears. The video to be identified may contain multiple scene segments. For each scene segment, the identification of whether it contains advertising information is performed. Therefore, the video to be identified may contain one or more advertising segments.

[0060] S22: Obtain the text content contained in the image frame of the scene fragment.

[0061] In the embodiments of this application, OCR technology can be used to recognize subtitles, title bars, information bars, bullet comments, and text on objects in the scene included in the image frame in order to obtain text content.

[0062] In some embodiments, scene segments can be periodically frame-sampling to extract n image frames for text content recognition, thereby obtaining the text content contained in the image frames of the scene segment. For example, one frame can be extracted every five frames. Since the similarity between frames in the same scene is high, subsequent processing through frame extraction can reduce the amount of computation and save computing power.

[0063] In other embodiments, frame extraction can also be performed as keyframe extraction, time-interval extraction, or other similar methods. For example, a frame can be extracted every 5 or 10 seconds. This application does not limit the frame extraction method.

[0064] S23: Obtain the audio text information corresponding to the audio data of the scene segment.

[0065] In the embodiments of this application, audio data corresponding to a scene segment is obtained based on the start and end times of the scene segment. Then, an automatic speech recognition system is used to perform audio-to-text processing on the audio data to obtain audio-text information.

[0066] In one embodiment of this application, obtaining audio text information by performing audio-to-text processing on audio data using an automatic speech recognition system may include the following steps: The input audio data undergoes format conversion, noise reduction, and muting, and is segmented into short frames. Parameters reflecting speech characteristics (e.g., Mel frequency cepstral coefficients, Mel filter bank energy, perceptual linear prediction, etc.) are extracted from each frame, transforming the original audio into a series of feature vectors. These feature vectors are input into an acoustic model (e.g., a deep neural network-based model), which outputs the probability distribution of phonemes, characters, etc., for each frame. Combined with a language model, the output of the acoustic model is searched and decoded to generate the most probable word or sentence sequence. To improve recognition accuracy, contextual information can be utilized. The recognized text results are then processed for punctuation restoration, format adjustment, and proper noun correction to obtain the audio text information.

[0067] In some embodiments, the language model may be a pre-trained large model based on the Transformer architecture, an N-gram language model, or a neural network language model, etc. For example, the preset learning model may be one of the following: a large language model, a chat generation pre-trained transformation model, a generative pre-trained transformation model, and a general language model.

[0068] S24: Obtain a text summary of the image content representing the scene fragment.

[0069] In one embodiment of this application, image frames of a scene segment are extracted; for each extracted image frame, image features are extracted using a preset image encoder; mean pooling is performed on the image features of all extracted image frames to obtain a visual feature vector; and the text summary is generated based on the visual feature vector using a preset text decoder.

[0070] Frame extraction can be performed on image frames of a scene segment using the methods described above.

[0071] In the embodiments of this application, for each image frame extracted from each scene, image features are extracted using a GIT image encoder. These image features are high-dimensional vectors, which can be vectors with high dimensionality, such as 512-dimensional, 768-dimensional, or 1024-dimensional vectors. These high-dimensional vectors include key information from the image frame.

[0072] Next, mean pooling is performed on the image features of all extracted image frames to obtain a vector representing the overall visual features of the scene. It should be understood that the n image frames extracted from the scene segment correspond to n image features, each with a dimension of D. Mean pooling is the averaging of these n image features along each dimension, resulting in a new D-dimensional vector, i.e., the visual feature vector. This D-dimensional vector is used to represent the visual features of the scene.

[0073] The visual feature vector is fed into a pre-defined text decoder to generate a text summary of the scene fragment. The pre-defined text decoder makes predictions step-by-step based on the input visual feature vector until the prediction ends. During the prediction process, the pre-defined text decoder jointly infers the next word based on previously generated words and visual features, ultimately obtaining a natural language text as the text summary of the scene fragment.

[0074] In the embodiments of this application, the text summary is a representation of the visual content of a scene segment, such as the main content involved in the scene segment, including information such as objects, actions, scenes, and products.

[0075] S25: Input the time-aligned text content, audio text information, and text summary into the preset learning model to determine whether the scene segment includes advertising information.

[0076] In the embodiments of this application, the preset learning model is one of the following: a large language model, a chat generation pre-trained transformation model, a generative pre-trained transformation model, and a general language model.

[0077] In this embodiment, by inputting the text content, audio text information, and text summary obtained in the aforementioned steps into a preset learning model, the accuracy of advertising information recognition can be improved. Furthermore, since the input to the preset learning model is all text information, recognition efficiency can be enhanced.

[0078] In this embodiment, advertising information may include one of the following: advertising brand, advertising slogan, advertising effect, advertising text, video clips featuring the advertising text, and advertising promotional information, etc., all advertising-related information. Therefore, in this embodiment, when advertising information is included in a scene clip, the advertising information is compared with a preset database to determine whether the advertising information has been declared. For example, based on the advertising brand and advertising slogan, it can be checked whether the advertisement is a user-added soft advertisement that has not been declared.

[0079] In some embodiments, a database can be established to store all advertising information that users have submitted. By comparing the identified advertising information with the database, it can be determined whether the advertising information has been submitted. Furthermore, if the advertising information has not been submitted, it may involve violations or non-compliance, and this information can be reported to the advertiser or regulatory platform. This allows for further actions such as reminding or penalizing the video provider, or restricting or prohibiting playback of the video.

[0080] In this embodiment of the application, the advertisement recognition method further includes: When playing a video to be identified, and the scene segment of the video to be identified includes advertising information, skip the scene segment based on the start and end times of the scene segment.

[0081] Therefore, the embodiments of this application can provide a "skip ad" function when the advertising information appears in the video during subsequent video playback, thereby improving the user's video viewing experience.

[0082] See Figure 3 In another embodiment of this application, the advertisement recognition method includes: S31: Determine the prompt word.

[0083] In embodiments of this application, the method further includes designing prompt words for input into a preset learning model. These prompt words provide identification instructions to guide the preset learning model in outputting corresponding recognition results based on the input text content, audio text information, and text summary. The recognition results include at least whether the scene fragment contains advertising information.

[0084] In this embodiment, the prompt can be a text description and can be designed according to actual needs. For example, the prompt could be "Please summarize whether the video contains advertising information based on the input information, extract the advertising brand, advertising slogan, advertising effect, advertising promotional information, etc., and explain the reasons." Based on this prompt, the preset learning model will output: whether the video contains advertising information, advertising-related information, and the reasons. Alternatively, the prompt could be "Please summarize the main content of the video based on the input information, and determine whether it contains advertising information based on the main content." In this case, the preset learning model will output: the main content of the video, and whether it contains advertising information.

[0085] S32: Input the prompt word, the text content, the audio text information, and the text summary into the preset learning model to determine whether the scene segment includes advertising information.

[0086] S33: Output the recognition results based on the prompt words, text content, audio text information, and text summary.

[0087] In this embodiment, prompt words can make the recognition results output by the preset learning model clearer and more comprehensive, making the recognition results readily apparent. Based on the aforementioned example, the output recognition results could be: 1. The video contains advertising information; 2. The advertising brand is XX; 3. The advertising slogan is "Enjoy a cool summer"; 4. The advertising promotion information is a limited-time buy-one-get-one-free offer; 5. Advertising effects are not included; 6. Reason: The brand name and promotional information appear multiple times in the clip, indicating clear advertising content.

[0088] In one embodiment of this application, the advertisement recognition method further includes: When advertising information is included in a scene segment, the duration for which the user watches the scene segment is obtained; If the overlap between the duration of the advertisement and the duration of the scene segment exceeds a preset duration, the advertisement information is determined to be a valid advertisement.

[0089] In the embodiments of this application, when a user watches or plays the video to be identified, the playback progress is recorded, and it is counted whether the user has watched a scene segment containing advertising information. If the overlap between the duration of watching the scene segment and the total duration of the scene segment exceeds a preset duration, the advertising information is determined to be valid advertising information. The preset duration can be 3 seconds, 5 seconds, etc. Therefore, the embodiments of this application can further determine whether the advertising information is valid after it has been identified. The duration of the scene segment can be determined based on the start and end times of the scene segment, as described in the previous embodiments. The duration of the user watching the scene segment can be determined by the recorded playback progress.

[0090] In this embodiment, the preset learning model can be trained using a training dataset. See also... Figure 4 The schematic diagram of the training process of the preset learning model in this application embodiment includes the following process: S41: Construct training data.

[0091] The training data includes input data and its corresponding target output. The input data includes text content, audio text information and text summary corresponding to scene segments in the video. The target output includes the recognition result of whether the scene segment contains advertising information.

[0092] S42: Initialize the network structure and parameters of the preset learning model.

[0093] In the embodiments of this application, the network structures and network parameters of different preset learning models are different. For example, for a large language model, its network structure includes an input layer, multiple Transform modules, and an output layer, and its network parameters include the weight parameters, layer normalization parameters, and output layer parameters for each layer.

[0094] S43: Input the training data into the initialized model, perform forward propagation, and obtain the model output results.

[0095] S44: Calculate the loss value between the model output and the target output.

[0096] S45: Based on the loss value, calculate the gradient of the model parameters using the backpropagation algorithm.

[0097] S46: Use optimization algorithms to update model parameters based on gradients.

[0098] S47: When the loss value converges to the set value or reaches the preset number of training rounds, save the trained model parameters and obtain the trained preset learning model.

[0099] It should be understood that the training process of the preset learning model in the embodiments of this application can be improved by adding or removing steps based on the above steps S41-S47, as long as a trained model can be obtained. The embodiments of this application do not impose any restrictions on this. For example, the training process of a large language model may also include word segmentation, embedding layer processing, positional encoding, batch training, distributed training, and other processing procedures.

[0100] In embodiments of this application, the advertising recognition method further includes: when a scene fragment contains advertising information, using the text content, audio text information, text summary, and the recognition result of the scene fragment containing advertising information as training data for a preset learning model.

[0101] The advertising recognition method of this application embodiment involves segmenting the video to be recognized into scenes to obtain at least one scene segment. For each scene segment in the video to be recognized: text recognition is performed on the image frames in the scene segment to obtain the text content contained in the scene segment; audio data corresponding to the scene segment is obtained according to the start and end times of the scene segment; audio-to-text processing is performed on the audio data to obtain the audio text information corresponding to the audio data; visual feature vectors representing the screen content of the scene segment are extracted based on the image frames in the scene segment; a text summary describing the screen content is generated based on the visual feature vectors; and the time-aligned text content, audio text information, and text summary are input into a preset learning model to determine whether the scene segment includes advertising information. Thus, a unified time-aligned text is constructed for recognition, which can improve recognition accuracy and efficiency. By introducing a preset learning model, contextual understanding and semantic reasoning can be used to identify advertising intent hidden in the plot, dialogue, and subtitles. Compared with related technologies, this method has more comprehensive semantic expression capabilities and interpretability. Furthermore, the identification method in this application embodiment, through joint identification of text content, audio text information, and text summaries of scene fragments, possesses the ability to identify both soft advertisements and undeclared advertisements. It can effectively identify "spoken" and "narrative" soft advertisements in unstructured video content and determine the accurate time range of the advertisements. It can support users skipping advertisements and platform statistical analysis, improving user experience and advertising supervision.

[0102] Based on the same technical concept, see [link / reference] Figure 5 This application also provides an advertisement recognition device, including: The video acquisition module 501 is used to acquire a video to be identified, the video to be identified including at least one scene segment, the scene segment including image frames and audio data; The text content acquisition module 502 is used to acquire the text content contained in the image frames of the scene fragment; The audio text information acquisition module 503 is used to acquire the audio text information corresponding to the audio data of the scene segment; The text summary acquisition module 504 is used to acquire a text summary representing the content of a scene fragment; The recognition module 505 is used to input time-aligned text content, audio text information and text summary into a preset learning model to determine whether the scene segment includes advertising information.

[0103] The modules described above correspond to the method steps; further details can be found in the method embodiments. The above description involves various modules. It should be noted that the division of these modules in the description is for clarity. However, in actual implementation, the boundaries between modules may be blurred. For example, any or all functional modules in this application may share various hardware and / or software elements. As another example, any and / or all functional modules in this application may be wholly or partially implemented by a shared processor executing software instructions. Furthermore, various software sub-modules executed by one or more processors may be shared among various software modules. Accordingly, unless explicitly required, the scope of this application is not limited by mandatory boundaries between various hardware and / or software elements.

[0104] Based on the same technical concept, embodiments of this application also provide a computing device, see reference. Figure 6 It includes a memory 601 and a processor 602. The memory 601 is used to store computer instructions. When the processor 602 executes the computer instructions, it implements the method steps in any method embodiment.

[0105] The specific entity of a computing device can be a server, which can be a rack server, blade server, tower server, or cabinet server (including a standalone server or a server cluster composed of multiple servers), etc.

[0106] The memory 601 includes at least one type of computer-readable storage medium, including flash memory, hard disk, multimedia card, random access memory (RAM), static random access memory (SRAM), read-only memory (ROM), magnetic disk, optical disk, etc. In some embodiments, the computer-readable storage medium may be an internal storage unit of an electronic device, such as the hard disk or memory of the electronic device. In other embodiments, the computer-readable storage medium may also be an external storage device of the electronic device, such as a plug-in hard disk, secure digital card (SD card), flash memory card, etc., equipped on the electronic device. Of course, the computer-readable storage medium may include both internal storage units and external storage devices of the electronic device. In this embodiment, the computer-readable storage medium is typically used to store the operating system and various application software installed on the electronic device, such as the program code of the advertising recognition method in the embodiment. In addition, the computer-readable storage medium may also be used to temporarily store various types of data that have been output or will be output.

[0107] In some embodiments, processor 602 may be a central processing unit (CPU), a controller, a microcontroller, a microprocessor, or other chip. This processor 1002 is typically used to control the overall operation of the processing device, such as performing control and processing related to data interaction or communication with other entities. In this embodiment, processor 602 is used to run program code stored in memory 601 or process data.

[0108] Based on the same technical concept, this application also provides a computer-readable storage medium, which includes a computer program or instructions stored in the storage medium. When the computer program or instructions are executed by a processing device, they implement the method steps in any method embodiment. Further details can be found in the method embodiments, which will not be repeated here. In this embodiment, the computer-readable storage medium can be non-volatile or volatile. Computer-readable storage media include flash memory, hard disk, multimedia card, random access memory (RAM), static random access memory (SRAM), read-only memory (ROM), magnetic disk, optical disk, etc. In some embodiments, the computer-readable storage medium can be an internal storage unit of an electronic device, such as the hard disk or memory of the electronic device. In other embodiments, the computer-readable storage medium can also be an external storage device of the electronic device, such as a plug-in hard disk, secure digital card (SD card), flash memory card, etc., equipped on the electronic device. Of course, the computer-readable storage medium can also include both internal storage units and external storage devices of the electronic device. In this embodiment, the computer-readable storage medium is typically used to store the operating system and various application software installed on the electronic device, such as the program code of the advertising recognition method in the embodiment. Furthermore, the computer-readable storage medium can also be used to temporarily store various types of data that have been output or will be output.

[0109] Based on the same technical concept, embodiments of this application also provide a computer program product or computer program, which includes computer instructions stored in a computer-readable storage medium. A processor of a computer device reads the computer instructions from the computer-readable storage medium and executes the computer instructions, causing the computer device to perform the advertising recognition method provided in the above-described method embodiments.

[0110] It should be noted that the order of description of the embodiments in this application is not intended to limit the priority of the embodiments.

[0111] Unless otherwise defined, all technical and scientific terms used in this application have the same meaning as commonly understood by one of ordinary skill in the art to which this application belongs. The terminology used in this application and in its specification is for the purpose of describing particular embodiments only and is not intended to be limiting of the application.

[0112] The embodiments of this application have been described above with reference to the accompanying drawings. However, this application is not limited to the specific embodiments described above. The specific embodiments described above are merely illustrative and not restrictive. Those skilled in the art can make many forms under the guidance of this application without departing from the spirit and scope of protection of the claims. All equivalent transformations made under the inventive concept of this application using the content of this application's specification and drawings, or direct / indirect applications in other related technical fields, are included within the patent protection scope of this application.

Claims

1. An advertisement recognition method, characterized in that, include: Acquire a video to be identified, the video to be identified including at least one scene segment, the scene segment including image frames and audio data; Obtain the text content contained in the image frames of the scene fragment; Obtain the audio text information corresponding to the audio data of the scene segment; Obtain a text summary representing the image content of the scene segment; The time-aligned text content, the audio text information, and the text summary are input into a preset learning model to determine whether the scene segment includes advertising information.

2. The method according to claim 1, characterized in that, The method further includes: The scene switching point is determined based on the inter-frame similarity of the image frames in the video to be identified, wherein the next frame in the adjacent video frames with an inter-frame similarity lower than a preset threshold is taken as the scene switching point. Based on the scene switching point, the video to be identified is segmented into scenes to obtain at least one scene segment.

3. The method according to claim 2, characterized in that, The text content contained in the image frame of the scene fragment includes: Frames are extracted from the image frames of the scene segment; Text recognition is performed on the image frames obtained by frame extraction to obtain the text content.

4. The method according to claim 3, characterized in that, The step of obtaining the audio text information corresponding to the audio data of the scene segment includes: Based on the start and end times of the scene segment, obtain the audio data corresponding to the scene segment; The audio data is processed by an automatic speech recognition system to convert it into text, thereby obtaining the audio text information.

5. The method according to claim 4, characterized in that, The process of obtaining a text summary representing the content of the scene fragment includes: Frames are extracted from the image frames of the scene segment; For each extracted image frame, image features are extracted using a preset image encoder; The image features of all extracted image frames are averaged and pooled to obtain the visual feature vector. The text summary is generated based on the visual feature vector using a preset text decoder.

6. The method according to any one of claims 1-5, characterized in that, The method further includes: The prompt words, along with the text content, the audio text information, and the text summary, are input into the preset learning model. The prompt words are used to provide recognition instructions, guiding the preset learning model to output corresponding recognition results based on the input text content, the audio text information, and the text summary. Based on the prompt words, the text content, the audio text information, and the text summary, the recognition result is output, and the recognition result includes at least whether the scene segment contains advertising information.

7. The method according to any one of claims 1-5, characterized in that, The method further includes: When the scene segment contains advertising information, the text content, the audio text information, the text summary, and the recognition result of the scene segment containing advertising information are used as training data for the preset learning model.

8. The method according to any one of claims 1-5, characterized in that, The method further includes: When the scene segment includes advertising information, the duration for which the user watches the scene segment is obtained; When the overlap between the duration of the advertisement and the duration of the scene segment exceeds a preset duration, the advertisement information is determined to be a valid advertisement.

9. The method according to any one of claims 1-5, characterized in that, The method further includes: When advertising information is included in the scene segment, the advertising information is compared with a preset database to determine whether the advertising information has been declared, wherein the preset database includes advertising information that has already been declared.

10. The method according to any one of claims 1-5, characterized in that, The method further includes: When the video to be identified is played, and the scene segment includes advertising information, the scene segment is skipped according to its start and end times.

11. The method according to any one of claims 1-5, characterized in that, The preset learning model is one of the following: a large language model, a chat generation pre-trained transformation model, a generative pre-trained transformation model, and a general language model.

12. An advertising recognition device, characterized in that, include: A video acquisition module is used to acquire a video to be identified, the video to be identified including at least one scene segment, the scene segment including image frames and audio data; The text content acquisition module is used to acquire the text content contained in the image frames of the scene fragment; The audio text information acquisition module is used to acquire the audio text information corresponding to the audio data of the scene segment; The text summary acquisition module is used to acquire a text summary representing the image content of the scene segment; The recognition module is used to input the time-aligned text content, the audio text information, and the text summary into a preset learning model to determine whether the scene segment includes advertising information.

13. A computing device, characterized in that, It includes a memory and a processor, the memory being used to store computer programs or instructions; when the computer programs or instructions are executed by the processor, the method of any one of claims 1-11 is implemented.

14. A computer-readable storage medium, characterized in that, The storage medium stores a computer program or instructions, which, when executed by a processor, implement the method of any one of claims 1-11.

15. A computer program product, comprising a computer program, characterized in that, When the computer program is executed by a processor, it implements the steps of the method according to any one of claims 1-11.

Images