A method, device, chip and storage medium for real-time replacement of advertising programs

Abstract

Embodiments of the present application disclose a method, apparatus, electronic device, chip, and computer-readable storage medium for real-time replacement of advertising programs. Among them, the method includes: constructing and training an image feature extraction network; continuously using the image feature extraction network to extract image features from frame images of a target program, and matching the extracted image features to be matched of the frame images with a feature library to obtain a matching result of the frame images; if the matching results of consecutive M frame images all meet a preset condition, it is determined that there is a first advertising program to be replaced in the target program, where M is a positive integer; based on a preset replacement rule, replacing the first advertising program to be replaced with a first new advertising program.

Description

Technical Field

[0001] The present application relates to the technical field of video processing, and in particular to a method, apparatus, electronic device, chip and computer-readable storage medium for real-time replacement of advertising programs. Background Art

[0002] In the process of relaying TV programs to new media programs, there is a need for real-time replacement of advertising content. At present, there are mainly two ways to replace advertising programs in real time: manual replacement and automatic replacement. Manual replacement is carried out by relevant personnel observing the program to be processed, searching for replaceable time periods, and switching the advertising content when the replaced advertisement is found. Facing a large amount of program content, this manual replacement method relying on manual labor will, on the one hand, consume a large amount of labor costs, and on the other hand, it is difficult to ensure the accuracy and replacement efficiency of the replacement, which may lead to losses in broadcast benefits and even potential safety hazards. Automatic replacement is mainly achieved based on the detection of specific identifiers or features, with low accuracy, poor stability, and prone to situations such as missed replacement and wrong replacement, and generally cannot meet the real-time replacement requirements. Summary of the Invention

[0003] To solve the above technical problems, embodiments of the present application provide a method, apparatus, electronic device, chip and computer-readable storage medium for real-time replacement of advertising programs.

[0004] The technical solution of the embodiments of the present application is realized as follows:

[0005] In a first aspect, an embodiment of the present application provides a method for real-time replacement of advertising programs, where the method for real-time replacement of advertising programs includes:

[0006] Construct and train an image feature extraction network;

[0007] Continuously extract image features of the frame images of the target program by using the image feature extraction network, and match the extracted image features to be matched of the frame images with a feature library to obtain a matching result of the frame images;

[0008] If the matching results of M consecutive frame images all meet a preset condition, it is determined that there is a first advertising program to be replaced in the target program, where M is a positive integer;

[0009] Based on a preset replacement rule, replace the first advertising program to be replaced with a first new advertising program.

[0010] In a second aspect, an embodiment of the present application provides an apparatus for real-time replacement of advertising programs, where the apparatus includes:

[0011] A construction module: used to construct and train an image feature extraction network;

[0012] Extraction module: Continuously extract image features from the frame images of the target program by using the image feature extraction network;

[0013] Matching module: Used to match the image features to be matched of the extracted frame images with the feature library to obtain the matching result of the frame images;

[0014] Determination module: Used to determine that there is a first replaced advertising program in the target program if the matching results of consecutive M frame images all meet the preset conditions, where M is a positive integer;

[0015] Replacement module: Based on the preset replacement rule, replace the first replaced advertising program with the first new advertising program.

[0016] In a third aspect, the present application provides an electronic device, including: a processor and a memory. The memory is used to store a computer program, and the processor is used to call and run the computer program stored in the memory to execute any one of the advertising program real-time replacement methods provided by the embodiments of the present application.

[0017] In a fourth aspect, the present application provides a chip, including: a processor, used to call and run a computer program from a memory, so that a device installed with the chip executes any one of the advertising program real-time replacement methods provided by the embodiments of the present application.

[0018] In a fifth aspect, the present application provides a computer-readable storage medium, used to store a computer program, and the computer program enables a computer to execute any one of the advertising program real-time replacement methods provided by the embodiments of the present application.

[0019] For the technical solutions provided by the embodiments of the present application, on the one hand, with the help of deep convolutional neural network technology, real-time extraction of frame-level features is achieved, and the efficiency is higher. At the same time, the features extracted by the convolutional neural network are more effective, and the matching accuracy is higher. At the same time, due to the use of the frame-level matching strategy, the accuracy of advertising replacement is better, and the probability of missing replacement or mis-replacing frames is smaller. Description of the Drawings

[0020] Figure 1 It is a schematic flowchart of the advertising program real-time replacement method provided by the embodiments of the present application;

[0021] Figure 2 It is a schematic diagram of reliability determination provided by the embodiments of the present application;

[0022] Figure 3 It is a schematic diagram of Application Example 1 of the advertising program real-time replacement method provided by the present application;

[0023] Figure 4It is a schematic flowchart of real-time extraction of frame image features in Application Example 1 provided by this application;

[0024] Figure 5 It is a schematic flowchart of fast matching of frame image features in Application Example 1 provided by this application;

[0025] Figure 6 It is a schematic structural diagram of an advertisement program real-time replacement device provided by an embodiment of this application;

[0026] Figure 7 It is a schematic structural diagram of an electronic device provided by an embodiment of this application;

[0027] Figure 8 It is a schematic structural diagram of a chip provided by an embodiment of this application. Detailed implementation manners

[0028] Next, the technical solutions in the embodiments of this application will be described in conjunction with the accompanying drawings in the embodiments of this application. Obviously, the described embodiments are part of the embodiments of this application, rather than all the embodiments. Based on the embodiments in this application, all other embodiments obtained by those of ordinary skill in the art without creative efforts shall fall within the protection scope of this application.

[0029] It should be noted that in the embodiments of this application, the term "and / or" only describes the association relationship of associated objects and indicates that there can be three relationships. For example, A and / or B can represent: A exists alone, A and B exist simultaneously, and B exists alone. In addition, in the embodiments of this application, the character " / " generally represents an "or" relationship between the associated objects before and after.

[0030] In the description of the embodiments of this application, the term "corresponding" can represent a direct or indirect corresponding relationship between two parties, can also represent an association relationship between two parties, or can be a relationship such as indication and being indicated, configuration and being configured, etc.

[0031] To facilitate understanding of the technical solutions of the embodiments of this application, the related technologies of the embodiments of this application are described below. The following related technologies can be arbitrarily combined with the technical solutions of the embodiments of this application as optional solutions, and they all fall within the protection scope of the embodiments of this application.

[0032] Watching TV programs is one of the main forms of entertainment in people's daily lives. Traditional radio and TV programs are generally sent out by TV stations and transmitted to various places for broadcasting. With the development of the Internet and multimedia technologies, various new media such as the Internet and mobile phones have rapidly emerged. The audio-visual dissemination channels have become more diverse, the user group has become larger, and user interests have become more diversified. Traditional radio and TV media and new media are gradually moving towards a new stage of integrated development. As a channel operating independently from radio and TV, a large amount of TV program content is included in the program source of new media, which has given rise to the need for advertising program replacement. For example, replacing the advertising content in TV programs to carry out independent advertising operation on the new media side, replacing or adding cooperative advertisers, or conducting targeted advertising according to user interests, etc. There are mainly two ways of advertising program replacement: manual replacement and automatic replacement. Manual replacement relies on manual observation and search to complete the replacement. For the current massive program content, especially in the new media field where the same program may have multiple resolution and bitrate forms, relying on manual replacement of program segments will consume a large amount of human resources, with low efficiency and unable to guarantee the accuracy of replacement. Therefore, it is very necessary to achieve automatic replacement of program segments. The following introduces several automatic advertising program replacement schemes:

[0033] Scheme A: Based on the detection of specific identifiers (such as black and white frames, still frames, station logos, trademarks, etc.). For example, for advertising content, black frames, white frames or still frames often appear at the beginning and end. The positions of black and white frames can be detected by statistically analyzing color features such as brightness and chromaticity; the positions of still frames can be detected by calculating the inter-frame difference information, and this is used as a basis to determine the start and end of advertising content. Another example is that when a TV station broadcasts normal programs, it will display the station logo information, while when broadcasting advertisements, the station logo will be hidden. Based on this rule, the detection of the station logo can be used to determine whether it is advertising content. The detection of the station logo can be achieved through methods such as cumulative frame difference or edge detection.

[0034] Scheme B: Based on the detection of traditional audio-visual features. For example, in order to attract the attention of the audience, advertising content usually has characteristics such as bright colors and fast shot switching. And some rendering effects are also added during the advertising production process. Therefore, this type of method generally first uses edge detection technology to cut the video into shots, extracts key frames, and extracts the spatial features (such as average brightness, color histogram, etc.) and temporal features (such as shot change rate, inter-frame difference, etc.) of the key frames. More complexly, manually designed feature operators such as the Scale-invariant feature transform (SIFT) algorithm can be used to achieve multi-scale information extraction. Then, by setting appropriate threshold parameters to distinguish normal video content from target video content. Audio features mainly include temporal features (such as short-time energy, short-time zero-crossing rate, etc.) and frequency domain features (such as spectrum, power spectrum, etc.).

[0035] Solution C: Based on target recognition, with the help of machine learning or deep learning technologies, identify targets such as human faces, texts, and commodities. For example, determine whether it is replacement content by identifying commodities in advertisements. This type of method requires selecting different recognition models according to the actual application background. By collecting a large amount of image data related to the target, building a suitable recognition network, and training a high-precision target recognizer. The program to be processed passes through the target recognizer, and it returns whether each frame or key frame contains a specific target to guide the replacement of program segments.

[0036] The defects existing in the above solutions are as follows:

[0037] Solution A: The disadvantage of this method is the poor generality of the relied-on identifiers. First of all, identifiers such as black-and-white frames and static frames are easily removed by TV stations, and some normal programs (such as TV dramas and movies) may also contain black-and-white frames and static frames; secondly, for TV station logos, at present, many TV stations no longer hide their logos when broadcasting advertisements, and the complication of the production techniques and presentation forms of TV station logos and trademarks (such as transparency, animation effects, etc.) brings greater challenges to detection, and it is thus difficult to ensure the accuracy of recognition. Therefore, the method based on specific identifier recognition is not applicable to the replacement of all TV stations and TV programs.

[0038] Solution B: In addition to simple hard cuts, the shot transition forms of TV programs also include fade transition forms (such as fade-in, fade-out, dissolve, etc.). Shot cutting and key frame extraction cannot be completely accurate; for the extracted features, generally, multiple threshold parameters are set for judgment. For example, if the shot shear rate is higher than the threshold, it is considered an advertisement. Many features are sensitive to threshold parameters, and it is generally difficult to find a unified threshold to adapt to various program contents; for manually designed complex features, the calculation process is complex and time-consuming, and it is difficult to meet the real-time requirement.

[0039] Solution C: When similar targets appear in normal programs, incorrect replacement is likely to occur; the target recognizer needs to be pre-trained. Whenever a new target is added, a large amount of training data needs to be collected again, and training needs to be completed again, with a high time cost and poor flexibility.

[0040] In view of the defects existing in the above related technologies, the following technical solutions of the embodiments of the present application are proposed.

[0041] In order to be able to understand the features and technical content of the present application in more detail, the implementation of the present application will be elaborated in detail below with reference to the accompanying drawings. The attached drawings are only for reference and explanation purposes and are not used to limit the present application.

[0042] Figure 1 It is a schematic diagram of the implementation process of an advertisement program real-time replacement method provided by the embodiments of the present application, as Figure 1As shown in the figure, an embodiment of the present application provides a method for real-time replacement of advertising programs. The method includes:

[0043] Step 101: Construct and train an image feature extraction network.

[0044] Here, an image feature extraction network can be constructed using a Visual Geometry Group Network (VGG) model, a Residual Network (ResNet) model, or a MobileNet model. An image feature extraction network can also be constructed through other networks according to actual needs, which will not be elaborated here. After constructing the image feature extraction network, it can be trained using existing advertising programs until its extraction accuracy meets the preset extraction accuracy.

[0045] Step 102: Continuously extract image features of the frame images of the target program using the image feature extraction network, and match the to-be-matched image features of the extracted frame images with the feature library to obtain the matching result of the frame images.

[0046] Here, the target program can be a live or rebroadcast TV program by a TV station, or an existing stored TV program. The feature library can use an existing feature library or can be constructed according to actual needs to meet the current application scenario.

[0047] Specifically, for the method for real-time replacement of advertising programs provided by an embodiment of the present application, before continuously extracting image features of the frame images of the target program using the image feature extraction network, it further includes:

[0048] Extract the image features of each frame of all the replaced advertising programs using the image feature extraction network;

[0049] Construct the feature library based on the image features of each frame of all the replaced advertising programs.

[0050] Furthermore, for the method for real-time replacement of advertising programs provided by another embodiment of the present application, before extracting the image features of each frame of all the replaced advertising programs, it further includes:

[0051] Perform a preprocessing operation on each frame image of each replaced advertising program in all the replaced advertising programs so that each frame image of each replaced advertising program conforms to the input format of the image feature extraction network.

[0052] Here, the image feature extraction network constructed based on different algorithms or neural networks has different requirements for the input video format. Therefore, the video input to the image feature extraction network is first preprocessed so that it can be adapted to the image feature extraction network. The preprocessing operations include but are not limited to color space conversion and downsampling operations.

[0053] Here, the image feature extraction network can be used to perform forward inference and max-pooling dimensionality reduction processing on the image data of the replaced advertisement program after preprocessing, and extract a one-dimensional feature vector.

[0054] Based on this, in the advertisement program real-time replacement method provided by another embodiment of the present application, the step of using the image feature extraction network to extract the image features of each frame of all the replaced advertisement programs includes: processing the image data of each replaced advertisement program that has been preprocessed and input to the image feature extraction network, and extracting the one-dimensional feature vector of each frame of image.

[0055] After obtaining the one-dimensional feature vectors of each frame of all the replaced advertisement programs, a feature library is constructed based on the one-dimensional feature vectors of each frame of all the replaced advertisement programs extracted.

[0056] The feature library constructed by the solution provided by the embodiment of the present application only contains the features of all the replaced advertisement programs, avoiding unnecessary matching processes and improving the matching efficiency.

[0057] Further, a clustering method can be used to cluster the features in the feature library to obtain an atomic feature set, and each atomic feature corresponds to a cluster formed by several image features. Here, the clustering method can adopt the k-means (K-Means) clustering algorithm, the mean shift clustering method, the agglomerative hierarchical clustering method, etc., and the present application does not limit this.

[0058] Based on this, in the advertisement program real-time replacement method provided by an embodiment of the present application, after constructing the feature library, all the image features in the feature library are feature-clustered to obtain N atomic feature sets, and each atomic feature corresponds to a cluster formed by several image features; where N is a positive integer.

[0059] The real-time advertisement program replacement method provided by the embodiment of the present application sequentially and continuously extracts frame images according to the time domain of the target program, and matches the extracted frame images with the feature library to obtain a matching result. When there is no matching result after the extracted frame image is matched with the feature library, it indicates that the current frame does not belong to the program to be replaced. When there is a matching result after the extracted frame image is matched with the feature library, it indicates that the current frame belongs to one of the programs to be replaced in the program to be replaced. Further, when M consecutive matching results all meet the preset conditions, it is determined that there is a first program to be replaced in the current program. For example, when M = 5, that is, when the matching results of 5 consecutive frames meet the preset conditions, it can be determined that there is a first replaced advertisement program in the current target program, and subsequent advertisement program replacement operations can be performed to improve the recognition efficiency. The preset conditions may be: the matching results of M consecutive frame images all belong to the first replaced advertisement program and / or the time sequence of the frame images corresponding to the matching results of M consecutive frame images for the target program is the same as the time sequence of the M matching results corresponding to the first replaced advertisement program.

[0060] Based on this, in the real-time advertisement program replacement method provided by another embodiment of the present application, the preset conditions include at least one of the following conditions:

[0061] The first condition, the first condition is that the matching results of the M consecutive frame images all belong to the first replaced advertisement program;

[0062] The second condition, the second condition is that the time sequence of the frame images corresponding to the matching results of the M consecutive frame images for the target program is the same as the time sequence of the M matching results corresponding to the first replaced advertisement program.

[0063] In the real-time advertisement program replacement method provided by another embodiment of the present application, before continuously performing image feature extraction on each frame image of the target program by using the image feature extraction network, it further includes:

[0064] Performing a preprocessing operation on the extracted frame images of the target program to make the extracted frame images of the target program conform to the input format of the image feature extraction network.

[0065] Here, image feature extraction networks constructed according to different algorithms or neural networks have different requirements for the input video format. Therefore, the video input to the image feature extraction network is first preprocessed so that it can be adapted to the image feature extraction network. The preprocessing operations include but are not limited to color space conversion and downsampling operations.

[0066] Here, the image feature extraction network can be used to perform forward inference and max-pooling dimensionality reduction processing on the image data of the input video after preprocessing to extract a one-dimensional feature vector.

[0067] Based on this, for the real-time advertisement program replacement method provided by another embodiment of this application, continuously extracting image features from the frame images of the target program by using the image feature extraction network includes:

[0068] Continuously processing each preprocessed frame image of the target program extracted by using the image feature extraction network to extract a one-dimensional feature vector of each frame image.

[0069] Specifically, after obtaining the image features to be matched of the frame images of the target program, first perform a rough match between each image feature to be matched and the feature library, including: calculating the similarity measure between the image features to be matched of the frame image and each atomic feature set in the feature library to obtain N atomic feature similarity measure calculation results, and comparing each atomic feature similarity measure calculation result among the N atomic feature similarity measure calculation results with a preset first threshold. If all N atomic feature similarity measure calculation results are less than the preset first threshold, it is determined that this frame image does not belong to the advertisement program to be replaced;

[0070] Then perform a fine match. Specifically, if there is a calculation result greater than or equal to the preset first threshold among the N atomic feature similarity measure calculation results, then calculate the similarity measure between the image features to be matched of the frame image and each image feature in the cluster corresponding to the atomic feature set corresponding to the calculation result greater than or equal to the preset first threshold among the N atomic feature similarity measure calculation results to obtain K image feature similarity measure calculation results. If each image feature similarity measure calculation result among the K image feature similarity measure calculation results is less than a preset second threshold, it is determined that this frame image does not belong to the advertisement program to be replaced; if there is a calculation result greater than or equal to the preset second threshold among the K image feature similarity measure calculation results, it is determined that this frame image belongs to the advertisement program to be replaced, and the frame image in the feature library corresponding to the calculation result greater than the preset second threshold among the K image feature similarity measure calculation results is the finally obtained matching result, where K is a positive integer.

[0071] Here, the preset first threshold and the preset second threshold satisfy the relationship that the preset first threshold is less than the preset second threshold. The preset second threshold is used to eliminate the rough match error and obtain a highly matching result because the frame images that meet the rough match conditions may not belong to the advertisement to be replaced. The calculation method for the above similarity measure can be calculated based on the cosine similarity. It can also be calculated based on other distance formulas, and this application does not limit this.

[0072] Based on this, for the real-time advertisement program replacement method provided by an embodiment of the present application, the matching of the image features to be matched of each frame of image with the feature library includes:

[0073] Performing a similarity measurement calculation on the image features to be matched of the frame image with each atomic feature set in the feature library to obtain N atomic feature similarity measurement calculation results, and comparing each atomic feature similarity measurement calculation result in the N atomic feature similarity measurement calculation results with a preset first threshold. If all N atomic feature similarity measurement calculation results are less than the preset first threshold, it is determined that this frame of image does not belong to the advertisement program to be replaced;

[0074] If there is a calculation result greater than or equal to the preset first threshold among the N atomic feature similarity measurement calculation results, then performing a similarity measurement calculation on the image features to be matched of the frame image with each image feature in the cluster corresponding to the atomic feature set corresponding to the calculation result greater than or equal to the preset first threshold among the N atomic feature similarity measurement calculation results to obtain K image feature similarity measurement calculation results. If each image feature similarity measurement calculation result in the K image feature similarity measurement calculation results is less than the preset second threshold, it is determined that this frame of image does not belong to the advertisement program to be replaced; if there is a calculation result greater than or equal to the preset second threshold among the K image feature similarity measurement calculation results, it is determined that this frame of image belongs to the advertisement program to be replaced, and the frame image in the feature library corresponding to the calculation result greater than or equal to the preset second threshold among the K image feature similarity measurement calculation results is the finally obtained matching result, where K is a positive integer.

[0075] Further, after obtaining the image features to be matched of the frame image of the target program, performing a rough match on each of the image features to be matched with the feature library first, including: performing a similarity measurement calculation on the image features to be matched of the frame image with each atomic feature set in the feature library to obtain N atomic feature similarity measurement calculation results, and comparing each atomic feature similarity measurement calculation result in the N atomic feature similarity measurement calculation results with a preset third threshold. If all N atomic feature similarity measurement calculation results are greater than the preset third threshold, it is determined that this frame of image does not belong to the advertisement program to be replaced;

[0076] Then perform fine matching. Specifically, if there is a calculation result less than or equal to a preset third threshold among the calculation results of the similarity metrics of N atomic features, then calculate the similarity metrics between the image features to be matched of the frame image and each image feature in the cluster corresponding to the set of atomic features corresponding to the calculation results less than or equal to the preset third threshold among the calculation results of the similarity metrics of N atomic features, obtaining J calculation results of image feature similarity metrics. If each calculation result of the J calculation results of image feature similarity metrics is greater than a preset fourth threshold, it is determined that this frame image does not belong to the replaced advertisement program; if there is a calculation result less than or equal to the preset fourth threshold among the J calculation results of image feature similarity metrics, it is determined that this frame image belongs to the replaced advertisement program, and the frame image in the feature library corresponding to the calculation result less than or equal to the preset fourth threshold among the J calculation results of image feature similarity metrics is the finally obtained matching result, where J is a positive integer.

[0077] Here, the preset third threshold and the preset fourth threshold satisfy the relationship that the preset fourth threshold is less than the preset third threshold. The preset fourth threshold is used to eliminate the coarse matching error and obtain a highly matching result, because the frame images that meet the coarse matching conditions may not belong to the replaced advertisement. For the above calculation method of similarity metrics, it can be based on Manhattan Distance, Euclidean Distance, etc., and this application does not limit it. Taking the Euclidean distance as an example, the calculation formula of the Euclidean distance is as follows: where x and y are the feature vectors to be matched and the target feature vector, both of which are one-dimensional vectors of length Q, and x j and y j are the j-th components of the vector.

[0078] Based on this, in the advertisement program real-time replacement method provided by an embodiment of this application, the step of matching the image features to be matched of each frame image with the feature library includes:

[0079] Calculate the similarity metrics between the image features to be matched of the frame image and each atomic feature set in the feature library, obtaining N calculation results of atomic feature similarity metrics, and compare each calculation result of the N calculation results of atomic feature similarity metrics with the preset third threshold. If all N calculation results of atomic feature similarity metrics are greater than the preset third threshold, it is determined that this frame image does not belong to the replaced advertisement program;

[0080] If there is a calculation result less than or equal to a preset third threshold among the calculation results of the N atomic feature similarity metrics, then perform a similarity metric calculation between the image feature to be matched in the frame image and each image feature in the cluster corresponding to the set of atomic features corresponding to the calculation results less than or equal to the preset third threshold among the N atomic feature similarity metric calculation results, obtaining J calculation results of the image feature similarity metric. If each calculation result of the J calculation results of the image feature similarity metric is greater than a preset fourth threshold, then it is determined that this frame image does not belong to the replaced advertisement program; if there is a calculation result less than or equal to the preset fourth threshold among the J calculation results of the image feature similarity metric, then it is determined that this frame image belongs to the replaced advertisement program, and the frame image in the feature library corresponding to the calculation result less than or equal to the preset fourth threshold among the J calculation results of the image feature similarity metric is the finally obtained matching result, where J is a positive integer.

[0081] Step 103: If the matching results of M consecutive frame images all meet the preset conditions, then it is determined that there is a first replaced advertisement program in the target program, where M is a positive integer.

[0082] After the image feature matching in step 102, it can be basically determined whether the current frame of the target program is a certain frame of the replaced advertisement program, but there is still a small probability of matching errors. In addition, the program content is rich and diverse, and there is a small probability that there is a frame content that does not belong to the replaced advertisement and is very similar to a certain frame of the replaced advertisement, resulting in false detection. Therefore, there is still a small probability of missed detection and false detection in the extraction and matching of single-frame image features.

[0083] Therefore, it is necessary to determine the reliability of the M matching results. Refer to Figure 2 , Figure 2 is a schematic diagram of reliability determination provided by an embodiment of the present application. As Figure 2 shown, based on the matching results of several frames before and after the current frame to determine the reliability of the current frame matching result. Specifically: when the matching result of a single frame or a short sequence of consecutive frames is inconsistent with the replaced program segments before and after it, it can be determined that the matching result of the current frame is an unreliable matching result; when the corresponding time sequence of a single frame or a short sequence of consecutive frames in the replaced advertisement is inconsistent with the time sequence before and after it, with reverse order or time sequence interval not conforming, it can be determined that the matching result of the current frame is an unreliable matching result. For example, when a = 2, when the matching result passes through the T-th frame, based on the matching results of the (T - 1)-th frame, (T - 2)-th frame, (T + 1)-th frame, (T + 2)-th frame and the time sequence of the (T - 1)-th frame, (T - 2)-th frame, (T + 1)-th frame, (T + 2)-th frame in the replaced advertisement, determine the reliability of the matching result at the T-th frame. In this way, time domain optimization can be performed through a small number of cached frames.

[0084] Based on this, for the real-time advertisement program replacement method provided by another embodiment of the present application, the reliability determination of the K matching results includes:

[0085] When the matching result of a single frame or consecutive i frames of the to-be-matched images is inconsistent with the replaced program segments before and after it, it is determined as an unreliable matching result;

[0086] When the time sequence corresponding to the matching result of a single frame or consecutive i frames in the replaced advertisement is inconsistent with the time sequences before and after it, it is determined as an unreliable matching result; where i is a positive integer.

[0087] Furthermore, for the real-time advertisement program replacement method provided by another embodiment of the present application, after the matching result of the current frame is determined as an unreliable matching result, its matching result is corrected according to a preset correction strategy.

[0088] Specifically, for the frame image determined as an unreliable matching result, the reliable matching result of the frame image before or after it is used to replace its matching result.

[0089] Based on this, for the real-time advertisement program replacement method provided by another embodiment of the present application, the correction of its matching result according to the preset correction strategy includes:

[0090] The reliable matching result of the frame image before or after the unreliable matching result is used to replace its matching result.

[0091] Step 104: Based on the preset replacement rule, replace the first replaced advertisement program with the first new advertisement program.

[0092] Here, in the preset replacement rule, the relationship between the new advertisement program and the replaced advertisement can be a one-to-one correspondence, a one-to-many correspondence, or the correspondence between the new advertisement program and the replaced advertisement program can be specified by using an advertisement replacement mapping table. Each new advertisement program in the new advertisement program and its corresponding replaced advertisement program satisfy the relationship of consistent number of frames, that is, frame-by-frame correspondence.

[0093] Based on this, for the real-time advertisement program replacement method provided by an embodiment of the present application, in the preset replacement rule, the new advertisement program and the replaced advertisement program have a corresponding relationship, and each new advertisement program in the new advertisement program and its corresponding replaced advertisement program have a one-to-one correspondence of image frames.

[0094] The real-time advertisement program replacement method provided by the embodiments of the present application has its main advantages reflected in the high efficiency and accuracy of advertisement replacement. On the one hand, by means of the deep convolutional neural network technology, the real-time extraction of frame-level features is realized, which is more efficient compared with the traditional manually designed feature extraction methods. At the same time, a series of optimization means such as feature dimensionality reduction and clustering accelerate the feature matching efficiency. On the other hand, the features extracted by the convolutional neural network are more effective and the matching accuracy is higher. At the same time, due to the use of the frame-level matching strategy and the time-domain correction operation, the accuracy of advertisement replacement is better, and the probability of missing or misplacing frames is smaller compared with the replacement completed by the traditional methods of shot cutting and key frame extraction.

[0095] For the real-time advertisement program replacement method provided by the embodiments of the present application, first, frame-level feature extraction is performed on the video content of the input program, and the extracted features are quickly matched in a preset feature library; then, the time-domain correction is completed in combination with the matching results of adjacent frames to improve the matching accuracy; finally, according to the preset replacement rules, the advertisement program replacement is completed. This method mainly involves key technologies such as image feature extraction and feature quick matching. Among them, the image feature extraction link should ensure that the extracted features have high distinctiveness, and all the above technical links meet the real-time requirements. In addition, the embodiments of the present application select a general image feature extraction network model, which only targets the image itself and is independent of the scene where it is located, and has strong versatility.

[0096] Reference Figure 3 , Figure 3 is a schematic diagram of Application Example 1 of the real-time advertisement program replacement method provided by the present application. As Figure 3 shown, for the target program being played, frame image features of the target program are continuously extracted, and the frame images of the input target program video are subjected to real-time feature extraction. Then, the extracted frame image feature vectors are quickly matched with the preset feature library; next, the frame image matching results are optimized in the time domain to filter out some false detections and missed detections; finally, based on the final matching results, it is determined whether to perform advertisement replacement. If the final matching result indicates that there is no advertisement to be replaced in the target program, no advertisement replacement is performed. If the matching result indicates that there is an advertisement to be replaced in the target program, the advertisement replacement is completed according to the preset replacement rules, that is, the new advertisement segment replaces the advertisement to be replaced to obtain the processed program. In this way, the real-time replacement of advertisement programs in TV programs is realized.

[0097] Reference Figure 4 , Figure 4 is a schematic flow diagram of the real-time extraction of frame image features in Application Example 1 above of the present application. As Figure 4As shown, the frame images of the input video are first subjected to an image preprocessing process to make them meet the input format of the image feature extraction network. Then, the preprocessed frame images are subjected to image feature extraction to obtain frame image feature vectors.

[0098] Reference Figure 5 , Figure 5 is a schematic flow diagram of the fast matching of frame image features provided by the above application example of the present application. As Figure 5 shown, the image features of all frames of all replaced advertisements are stored in the preset feature library. These image features are clustered to form an atomic feature set, and each atomic feature corresponds to a cluster formed by several image features. Each frame image feature vector extracted from the input target program is first roughly matched with the atomic feature set. If there is a highly matching atomic feature, then the feature to be matched is finely matched with the features in the cluster corresponding to the atomic feature to obtain the frame image matching result. Among them, the matching process can be performed through similarity measurement, and the calculation method for similarity measurement can adopt Minkowski Distance, Manhattan Distance, or Euclidean Distance.

[0099] To implement the advertisement program real-time replacement method described in the application embodiment of the present application, the embodiment of the present application also provides a device 600 for advertisement program real-time replacement; Figure 6 is a schematic structural diagram of the device 600 for advertisement program real-time replacement provided by the embodiment of the present application. As Figure 6 shown, the device 600 for advertisement program real-time replacement provided by the application embodiment of the present application includes:

[0100] Construction module 601: used to construct and train an image feature extraction network;

[0101] Extraction module 602: used to continuously perform image feature extraction on the frame images of the target program by using the image feature extraction network;

[0102] Matching module 603: match the image features to be matched of the extracted frame images with the feature library to obtain the matching result of the frame images;

[0103] Determination module 604: if the matching results of consecutive M frame images all meet the preset conditions, it is determined that there is a first replaced advertisement program in the target program, where M is a positive integer;

[0104] Replacement module 605: used to replace the first replaced advertisement program with a first new advertisement program based on the preset replacement rules.

[0105] Among them, in the preset replacement rule, the new advertisement program has a corresponding relationship with the replaced advertisement program, and each new advertisement program in the new advertisement program corresponds one-to-one with the image frames of its corresponding replaced advertisement program.

[0106] In other embodiments of the present application, the preset conditions include at least one of the following conditions: the first condition, the first condition is that the matching results of the continuous M frame images all belong to the first replaced advertisement program; the second condition, the second condition is that the timing of the matching results of the continuous M frame images corresponding to the frame images of the target program is the same as the timing of the M matching results corresponding to the first replaced advertisement program.

[0107] In other embodiments of the present application, the extraction module 602 is further configured to use the image feature extraction network to extract the image features of each frame of all the replaced advertisement programs; the construction module 601: is further configured to construct the feature library based on the image features of each frame of all the replaced advertisement programs.

[0108] In other embodiments of the present application, the extraction module 602: is further configured to perform a preprocessing operation on each frame of each replaced advertisement program in all the replaced advertisement programs, so that each frame of each replaced advertisement program conforms to the input format of the image feature extraction network.

[0109] In other embodiments of the present application, the extraction module 602: is further configured to process the image data of each preprocessed replaced advertisement program input into the image feature extraction network, and extract the one-dimensional feature vectors of each frame of image.

[0110] In other embodiments of the present application, the construction module 601: is further configured to perform feature clustering on all the image features in the feature library after constructing the feature library, to obtain N atomic feature sets, and each atomic feature corresponds to a clustering formed by several image features; where N is a positive integer.

[0111] In other embodiments of the present application, the extraction module 602: is further configured to perform a preprocessing operation on the extracted frame images of the target program, so that the extracted frame images of the target program conform to the input format of the image feature extraction network.

[0112] In other embodiments of the present application, the matching module 603 is specifically configured to perform a similarity measurement calculation on the image features to be matched of the frame image with each atomic feature set in the feature library, obtaining N atomic feature similarity measurement results, and comparing each atomic feature similarity measurement result in the N atomic feature similarity measurement results with a preset first threshold. If all N atomic feature similarity measurement results are less than the preset first threshold, it is determined that the frame image does not belong to the replaced advertisement program; if there is a calculation result greater than the preset first threshold among the N atomic feature similarity measurement results, then perform a similarity measurement calculation on the image features to be matched of the frame image with each image feature in the cluster corresponding to the atomic feature set corresponding to the calculation result greater than the preset first threshold among the N atomic feature similarity measurement results, obtaining K image feature similarity measurement results. If each image feature similarity measurement result in the K image feature similarity measurement results is less than a preset second threshold, it is determined that the frame image does not belong to the replaced advertisement program; if there is a calculation result greater than the preset second threshold among the K image feature similarity measurement results, it is determined that the frame image belongs to the replaced advertisement program, and the frame image in the feature library corresponding to the calculation result greater than the preset second threshold among the K image feature similarity measurement results is the finally obtained matching result, where K is a positive integer.

[0113] In other embodiments of the present application, the matching module 603 is further specifically configured to perform a similarity measurement calculation on the image features to be matched of the frame image with each atomic feature set in the feature library, obtaining N atomic feature similarity measurement results, and comparing each atomic feature similarity measurement result in the N atomic feature similarity measurement results with a preset third threshold. If all N atomic feature similarity measurement results are greater than the preset third threshold, it is determined that the frame image does not belong to the replaced advertisement program;

[0114] If there is a calculation result less than or equal to a preset third threshold among the calculation results of the N atomic feature similarity metrics, then perform a similarity metric calculation between the image features to be matched in the frame image and each image feature in the cluster corresponding to the set of atomic features corresponding to the calculation results less than or equal to the preset third threshold among the N atomic feature similarity metric calculation results, to obtain J image feature similarity metric calculation results. If each image feature similarity metric calculation result among the J image feature similarity metric calculation results is greater than a preset fourth threshold, then it is determined that this frame image does not belong to the replaced advertising program; if there is a calculation result less than or equal to the preset fourth threshold among the J image feature similarity metric calculation results, then it is determined that this frame image belongs to the replaced advertising program, and the frame image in the feature library corresponding to the calculation result less than or equal to the preset fourth threshold among the J image feature similarity metric calculation results is the finally obtained matching result, where J is a positive integer.

[0115] In other embodiments of the present application, the determining module 604: is further configured to determine the reliability of the matching result.

[0116] In other embodiments of the present application, the determining module 604: is specifically configured to determine it as an unreliable matching result when the matching result of a single frame or consecutive i frames of the image to be matched is inconsistent with the replaced program segments before and after it; determine it as an unreliable matching result when the timing corresponding to the matching result of a single frame or consecutive i frames corresponds to an inconsistent timing before and after in the replaced advertisement; where i is a positive integer.

[0117] In other embodiments of the present application, the determining module 604: is further configured to, after determining that the matching result is an unreliable matching result, correct its matching result according to a preset correction strategy.

[0118] In other embodiments of the present application, the determining module 604: is specifically configured to replace the matching result with the reliable matching result of the frame image before or after the unreliable matching result.

[0119] Those skilled in the art should understand that Figure 6 The implementation functions of the units in the shown real-time advertisement program replacement device can be understood with reference to the relevant descriptions of the foregoing method. Figure 6 The functions of the units in the shown real-time advertisement program replacement device can be implemented by a program running on a processor, or can be implemented by specific logic circuits.

[0120] Figure 7 It is a schematic structural diagram of an electronic device 700 provided by an embodiment of the present application. Figure 7The electronic device 700 shown includes a processor 710, and the processor 710 can call and run a computer program from a memory to implement the method in the embodiment of the present application.

[0121] Optionally, as Figure 7 shown, the electronic device 700 may further include a memory 720. Among them, the processor 710 can call and run a computer program from the memory 720 to implement the method in the embodiment of the present application.

[0122] Among them, the memory 720 can be a separate device independent of the processor 710 or can be integrated in the processor 710.

[0123] Optionally, as Figure 7 shown, the electronic device 700 may further include a transceiver 730. The processor 710 can control the transceiver 730 to communicate with other devices. Specifically, it can send information or data to other devices or receive information or data sent by other devices.

[0124] Among them, the transceiver 730 can include a transmitter and a receiver. The transceiver 730 may further include an antenna, and the number of antennas can be one or more.

[0125] The electronic device 700 can specifically be the real-time advertisement program replacement device in the embodiment of the present application, and the electronic device 700 can implement the corresponding processes in each method in the embodiment of the present application that are implemented by the real-time advertisement program replacement device. For the sake of brevity, it will not be elaborated here.

[0126] Figure 8 is a schematic structural diagram of a chip in the embodiment of the present application. Figure 8 The chip 800 shown includes a processor 810, and the processor 810 can call and run a computer program from a memory to implement the method in the embodiment of the present application.

[0127] Optionally, as Figure 8 shown, the chip 800 may further include a memory 820. Among them, the processor 810 can call and run a computer program from the memory 820 to implement the method in the embodiment of the present application.

[0128] Among them, the memory 820 can be a separate device independent of the processor 810 or can be integrated in the processor 810.

[0129] Optionally, the chip 800 may further include an input interface 830. Among them, the processor 810 can control the input interface 830 to communicate with other devices or chips. Specifically, it can obtain information or data sent by other devices or chips.

[0130] Optionally, the chip 800 may further include an output interface 840. Among them, the processor 810 may control the output interface 840 to communicate with other devices or chips. Specifically, it may output information or data to other devices or chips.

[0131] The chip can be applied to the advertisement program real-time replacement device in the embodiments of the present application, and the chip can implement the corresponding processes implemented by the advertisement program real-time replacement device in each method of the embodiments of the present application. For the sake of brevity, it will not be elaborated here.

[0132] It should be understood that the chip mentioned in the embodiments of the present application may also be referred to as a system-on-chip, system chip, chip system, or system-on-chip, etc.

[0133] It should be understood that the processor in the embodiments of the present application may be an integrated circuit chip with signal processing capabilities. In the implementation process, each step of the above method embodiments may be completed by the integrated logic circuit in the hardware of the processor or instructions in the form of software. The above-mentioned processor may be a general-purpose processor, a digital signal processor (DSP), an application-specific integrated circuit (ASIC), a field-programmable gate array (FPGA), or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components. It can implement or execute the various methods, steps, and logic block diagrams disclosed in the embodiments of the present application. The general-purpose processor may be a microprocessor or the processor may also be any conventional processor, etc. The steps of the method disclosed in combination with the embodiments of the present application may be directly embodied as being executed by the hardware decoding processor, or completed by a combination of the hardware and software modules in the decoding processor. The software module may be located in a mature storage medium in the art such as a random access memory, flash memory, read-only memory, programmable read-only memory, or electrically erasable programmable memory, register, etc. The storage medium is located in the memory, and the processor reads the information in the memory and combines its hardware to complete the steps of the above method.

[0134] It can be understood that the memory in the embodiments of the present application can be a volatile memory or a non-volatile memory, or can include both volatile and non-volatile memories. Among them, the non-volatile memory can be a read-only memory (ROM), a programmable read-only memory (PROM), an erasable programmable read-only memory (EPROM), an electrically erasable programmable read-only memory (EEPROM), or a flash memory. The volatile memory can be a random access memory (RAM), which is used as an external cache. By way of example but not limitation, many forms of RAM are available, such as static random access memory (SRAM), dynamic random access memory (DRAM), synchronous dynamic random access memory (SDRAM), double data rate synchronous dynamic random access memory (DDR SDRAM), enhanced synchronous dynamic random access memory (ESDRAM), synchlink dynamic random access memory (SLDRAM), and direct rambus random access memory (DR RAM). It should be noted that the memory of the systems and methods described herein is intended to include but not be limited to these and any other suitable types of memory.

[0135] It should be understood that the above memory is by way of example but not limitation. For example, the memory in the embodiments of the present application can also be a static random access memory (SRAM), a dynamic random access memory (DRAM), a synchronous dynamic random access memory (SDRAM), a double data rate synchronous dynamic random access memory (DDR SDRAM), an enhanced synchronous dynamic random access memory (ESDRAM), a synch link dynamic random access memory (SLDRAM), and a direct rambus random access memory (DR RAM), etc. That is to say, the memory in the embodiments of the present application is intended to include but not be limited to these and any other suitable types of memory.

[0136] The embodiments of the present application also provide a computer-readable storage medium for storing a computer program. The computer-readable storage medium can be applied to the advertisement program real-time replacement device in the embodiments of the present application, and the computer program enables the computer to execute the corresponding processes implemented by the advertisement program real-time replacement device in each method of the embodiments of the present application. For the sake of brevity, it will not be elaborated here.

[0137] The embodiments of the present application also provide a computer program product, including computer program instructions. The computer program product can be applied to the advertisement program real-time replacement device in the embodiments of the present application, and the computer program instructions enable the computer to execute the corresponding processes implemented by the advertisement program real-time replacement device in each method of the embodiments of the present application. For the sake of brevity, it will not be elaborated here.

[0138] The embodiments of the present application also provide a computer program. The computer program can be applied to the advertisement program real-time replacement device in the embodiments of the present application. When the computer program runs on a computer, it enables the computer to execute the corresponding processes implemented by the advertisement program real-time replacement device in each method of the embodiments of the present application. For the sake of brevity, it will not be elaborated here.

[0139] Those of ordinary skill in the art can realize that the units and algorithm steps of each example described in combination with the embodiments disclosed herein can be implemented by electronic hardware, or a combination of computer software and electronic hardware. Whether these functions are executed in a hardware or software manner depends on the specific application and design constraints of the technical solution. Professional technicians can use different methods to implement the described functions for each specific application, but such implementation should not be considered to exceed the scope of the present application.

[0140] Those skilled in the art can clearly understand that for the convenience and brevity of description, the specific working processes of the above-described systems, devices, and units can refer to the corresponding processes in the foregoing method embodiments, and will not be elaborated here.

[0141] In several embodiments provided by the present application, it should be understood that the disclosed systems, devices, and methods can be implemented in other ways. For example, the device embodiments described above are merely illustrative. For example, the division of the units is only a logical function division. In actual implementation, there can be other division methods. For example, multiple units or components can be combined or integrated into another system, or some features can be ignored or not executed. Another point is that the displayed or discussed couplings or direct couplings or communication connections to each other can be through some interfaces. The indirect couplings or communication connections of devices or units can be in an electrical, mechanical, or other form.

[0142] The unit described as a separation component may or may not be physically separated. The component displayed as a unit may or may not be a physical unit, that is, it may be located in one place or distributed to multiple network units. Some or all of the units can be selected according to actual needs to achieve the purpose of the solution of this embodiment.

[0143] In addition, in each embodiment of the present application, each functional unit can be integrated in a processing unit, or each unit can exist physically alone, or two or more units can be integrated in one unit.

[0144] If the above-mentioned function is implemented in the form of a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on such an understanding, the technical solution of the present application, in essence, or the part that contributes to the prior art, or a part of the technical solution, can be embodied in the form of a software product. The computer software product is stored in a storage medium and includes several instructions for causing a computer device (which can be a personal computer, a server, or a network device, etc.) to execute all or part of the steps of the methods described in each embodiment of the present application. The aforementioned storage medium includes: various media such as USB flash drives, mobile hard disks, read-only memory (ROM), random access memory (RAM), magnetic disks, or optical discs that can store program codes.

[0145] The above is only the specific implementation manner of the present application, but the protection scope of the present application is not limited thereto. Any person skilled in the art can easily think of changes or substitutions within the technical scope disclosed in the present application, and all of them should be covered by the protection scope of the present application. Therefore, the protection scope of the present application should be subject to the protection scope of the claims.

Claims

1. A method for real-time replacement of advertising programs, characterized in that, Including: Construct and train an image feature extraction network; Use the image feature extraction network to extract the image features of each frame of all the replaced advertising programs; Based on the image features of each frame of all the replaced advertising programs, construct a feature library; Use the image feature extraction network to continuously extract the image features of the frame images of the target program, and match the to-be-matched image features of the extracted frame images with the feature library to obtain the matching result of the frame images; If the matching results of consecutive M frame images all meet the preset conditions, it is determined that there is a first replaced advertising program in the target program, where M is a positive integer; the preset conditions include at least one of the following conditions: the first condition, the first condition is that the matching results of the consecutive M frame images all belong to the first replaced advertising program; the second condition, the second condition is that the time sequence of the frame images of the target program corresponding to the matching results of the consecutive M frame images is the same as the time sequence of the M matching results corresponding to the first replaced advertising program; Based on the preset replacement rule, replace the first replaced advertising program with a first new advertising program; wherein, in the preset replacement rule, the new advertising program has a corresponding relationship with the replaced advertising program, and each new advertising program in the new advertising program corresponds one-to-one with the image frames of its corresponding replaced advertising program.

2. The method for real-time replacement of advertising programs according to claim 1, characterized in that, Before extracting the image features of each frame of all the replaced advertising programs, it further includes: Perform a preprocessing operation on each frame image of each replaced advertising program in all the replaced advertising programs, so that each frame image of each replaced advertising program conforms to the input format of the image feature extraction network.

3. The method for real-time replacement of advertising programs according to claim 2, characterized in that, The step of using the image feature extraction network to extract the image features of each frame of all the replaced advertising programs includes: Process the image data of each preprocessed replaced advertising program input into the image feature extraction network, and extract the one-dimensional feature vector of each frame image.

4. The method for real-time replacement of advertising programs according to any one of claims 1-3, characterized in that, The method further includes: After constructing the feature library, perform feature clustering on all the image features in the feature library to obtain N atomic feature sets, and each atom feature corresponds to a clustering formed by several image features; where N is a positive integer.

5. The method for real-time replacement of advertising programs according to claim 4, characterized in that, Before using the image feature extraction network to continuously extract the image features of each frame image of the target program, it further includes: Perform a preprocessing operation on the extracted frame images of the target program, so that the extracted frame images of the target program conform to the input format of the image feature extraction network.

6. The method for real-time replacement of advertising programs according to any one of claims 2, 3 or 5, characterized in that, The preprocessing operation includes at least one of the following operations: color space conversion, downsampling.

7. The method for real-time replacement of advertising programs according to claim 5, characterized in that, The step of using the image feature extraction network to continuously extract the image features of the frame images of the target program includes: Use the image feature extraction network to continuously process each preprocessed extracted frame image of the target program, and extract the one-dimensional feature vector of each frame image.

8. The method for real-time replacement of advertising programs according to claim 7, characterized in that, The step of matching the to-be-matched image features of the extracted frame images with the feature library to obtain the matching result of the frame images includes: Perform similarity measurement calculations between the image features to be matched of the frame image and each atomic feature set in the feature library to obtain N atomic feature similarity measurement calculation results, and compare each atomic feature similarity measurement calculation result among the N atomic feature similarity measurement calculation results with a preset first threshold. If all N atomic feature similarity measurement calculation results are less than the preset first threshold, it is determined that this frame image does not belong to the replaced advertisement program; If there is a calculation result greater than or equal to the preset first threshold among the N atomic feature similarity measurement calculation results, then perform similarity measurement calculations between the image features to be matched of the frame image and each image feature in the cluster corresponding to the atomic feature set corresponding to the calculation result greater than or equal to the preset first threshold among the N atomic feature similarity measurement calculation results to obtain K image feature similarity measurement calculation results. If each image feature similarity measurement calculation result among the K image feature similarity measurement calculation results is less than the preset second threshold, it is determined that this frame image does not belong to the replaced advertisement program; if there is a calculation result greater than or equal to the preset second threshold among the K image feature similarity measurement calculation results, it is determined that this frame image belongs to the replaced advertisement program, and the frame image in the feature library corresponding to the calculation result greater than or equal to the preset second threshold among the K image feature similarity measurement calculation results is the finally obtained matching result, where K is a positive integer.

9. The method for real-time replacement of advertising programs according to claim 7, characterized in that, The matching of the image features to be matched of the extracted frame image with the feature library to obtain the matching result of the frame image includes: Perform similarity measurement calculations between the image features to be matched of the frame image and each atomic feature set in the feature library to obtain N atomic feature similarity measurement calculation results, and compare each atomic feature similarity measurement calculation result among the N atomic feature similarity measurement calculation results with a preset third threshold. If all N atomic feature similarity measurement calculation results are greater than the preset third threshold, it is determined that this frame image does not belong to the replaced advertisement program; If there is a calculation result less than or equal to the preset third threshold among the N atomic feature similarity measurement calculation results, then perform similarity measurement calculations between the image features to be matched of the frame image and each image feature in the cluster corresponding to the atomic feature set corresponding to the calculation result less than or equal to the preset third threshold among the N atomic feature similarity measurement calculation results to obtain J image feature similarity measurement calculation results. If each image feature similarity measurement calculation result among the J image feature similarity measurement calculation results is greater than the preset fourth threshold, it is determined that this frame image does not belong to the replaced advertisement program; if there is a calculation result less than or equal to the preset fourth threshold among the J image feature similarity measurement calculation results, it is determined that this frame image belongs to the replaced advertisement program, and the frame image in the feature library corresponding to the calculation result less than or equal to the preset fourth threshold among the J image feature similarity measurement calculation results is the finally obtained matching result, where J is a positive integer.

10. The method for real-time replacement of advertising programs according to claim 8 or 9, characterized in that, After obtaining the matching result of the frame image, it further includes: Performing a reliability determination on the matching result.

11. The real-time replacement method of an advertisement program according to claim 10, characterized in that The performing a reliability determination on the matching result includes: When the matching result of a single frame or consecutive i frames of the images to be matched is inconsistent with the replaced advertising program segments before and after it, it is determined as an unreliable matching result; When the timing corresponding to the matching result of a single frame or consecutive i frames in the replaced advertising program is inconsistent with the timing before and after it, it is determined as an unreliable matching result; where i is a positive integer.

12. The real-time replacement method of an advertisement program according to claim 11, characterized in that After determining that the matching result is an unreliable matching result, correcting its matching result according to a preset correction strategy.

13. The real-time replacement method of an advertisement program according to claim 12, characterized in that The correcting its matching result according to a preset correction strategy includes: Replacing the matching result of the unreliable matching result with the reliable matching result of the frame image before or after it.

14. A real-time replacement device for an advertisement program, characterized in that The apparatus includes: A construction module: used for constructing and training an image feature extraction network; An extraction module, used for using the image feature extraction network to extract the image features of each frame of all the replaced advertising programs; The construction module: is further used for constructing a feature library based on the image features of each frame of all the replaced advertising programs; The extraction module: continuously performs image feature extraction on the frame images of the target program by using the image feature extraction network; A matching module: used for matching the image features to be matched of the extracted frame images with the feature library to obtain the matching result of the frame images; A determination module: used for determining that there is a first replaced advertising program in the target program if the matching results of consecutive M frames of images all meet the preset conditions, where M is a positive integer; the preset conditions include at least one of the following conditions: the first condition, the first condition is that the matching results of the consecutive M frames of images all belong to the first replaced advertising program; the second condition, the second condition is that the timing corresponding to the frame images of the target program of the matching results of the consecutive M frames of images is the same as the timing corresponding to the first replaced advertising program of the M matching results; A replacement module: used for replacing the first replaced advertising program with a first new advertising program based on a preset replacement rule; where, in the preset replacement rule, the new advertising program has a corresponding relationship with the replaced advertising program, and each new advertising program in the new advertising program corresponds one-to-one with the image frames of its corresponding replaced advertising program.

15. The real-time replacement device for an advertisement program according to claim 14, characterized in that The extraction module: is further used for performing a preprocessing operation on each extracted frame of image to make each frame of image conform to the input format of the image feature extraction network.

16. An electronic device, characterized in that It includes: A processor and a memory, the memory is used for storing a computer program, and the processor is used for calling and running the computer program stored in the memory to execute the advertising program real-time replacement method according to any one of claims 1 to 13.

17. A chip, characterized in that It includes: A processor, used for calling and running a computer program from the memory, so that the device installed with the chip executes the advertising program real-time replacement method according to any one of claims 1 to 13.

18. A computer-readable storage medium, characterized in that For storing a computer program which causes a computer to execute the method for real-time replacement of an advertisement program according to any one of claims 1 to 13.

Images