System, computer program product and method for automatic insertion of virtual advertisements in a movie

Abstract

This invention discloses a system for automatically inserting virtual advertisements into videos, comprising a video clip detection module, a background extraction module, a correction module, and an advertisement material insertion module. The video clip detection module detects target clips from sports event videos using a first training model. The background extraction module performs background extraction to extract a first background mask from the background of the first frame of the target clip. The correction module performs camera correction to detect a first transformation relationship between the first frame and a sports field template using a second training model. The advertisement material insertion module transforms the advertisement material according to the first transformation relationship to obtain a first transformed advertisement material, and performs advertisement material insertion according to the first background mask to insert the first transformed advertisement material into the first frame, thereby obtaining a first image frame with the inserted advertisement.

Description

Technical Field

[0001] This invention relates to a system for automatically inserting virtual advertisements into sports event videos. This invention also relates to a computer program product and method for automatically inserting virtual advertisements into sports event videos. Background Technology

[0002] Advertising is ubiquitous, found in many places such as shopping malls, radio stations, television, bus stations, train stations, baseball fields, football fields, and stadiums. Advertising can effectively increase product sales and thus revenue. Generally, advertising can be divided into two types: in-stream advertising and embedded advertising. In-stream advertising may be incorporated into various forms of content, such as television programs and TV films. However, when an in-stream advertisement is broadcast, the ongoing video is interrupted, potentially affecting the viewer's experience. Furthermore, embedded advertising is typically static, and therefore its effectiveness is usually limited. Summary of the Invention

[0003] According to one embodiment of the present invention, a system for automatically inserting virtual advertisements into sports event videos is proposed, comprising a video clip detection module, a background extraction module, a correction module, and an advertisement material insertion module. The video clip detection module detects target clips from sports event videos using a first training model. The background extraction module performs background extraction to extract a first background mask from the background of the first frame of the target clip. The correction module performs camera correction to detect a first transformation relationship between the first frame and a sports field template using a second training model. The advertisement material insertion module transforms the advertisement material according to the first transformation relationship to obtain a first transformed advertisement material, and performs advertisement material insertion according to the first background mask to insert the first transformed advertisement material into the first frame, thereby obtaining a first image frame with the inserted advertisement.

[0004] According to another embodiment of the present invention, a method for automatically inserting virtual advertisements into sports event videos is proposed, comprising the following steps: detecting a target segment of the sports event video using a first training model; detecting the target segment of the sports event video using the first training model; performing background extraction to extract a first background mask from the background of a first frame of the target segment; performing camera correction to detect a first transformation relationship between the first frame and a sports field template using a second training model; transforming the advertisement material according to the first transformation relationship to obtain a first transformed advertisement material; and performing advertisement material insertion according to the first background mask to insert the first transformed advertisement material into the first frame to obtain a first image frame with the inserted advertisement.

[0005] According to one embodiment of the present invention, a computer program product is proposed, which, after being loaded onto a computer, performs the following steps: detecting a target segment of a sports event video using a first training model; performing background extraction to extract a first background mask from the background of the first frame of the target segment; performing camera correction to detect a first transformation relationship between the first frame and the sports field template using a second training model; transforming advertising material according to the first transformation relationship to obtain a first transformed advertising material; and performing advertising material placement according to the first background mask to place the first transformed advertising material into the first frame to obtain a first image frame with the placed advertisement. Attached Figure Description

[0006] Figure 1 A block diagram of a system for automatically inserting virtual advertisements into sports event videos, according to an embodiment of the present invention;

[0007] Figure 2 A flowchart illustrating the training process of a first training model in a system for automatically inserting virtual advertisements into sports event videos, according to an embodiment of the present invention.

[0008] Figures 3A to 3D This is a schematic diagram illustrating the training process of the first training model of a system for automatically inserting virtual advertisements into sports event videos, according to an embodiment of the present invention.

[0009] Figures 4A-4B This is a schematic diagram illustrating the background extraction module performing background extraction in a system for automatically inserting virtual advertisements into sports event videos, according to an embodiment of the present invention.

[0010] Figure 5 A flowchart illustrating the training process of a second training model in a system for automatically inserting virtual advertisements into sports event videos, according to an embodiment of the present invention.

[0011] Figures 6A to 6F This is a schematic diagram illustrating the training process of a second training model in a system for automatically inserting virtual advertisements into sports event videos, according to an embodiment of the present invention.

[0012] Figure 7A This is a schematic diagram of advertising materials provided by a system for automatically inserting virtual advertisements into sports event videos, according to an embodiment of the present invention.

[0013] Figure 7B This is a schematic diagram of the converted advertising material provided by a system for automatically inserting virtual advertisements into sports event videos, according to an embodiment of the present invention.

[0014] Figure 7C A schematic diagram of an image frame with embedded advertisements provided by a system for automatically inserting virtual advertisements into sports event videos, according to an embodiment of the present invention.

[0015] Figure 8A flowchart illustrating a method for automatically inserting virtual advertisements into sports event videos according to an embodiment of the present invention;

[0016] Figure 9 A flowchart illustrating a method for automatically inserting virtual advertisements into sports event videos, according to another embodiment of the present invention.

[0017] Explanation of reference numerals in the attached figures

[0018] 1: A system that automatically inserts virtual advertisements into sports event videos;

[0019] 11: Video clip detection module;

[0020] 12: Background extraction module;

[0021] 13: Calibration module;

[0022] 14: Camera attitude tracking module;

[0023] 15: Advertising creative placement module;

[0024] 16: Training module;

[0025] RV: Training video;

[0026] SV: Sports event videos;

[0027] FV: Sports event videos with embedded advertisements;

[0028] Type 1~Type 4: Fragment types;

[0029] F1: First frame;

[0030] M1: First background mask;

[0031] R1~R5: Training frames;

[0032] R0: Sports field template;

[0033] Ad: Advertising material;

[0034] Td: First conversion ad creative;

[0035] Pd1: The first image frame with the embedded advertisement;

[0036] S21~S23, S51~S53, S81~S89, S91~S95, S931~S932: Step-by-step process. Detailed Implementation

[0037] The following description, with reference to the accompanying drawings, illustrates embodiments of the system, computer program product, and method for automatically inserting virtual advertisements into sports event videos according to the present invention. For clarity and convenience, the dimensions and proportions of the components in the drawings may be exaggerated or reduced. For ease of understanding, the same elements in the following embodiments are indicated by the same symbols.

[0038] Please see Figure 1 This is a block diagram of a system for automatically inserting virtual advertisements into sports event videos according to an embodiment of the present invention. As shown in the figure, the system 1 for automatically inserting virtual advertisements into sports event videos includes a video segment detection module 11, a background extraction module 12, a correction module 13, a camera posture tracking module 14, an advertisement material insertion module 15, and a training module 16. This system 1 can insert advertisement materials into sports event videos (SV) to obtain sports event videos (FV) with inserted advertisements. In one embodiment, the sports event video (SV) can be, but is not limited to, videos of football matches, basketball matches, American football matches, volleyball matches, racing matches, etc.

[0039] The sports event video (SV) can be input into the video segment detection module 11. Then, the video segment detection module 11 detects at least one target segment of the sports event video SV using a first trained model. The video segment detection module 11 can perform Shot of Interest (SOI) detection on the sports event video SV, thus obtaining one or more SOIs as target segments. In one embodiment, the sports event video SV can be, but is not limited to, recorded sports event videos, live sports event video streams, etc.

[0040] Please see Figure 2 and Figures 3A to 3D These are, respectively, a flowchart and a schematic diagram of the training process of the first training model in a system for automatically inserting virtual advertisements into sports event videos according to an embodiment of the present invention. Please also refer to... Figure 1 .like Figure 1 As shown, the training module 16 can pre-execute a first training procedure to obtain a first training model.

[0041] First, the training module 16 receives multiple training videos (RVs). The type of the training video RV can be similar to the target motion video (SV). For example, if the motion video SV is a football match video, then the training video RV is also a football match video. Similarly, if the motion video SV is a basketball match video, then the training video RV is also a basketball match video.

[0042] Then, training module 16 classifies the segments of the training video RV into different types using a pre-classification algorithm to obtain labeled data. These types can include Type-1 (normal), Type-2 (focus), Type-3 (highlight), and Type-4 (others). Figure 3A As shown, segments labeled Type-1 (normal) are suitable as segments of interest (SOIs) because these segments depict the ongoing game and display part or all of the playing field from an appropriate perspective. Therefore, advertising creatives are more suitable for incorporating segments labeled Type-1 (normal). Figure 3B As shown, clips labeled Type-2 (close-up / focus) may contain only one or more players, game commentators, or sports analysts. For example... Figure 3C As shown, a segment labeled Type-3 (highlight) may only contain the focus of a goal (e.g., the goal scored, the player who scored, etc.). As shown in Figure 3D, a segment labeled Type-4 (others) may only contain team lineup information, player and coach interactions, etc. Therefore, segments labeled Type-2 (focus), Type-3 (highlight), and Type-4 (others) are not suitable for use in embedded advertising materials. In one embodiment, the pre-classification algorithm may be, but is not limited to, histogram-based algorithms, feature-based algorithms, support vector machine (SVM-based) algorithms, convolutional neural network (CNN-based) algorithms, or other related algorithms.

[0043] Then, the training module 16 inputs the labeled data into the first deep learning network model to train the labeled data using the first deep learning network model, thereby obtaining a first training model. In one embodiment, the first deep learning network model may be, but is not limited to, TransNet, MobileNet, or other similar deep learning network models. Figure 2 As shown, this first training procedure includes the following steps:

[0044] Step S21: Receive multiple training videos.

[0045] Step S22: Classify the segments of these training videos into different types using a pre-classification algorithm to obtain labeled data.

[0046] Step S23: Train these training videos using the first deep learning network model to obtain the first training model.

[0047] Please see Figures 4A-4B These are schematic diagrams illustrating the background extraction module performing background extraction in a system for automatically inserting virtual advertisements into sports event videos, according to an embodiment of the present invention. Please also refer to... Figure 1 .like Figure 1 As shown, the background extraction module 12 performs background extraction to extract the first frame F1 of the target segment (e.g., ...). Figure 4A The background shown (this background refers to objects not in the competition venue, such as spectators, cheerleaders, etc.) is used to obtain the first background mask M1; the first background mask M1 is the background mask corresponding to the first frame F1, as shown. Figure 4B As shown. When performing background extraction, the background extraction module 12 first filters the first frame F1 using a chroma key (which can be user-defined or adaptively adjustable within a certain range) to obtain an initial background mask (the foreground may include the playing field and objects in the playing field, such as players, referees, etc.). Next, the background extraction module 12 denoises the initial background mask. Finally, the background extraction module 12 adds a predefined mask to the initial background mask to obtain the first background mask M1.

[0048] Next, the calibration module 13 performs camera calibration to detect the playing field in the first frame (Fl) image using the second trained model. Figure 4A (as shown) and sports field template R0 (as shown) Figure 6F The first transformation relationship between the first frame F1 and the sports field template R0 is shown below. In this embodiment, the first transformation relationship can be the homography between the first frame F1 and the sports field template R0 (in another embodiment, the first transformation relationship can also be the homography between a camera with a camera pose corresponding to the first frame F1 and the sports field). The sports field template R0 corresponds to a preset camera pose. Since the camera can be adjusted by PTZ (pan / tilt / zoom) control, the camera poses corresponding to different frames may be different from each other.

[0049] Please see Figure 5 and Figures 6A to 6F These are, respectively, a training flowchart and a schematic diagram of the training process of the second training model of a system for automatically inserting virtual advertisements into sports event videos according to an embodiment of the present invention. Please also refer to... Figure 1 .like Figure 1 As shown, the training module 16 can perform a second training procedure in advance to obtain a second training model.

[0050] First, the training module 16 receives a large number of training frames, each of which has a corresponding transformation relationship. Figures 6A-6E Frames R1 to R5 shown are a subset of these training frames, as illustrated in the figure. Figure 6F As shown, in this embodiment, the sports field template R0 is an image of the sports field from above, obtained by a camera suspended above the field, with the center of the camera corresponding to the center of the field. The camera's viewpoint and focus are preset camera poses. In this embodiment, the transformation relationship between each training frame (e.g., frames R1 to R5) can be the homography between this frame and the sports field template R0 (in another embodiment, the transformation relationship between each training frame can also be the homography between a camera with a camera pose corresponding to this frame and the sports field), thus the camera poses corresponding to these training frames also have different viewpoints and focuses. In another embodiment, the sports field template R0 can be varied according to actual needs.

[0051] Next, training module 16 inputs these training frames into the second deep learning network model to train the model and obtain the second training model. In one embodiment, the second deep learning network model may be, but is not limited to, GAN, SiameseNet, and other similar deep learning network models. Thus, the second training model can effectively analyze the frames input into it and detect the transformation relationship between these frames and the sports field template R0. As shown in Figure 5, the second training procedure includes the following steps:

[0052] Step S51: Provide a sports field template corresponding to the preset camera posture.

[0053] Step S52: Receive multiple training frames, each of which has a corresponding transformation relationship.

[0054] Step S53: Train the training frames using the second deep learning network model to obtain the second training model.

[0055] Please see Figures 7A to 7C These are, respectively, schematic diagrams of advertising materials provided by a system for automatically inserting virtual advertisements into sports event videos according to an embodiment of the present invention, schematic diagrams of converted advertising materials, and schematic diagrams of image frames with inserted advertisements. As mentioned above, the correction module 13 performs camera correction to obtain a first conversion relationship between the first frame F1 and the sports field template R0.

[0056] Then, the ad creative placement module 15 converts the ad creative (e.g., Ad) according to the first conversion relationship. Figure 7A As shown) the first conversion ad creative Td is obtained (e.g. Figure 7B(As shown). In this step, the advertising material placement module 15 performs image warping on the advertising material Ad according to the first transformation relationship to obtain the first transformed advertising material Td. In one embodiment, the advertising material Ad may be, but is not limited to, a text advertisement, a static image, or an image with dynamic visual effects and / or various special effects.

[0057] Next, the ad creative placement module 15 performs ad creative placement based on the first background mask M1, placing the first transformed ad creative Td onto the first frame F1 to obtain a first image frame Pd1 with the placed ad. In this step, the ad creative placement module 15 first executes a blending algorithm, such as alpha blending, to blend the first frame F1 with the first background mask M1 to obtain a first blended frame. In this way, the ad creative Ad can be more easily blended into the first blended frame. Then, the ad creative placement module 15 places the first transformed ad creative Td into the first blended frame to obtain the first image frame Pd1 with the placed ad.

[0058] Next, as Figure 1 As shown, the background extraction module 12 extracts the background of the second frame of the target segment to obtain the second background mask (the second background mask is the background mask corresponding to the second frame).

[0059] Since performing camera correction consumes significant computational resources, performing camera correction on all frames of the target segment to obtain its transformation relationship (homography) is not an appropriate approach. Instead, this embodiment employs an alternative technique to replace this camera correction procedure. In this embodiment, the camera pose tracking module 14 performs camera pose tracking to calculate a second transformation relationship between the second frame and the sports field template R0 based on the second background mask. The second transformation relationship can be the homography between the second frame and the sports field template R0 (or the homography between the camera and the playing field corresponding to the camera pose of the second frame) (in another embodiment, the second transformation relationship can also be the homography between the camera and the playing field with the camera pose corresponding to the second frame).

[0060] During camera pose tracking, the camera pose tracking module 14 performs feature extraction to extract features from the first frame (F1, the previous frame) and the second frame (the current frame). Feature extraction can be performed using Scale-Invariant Feature Transform (SIFT), Speeded Up Robust Features (SURF), Histogram of Oriented Gradient (HOG), Maximumly Stable Extremal Region Extractor (MESR), Binary Robust Invariant scalable Keypoints (BRISK), Oriented BRIEF (ORB), Features from Accelerated Segment Test (FAST), KAZE, or other similar algorithms.

[0061] Next, the camera pose tracking module 14 performs feature matching on the first frame (previous frame) and the second frame (current frame) based on the second background mask, and filters out poorly matched parts to obtain the feature matching result. The second background mask can be used to remove moving objects that are not needed during feature matching, such as players, referees, etc. Feature matching and filtering out poorly matched parts can be performed using SIFT, SURF, MESR, BRISK, ORB, FAST, KAZE, or other similar algorithms.

[0062] Next, the camera pose tracking module 14 calculates the frame transition relationship between the first frame F1 (the previous frame) and the second frame (the current frame) based on the feature matching results. For example, the camera pose tracking module 14 calculates the affine transformation between the first frame F1 and the second frame to obtain the frame transition relationship described above.

[0063] Finally, the camera pose tracking module 14 multiplies the transformation relationship of this frame by the first transformation relationship to obtain the second transformation relationship. Through this mechanism, the second transformation relationship of the second frame can be directly obtained by the camera pose tracking module 14 through camera pose tracking, without having to go through the correction module 13 for camera correction, thus saving a lot of computing resources.

[0064] Similarly, after obtaining the second conversion relationship of the second frame, the ad creative placement module 15 converts the ad creative Ad according to the second conversion relationship to obtain the second converted ad creative, and performs ad creative placement to place the second converted ad creative into the second frame to obtain the second image frame with the placed ad.

[0065] The transformation relationships of other frames in the target segment (third frame, fourth frame, fifth frame...last frame) can also be obtained through camera pose tracking by camera pose tracking module 14. Ad placement module 15 can also transform the ad material Ad according to these frame transformation relationships to obtain corresponding transformed ad materials, and perform ad placement to insert the transformed ad materials onto these frames, thus obtaining the corresponding image frames with embedded ads. Finally, after processing all frames of all target segments in the sports event video SV through the above mechanism, the sports event video FV with embedded ads can be obtained.

[0066] However, if camera pose tracking is repeatedly performed, the error score of the frame transition relationship may continue to increase. Therefore, if the error score of the transition relationship of any frame exceeds the error threshold, the correction module 13 may need to re-perform camera correction.

[0067] The following example uses the second frame. As described above, the camera pose tracking module 14 performs camera pose tracking to calculate the second transformation relationship between the second frame and the sports field template R0 based on the second background mask. Furthermore, the camera pose tracking module 14 can simultaneously obtain the error score of the second transformation relationship. In one embodiment, this error score may be, but is not limited to, a reprojection error score or an accumulation error score. When the camera pose tracking module 14 determines that the error score of the second transformation relationship is less than an error threshold, the camera pose tracking module 14 transmits the second transformation relationship to the advertising material placement module 15. Then, the advertising material placement module 15 transforms the advertising material Ad according to the second transformation relationship to obtain the second transformed advertising material, and performs advertising material placement to place the second transformed advertising material onto the second frame, thus obtaining a second image frame with the placed advertisement.

[0068] Conversely, when the camera attitude tracking module 14 determines that the error score of the second transformation relationship is greater than or equal to the error threshold value, the camera attitude tracking module 14 sends a notification to the correction module 13. Then, the correction module 13 re-executes the camera correction for the second frame to detect the corrected second transformation relationship between the second frame and the sports field template R0 through the second training model. Then, the ad placement module 15 converts the ad material Ad according to the corrected second transformation relationship to obtain the second transformed ad material, and performs ad material placement according to the second background mask to place the second transformed ad material into the second frame, thus obtaining a second image frame with the placed ad. This system 1 will perform the above mechanism on all frames except the first frame F1 to determine whether camera correction needs to be re-performed, thus improving the accuracy of this system 1.

[0069] On the other hand, the correction module 13 can periodically and automatically re-perform camera correction to obtain the corrected transformation relationship of the current frame, which can also effectively improve the accuracy of the system 1.

[0070] As described above, the system 1 for automatically inserting virtual advertisements into sports event videos in this embodiment can effectively insert advertising material Ad into sports event videos SV to provide sports event videos FV with embedded advertisements. Therefore, viewers can directly see the advertisements when watching sports event videos FV with embedded advertisements, and it will not affect the viewers' experience of watching the video.

[0071] In addition, advertising creatives can have dynamic visual effects and / or various special effects, so ads placed in sports event videos can be more eye-catching and further enhance their advertising effectiveness.

[0072] System 1, which automatically inserts virtual advertisements into sports event videos, is applicable to various sports event videos. Furthermore, this system 1 can also be applied to other different types of videos, such as concert videos and speech videos.

[0073] Of course, this embodiment is only for illustrative purposes and is not intended to limit the scope of the invention. Equivalent modifications or changes made to the system 1 for automatically inserting virtual advertisements into sports event videos based on this embodiment should still be included within the scope of protection of the claims of the invention.

[0074] It is worth mentioning that when interstitial advertisements are played, the video playback will be interrupted, which may affect the viewer's viewing experience. Conversely, according to embodiments of the present invention, the system for automatically inserting virtual advertisements into sports event videos can effectively insert advertising materials into sports event videos, providing sports event videos with embedded advertisements, so that viewers can directly see the virtual advertisements while watching sports event videos with embedded advertisements, without affecting the viewer's viewing experience.

[0075] Furthermore, embedded advertisements are typically static, resulting in generally poor advertising effectiveness. In contrast, according to embodiments of the present invention, a system for automatically inserting virtual advertisements into sports event videos can effectively insert advertising materials with dynamic visual effects (e.g., 3D effects, rainbow effects, etc.) and / or add various special effects to the sports event videos to provide a sports event video with embedded advertisements. Therefore, placing such advertisements in sports event videos can attract more attention and further enhance their advertising effectiveness.

[0076] Furthermore, according to embodiments of the present invention, the system for automatically inserting virtual advertisements into sports event videos can embed advertising materials into sports event videos to provide sports event videos with embedded advertisements, and this advertising material can be replaced with another advertising material according to the requirements of the sponsor / advertiser. Thus, this system can not only be applied to various types of sports event videos, but also meet the needs of different sponsors / advertisers, making its application more extensive.

[0077] Furthermore, according to embodiments of the present invention, the advertising placement function of the system for automatically inserting virtual advertisements into sports event videos can be implemented with or without hardware equipment. Therefore, this system is more flexible in use and its cost can be significantly reduced. As can be seen from the above, the system for automatically inserting virtual advertisements into sports event videos according to embodiments of the present invention does indeed achieve good technical results.

[0078] Please see Figure 8 This is a flowchart illustrating a method for automatically inserting virtual advertisements into sports event videos according to an embodiment of the present invention. The method for automatically inserting virtual advertisements into sports event videos according to this embodiment includes the following steps:

[0079] Step S81: Detect target segments from sports event videos using the first training model.

[0080] Step S82: Perform background extraction to extract a first background mask from the background of the first frame of the target segment.

[0081] Step S83: Perform camera correction to detect the first transformation relationship between the first frame and the sports field template using the second training model.

[0082] Step S84: Convert the ad creative according to the first conversion relationship to obtain the first conversion ad creative.

[0083] Step S85: Perform ad creative placement according to the first background mask to place the first transformed ad creative into the first frame to obtain a first image frame with the placed ad.

[0084] Step S86: Perform background extraction on the second frame of the target segment to obtain a second background mask.

[0085] Step S87: Perform camera pose tracking to calculate the second transformation relationship between the second frame and the sports field template based on the second background mask.

[0086] Step S88: Convert the ad creative according to the second conversion relationship to obtain the second conversion ad creative.

[0087] Step S89: Perform ad creative placement to place the second conversion ad creative into the second frame to obtain a second image frame with the placed ad.

[0088] The above method can be implemented by a processor executing one or more instructions stored in a non-transitory computer-readable medium. Alternatively, the method can be executed by loading a computer program containing one or more corresponding instructions.

[0089] Please see Figure 9 This is a flowchart illustrating a method for automatically inserting virtual advertisements into sports event videos according to another embodiment of the present invention. The method for automatically inserting virtual advertisements into sports event videos according to this embodiment includes the following steps:

[0090] Step S91: Perform segment detection on the sports event video to obtain at least one target segment, and proceed to step S92. In this step, the video segment detection module 11 performs segment of interest (SOI) detection on the sports event video SV to obtain at least one segment of interest as the target segment.

[0091] Step S92: Perform background extraction on the input frame of the target segment, and proceed to step S93. In this step, the background extraction module 12 performs background extraction to extract the background of the input frame of the target segment, thus obtaining its background mask.

[0092] Step S93: Determine if the current frame is the first frame of the target segment. If yes, proceed to step S94; otherwise, proceed to step S931.

[0093] Step S94: Perform camera correction and proceed to step S95. In this step, if the current frame is the first frame F1 of the target segment, the correction module 13 needs to perform the first camera correction to detect the transformation relationship (homography) of the current frame. The transformation relationship obtained through this camera correction can serve as the basis for camera pose tracking in subsequent frames. In addition, if the error score of the transformation relationship of the current frame obtained through camera pose tracking is greater than or equal to the error threshold value, the correction module 13 needs to re-perform camera correction on the current frame.

[0094] Step S95: Perform ad creative placement. In this step, the ad creative placement module 15 performs image warping on the ad creative according to the transformation relationship of the current frame to obtain transformed ad creative. Then, the ad creative placement module 15 performs ad creative placement according to the background mask of the current frame, placing the transformed ad creative into the current frame to obtain an image frame with the placed ad.

[0095] Step S931: Perform camera pose tracking and proceed to step S932. In this step, the camera pose tracking module 14 directly performs camera pose tracking, calculating the transition relationship of the current frame based on the background mask of the current frame. During camera pose tracking, the camera pose tracking module 14 first performs feature extraction to extract features from the previous frame and the current frame. Then, the camera pose tracking module 14 performs feature matching on the previous and current frames based on the background mask of the current frame, filtering out poorly matched parts to obtain the feature matching result. Next, the camera pose tracking module 14 calculates the frame transition relationship between the previous and current frames based on the feature matching result. Finally, the camera pose tracking module 14 multiplies the frame transition relationship by the transition relationship of the previous frame to obtain the transition relationship of the current frame.

[0096] Step S932: Determine whether to re-perform camera correction? If yes, proceed to step S94; if no, proceed to step S95. In this step, the camera pose tracking module 14 obtains the error score of the transformation relationship of the current frame (e.g., the score of cumulative error or the score of reprojection error) by performing camera pose tracking; then, the camera pose tracking module 14 determines whether to re-perform camera correction based on the error score.

[0097] The above method can be implemented by a processor executing one or more instructions stored in a non-transitory computer-readable medium. Alternatively, the above method can be executed by loading a computer program containing one or more corresponding instructions.

[0098] In summary, according to embodiments of the present invention, the system for automatically inserting virtual advertisements into sports event videos can effectively insert advertising materials into sports event videos, providing sports event videos with embedded advertisements, so that viewers can directly see virtual advertisements when watching sports event videos with embedded advertisements, without affecting the viewers' experience of watching sports event videos.

[0099] Furthermore, according to embodiments of the present invention, the system for automatically inserting virtual advertisements into sports event videos can effectively insert advertising materials with dynamic visual effects (e.g., 3D effects, rainbow effects, etc.) and / or add various special effects to sports event videos to provide sports event videos with embedded advertisements. Therefore, placing such advertisements in sports event videos can attract more attention and further enhance their advertising effectiveness.

[0100] Furthermore, according to embodiments of the present invention, the system for automatically inserting virtual advertisements into sports event videos can embed advertising materials into sports event videos to provide sports event videos with embedded advertisements, and this advertising material can be replaced with another advertising material according to the requirements of the sponsor / advertiser. Thus, this system can not only be applied to various types of sports event videos, but also meet the needs of different sponsors / advertisers, making its application more extensive.

[0101] Furthermore, according to embodiments of the present invention, the advertising placement function of the system for automatically inserting virtual advertisements into sports event videos can be implemented with or without hardware equipment. Therefore, this system is more flexible in use and its cost can be significantly reduced.

[0102] Although the steps of the methods described in this invention are shown and described in a specific order, the order of operation of each method may be changed, some steps may be performed in reverse order, or some steps may be performed simultaneously with other steps. In another embodiment, different steps may be implemented intermittently and / or alternately.

[0103] It should be noted that at least some steps of the method described in this invention can be executed as software instructions stored on a computer-usable storage medium for execution by a computer (or processor). For example, examples of computer program products include a computer-usable storage medium for storing a computer-readable program.

[0104] Computer-usable or computer-readable storage media can be electronic, magnetic, optical, electromagnetic, infrared, or semiconductor systems (or devices, equipment, etc.). Examples of non-transitory computer-usable and computer-readable storage media include semiconductor or solid-state memory, magnetic tape, portable floppy disks, random access memory (RAM), read-only memory (ROM), hard disks, and optical discs. Examples of optical discs include optical discs with read-only memory (CD-ROM), rewritable optical discs (CD-R / W), and digital versatile discs (DVDs).

[0105] Furthermore, the various embodiments of the present invention (or the various modules of the system) can be implemented entirely in hardware, entirely in software, or in an implementation that includes both hardware and software components. Regarding software embodiments, the software may include, but is not limited to, firmware, resident software, microcode, etc. Regarding hardware embodiments, the hardware may be one or more application-specific integrated circuit chips (ASICs), digital signal processors (DSPs), digital signal processing devices (DSPDs), programmable logic devices (PLDs), field-programmable gate arrays (FPGAs), central processing units (CPUs), controllers, microcontrollers, microprocessors, electronic devices, other electronic units designed to perform the functions described herein or combinations thereof.

[0106] The above description is merely illustrative and not restrictive. Any equivalent modifications or alterations made without departing from the inventive concept and scope of this invention should be included within the scope of the claims of this application.

Claims

1. A system for automatically inserting virtual advertisements into videos, comprising: The video clip detection module is used to detect target clips from sports event videos using the first trained model. The background extraction module is used to perform background extraction to extract a first background mask from the background of the first frame of the target segment. The background extraction module is also used to perform the background extraction on the second frame of the target segment to obtain a second background mask. A correction module is used to perform camera correction in order to detect a first transformation relationship between the first frame and the sports field template through a second trained model; The advertising material insertion module is used to convert advertising materials according to the first conversion relationship to obtain first converted advertising materials, and to perform advertising material insertion according to the first background mask to insert the first converted advertising materials into the first frame to obtain a first image frame with inserted advertising. as well as A camera pose tracking module is used to perform camera pose tracking to calculate a second transformation relationship between the second frame and the sports field template based on the second background mask.

2. The system for automatically inserting virtual advertisements into a video according to claim 1 further includes a training module, which is used to perform a first training procedure on at least one training video through a first deep learning network model to obtain the first training model.

3. The system for automatically inserting virtual advertisements into a video according to claim 1 further includes a training module, which is used to execute a second training procedure on multiple training frames through a second deep learning network model to obtain a second training model, wherein... Each training frame has a corresponding transformation relationship.

4. The system for automatically inserting virtual advertisements into a video according to claim 1, wherein, The background extraction module is used to filter the first frame by chroma key to obtain an initial background mask, remove noise from the initial background mask, and add a predefined mask to the initial background mask to obtain the first background mask.

5. The system for automatically inserting virtual advertisements into a video according to claim 1, wherein, The advertising material placement module is used to perform the advertising material placement, to merge the first frame and the first background mask to obtain a first merged frame, and to place the first converted advertising material into the first merged frame to obtain the first image frame with the placed advertisement.

6. The system for automatically inserting virtual advertisements into a video according to claim 1, wherein, The first transformation relationship is the homography between the first frame and the sports field template, which corresponds to a preset camera pose.

7. The system for automatically inserting virtual advertisements into a video according to claim 1, wherein, The second transformation relationship is the homography between the second frame and the sports field template, which corresponds to a preset camera pose.

8. The system for automatically inserting virtual advertisements into a video according to claim 7, wherein, The advertising material placement module is used to transform the advertising material according to the second transformation relationship to obtain the second transformed advertising material, and to perform the advertising material placement to place the second transformed advertising material into the second frame to obtain a second image frame with the placed advertisement.

9. The system for automatically inserting virtual advertisements into a video according to claim 7, wherein, The camera pose tracking module is used to perform camera pose tracking based on the second background mask, to perform feature matching on the first frame and the second frame to obtain the feature matching result, and to calculate the frame transition relationship between the first frame and the second frame based on the feature matching result, and to multiply the frame transition relationship by the first transition relationship to obtain the second transition relationship.

10. The system for automatically inserting virtual advertisements into a video according to claim 7, wherein, When the camera pose tracking module determines that the error score of the second transformation relationship is greater than or equal to the error threshold value, the correction module re-executes the camera correction for the second frame to detect the corrected second transformation relationship between the second frame and the sports field template through the second training model. The advertising material placement module then converts the advertising material according to the corrected second transformation relationship to obtain the second transformed advertising material and performs the advertising material placement to place the second transformed advertising material into the second frame according to the second background mask to obtain a second image frame with the placed advertisement.

11. The system for automatically inserting virtual advertisements into a video according to claim 10, wherein, This error score is either a fraction of the reprojection error or a fraction of the cumulative error.

12. A method for automatically inserting virtual advertisements into a video, comprising: Target segments from sports event videos are detected using the first trained model; Perform background extraction to extract a first background mask from the background of the first frame of the target segment; Perform this background extraction on the second frame of the target segment to obtain a second background mask; Perform camera correction to detect the first transformation relationship between the first frame and the sports field template using a second trained model; Based on this first conversion relationship, the advertising creative is converted to obtain the first conversion advertising creative; The first background mask is used to insert advertising material into the first frame to obtain a first image frame with the inserted advertisement. as well as Perform camera pose tracking to calculate a second transformation relationship between the second frame and the sports field template based on the second background mask.

13. The method for automatically inserting virtual advertisements into a video according to claim 12, further comprising: A first training procedure is performed on at least one training video using a first deep learning network model to obtain the first training model.

14. The method for automatically inserting virtual advertisements into a video according to claim 12, further comprising: A second training procedure is performed on multiple training frames using a second deep learning network model to obtain a second trained model, wherein... Each training frame has a corresponding transformation relationship.

15. The method for automatically inserting virtual advertisements into a video according to claim 12, wherein, The step of performing the background extraction to extract the first background mask from the background of the first frame of the target segment further includes: The first frame is filtered using chroma keying to obtain the initial background mask; Denoise the initial background mask; and Add a predefined mask to the initial background mask to obtain the first background mask.

16. The method for automatically inserting virtual advertisements into a video according to claim 12, further comprising: The ad creative is converted according to the second conversion relationship to obtain the second converted ad creative; and Perform the ad creative placement to place the second converted ad creative into the second frame to obtain a second image frame with the placed ad.

17. The method for automatically inserting virtual advertisements into a video according to claim 12, wherein, The step of performing camera pose tracking to calculate the second transformation relationship between the second frame and the sports field template based on the second background mask further includes: The camera pose tracking is performed based on the second background mask to perform feature matching on the first frame and the second frame to obtain the feature matching result; Calculate the frame transition relationship between the first frame and the second frame based on the feature matching result; as well as Multiply the frame conversion relationship by the first conversion relationship to obtain the second conversion relationship.

18. The method for automatically inserting virtual advertisements into a video according to claim 12, further comprising: When the camera pose tracking module determines that the error score of the second transformation relationship is greater than or equal to the error threshold value, the camera correction is re-executed for the second frame so as to detect the corrected second transformation relationship between the second frame and the sports field template through the second training model; The ad creative is converted according to the corrected second conversion relationship to obtain the second converted ad creative; as well as Perform the ad creative placement to place the second conversion ad creative into the second frame according to the second background mask, so as to obtain a second image frame with the placed ad.

19. A computer program product that, after being loaded onto a computer, performs the following steps: Target segments from sports event videos are detected using the first trained model; Perform background extraction to extract a first background mask from the background of the first frame of the target segment; Perform this background extraction on the second frame of the target segment to obtain a second background mask; Perform camera correction to detect the first transformation relationship between the first frame and the sports field template using a second trained model; Based on this first conversion relationship, the advertising creative is converted to obtain the first conversion advertising creative; The first background mask is used to insert advertising material into the first frame to obtain a first image frame with the inserted advertisement. as well as Perform camera pose tracking to calculate a second transformation relationship between the second frame and the sports field template based on the second background mask.

20. The computer program product according to claim 19, after being loaded onto the computer, further comprises the following steps: A first training procedure is performed on at least one training video using a first deep learning network model to obtain the first training model.

21. The computer program product according to claim 19, after being loaded by the computer, further performs the following steps: A second training procedure is performed on multiple training frames using a second deep learning network model to obtain a second trained model, wherein... Each training frame has a corresponding transformation relationship.

22. The computer program product according to claim 19, wherein, The step of performing the background extraction to extract the first background mask from the background of the first frame of the target segment further includes the following steps: The first frame is filtered using chroma keying to obtain the initial background mask; Denoise the initial background mask; and Add a predefined mask to the initial background mask to obtain the first background mask.

23. The computer program product according to claim 19, after being loaded by the computer, further performs the following steps: The ad creative is converted according to the second conversion relationship to obtain the second converted ad creative; and Perform the ad creative placement to place the second converted ad creative into the second frame to obtain a second image frame with the placed ad.

24. The computer program product according to claim 19, wherein, The step of performing camera pose tracking to calculate the second transformation relationship between the second frame and the sports field template based on the second background mask further includes the following steps: The camera pose tracking is performed based on the second background mask to perform feature matching on the first frame and the second frame to obtain the feature matching result; Calculate the frame transition relationship between the first frame and the second frame based on the feature matching result; as well as Multiply the frame conversion relationship by the first conversion relationship to obtain the second conversion relationship.

25. The computer program product according to claim 19, after being loaded by the computer, further performs the following steps: When the camera pose tracking module determines that the error score of the second transformation relationship is greater than or equal to the error threshold value, the camera correction is re-executed for the second frame so as to detect the corrected second transformation relationship between the second frame and the sports field template through the second training model; The ad creative is converted according to the corrected second conversion relationship to obtain the second converted ad creative; as well as Perform the ad creative placement to place the second converted ad creative into the second frame according to the second background mask, so as to obtain a second image frame with the placed ad.

Images