Video content implantation method, model training method, device and electronic equipment

Abstract

The present application discloses a method for implanting video content, a model training method, a device and an electronic device, and belongs to the field of artificial intelligence technology. The method for implanting video content in the embodiment of the present application includes: obtaining the first media content and multiple frames of images containing a panel area in the target video; inputting the target frame image into the first model for prediction, and obtaining the corner point information and mask information of the panel area in the target frame image, wherein the target frame image is a frame image in the multiple frames, and the mask information is predicted under the supervision of the corner point information; according to the corner point information and mask information of the panel area in the target frame image, determining the corner point information and mask information of the panel area in other frame images in the multiple frames except the target frame image; according to the corner point information and mask information of the panel area in each frame image of the multiple frames, implanting the first media content into the target video. In this way, the authenticity of the video after the implantation of the content can be improved.

Description

Technical Field

[0001] The present application belongs to the field of artificial intelligence technology, and specifically relates to a video content implantation method, a model training method, a device and an electronic device. Background Art

[0002] With the widespread popularity of digital cameras and smartphones, videos have become an important medium for people to receive information in their daily lives, which has also stimulated the demand for a large number of video productions, especially the creation and implantation of advertising content. At present, when implanting content in a video, it is usually necessary to first locate the panel position in the video, and then implant the replacement content such as advertising content into the video according to the located panel position. In this case, since the panel area in the video may be blocked and have different postures, the authenticity of the video after the implantation of content will be poor. Summary of the invention

[0003] The purpose of the embodiments of the present application is to provide a video content implantation method, a model training method, a device and an electronic device to solve the problem of poor authenticity of videos after implantation of content.

[0004] In order to solve the above technical problems, this application is implemented as follows:

[0005] In a first aspect, a method for embedding video content is provided, comprising:

[0006] Acquire the first media content and a plurality of frames of images including a panel area in the target video;

[0007] Inputting the target frame image into the first model for prediction, obtaining corner point information and mask information of the panel area in the target frame image; wherein the target frame image is one frame image in the multiple frame images, and the mask information is predicted under the supervision of the corner point information;

[0008] Determine the corner point information and mask information of the panel area in other frame images in the multiple frame images except the target frame image according to the corner point information and mask information of the panel area in the target frame image;

[0009] The first media content is implanted into the target video according to the corner point information and the mask information of the panel area in each frame of the multiple frames of images.

[0010] In a second aspect, a model training method is provided, comprising:

[0011] Acquire an image sample set to be trained, wherein each image sample in the image sample set is annotated with a corner point label and a mask label, the corner point label is used to characterize the corner point information of the panel area in the image sample, and the mask label is used to characterize the mask information of the panel area in the image sample;

[0012] The image sample set is input into a preset model for training to obtain a first model, wherein the first model includes a first sub-model and a second sub-model, the first sub-model is used to predict the corner point information of the panel area, the second sub-model is used to predict the mask information of the panel area, and the loss function of the second sub-model uses the corner point information predicted by the first sub-model to supervise the prediction result of the second sub-model.

[0013] In a third aspect, a video content embedding device is provided, comprising:

[0014] A first acquisition module is used to acquire the first media content and a plurality of frames of images including a panel area in a target video;

[0015] A prediction module, used for inputting a target frame image into a first model for prediction, and obtaining corner point information and mask information of a panel area in the target frame image; wherein the target frame image is one frame image in the multiple frame images, and the mask information is predicted under the supervision of the corner point information;

[0016] A determination module, configured to determine the corner point information and mask information of the panel area in other frame images in the multiple frame images except the target frame image according to the corner point information and mask information of the panel area in the target frame image;

[0017] The implantation module is used to implant the first media content into the target video according to the corner point information and mask information of the panel area in each frame of the multiple frames of images.

[0018] In a fourth aspect, a model training device is provided, comprising:

[0019] A second acquisition module is used to acquire an image sample set to be trained, wherein each image sample in the image sample set is annotated with a corner point label and a mask label, the corner point label is used to characterize the corner point information of the panel area in the image sample, and the mask label is used to characterize the mask information of the panel area in the image sample;

[0020] A training module is used to input the image sample set into a preset model for training to obtain a first model, wherein the first model includes a first sub-model and a second sub-model, the first sub-model is used to predict the corner point information of the panel area, the second sub-model is used to predict the mask information of the panel area, and the loss function of the second sub-model uses the corner point information predicted by the first sub-model to supervise the prediction result of the second sub-model.

[0021] In a fifth aspect, an electronic device is provided, comprising a processor, a memory, and a program or instruction stored in the memory and executable on the processor, wherein the program or instruction, when executed by the processor, implements the steps of the method described in the first aspect, or the steps of the method described in the second aspect.

[0022] In a sixth aspect, a readable storage medium is provided, on which a program or instruction is stored. When the program or instruction is executed by a processor, the steps of the method described in the first aspect or the steps of the method described in the second aspect are implemented.

[0023] In an embodiment of the present application, after obtaining the first media content and multiple frames of images containing the panel area in the target video, the target frame image can be input into the first model for prediction to obtain the corner point information and mask information of the panel area in the target frame image, and the corner point information and mask information of the panel area in each frame of the multiple frames are determined based on the corner point information and mask information of the panel area in the target frame image, and the first media content is implanted into the target video based on the corner point information and mask information of the panel area in each frame. Thus, when implanting content into the video, the position information and occlusion of the panel area in the video can be considered, thereby ensuring the naturalness and continuity of the video after the implantation of the content, and improving the authenticity of the video after the implantation of the content. BRIEF DESCRIPTION OF THE DRAWINGS

[0024] Figure 1 is a schematic diagram of embedding video content in an embodiment of the present application;

[0025] Figure 2 is a flow chart of a method for embedding video content provided by an embodiment of the present application;

[0026] Figure 3 is a structural schematic diagram of the first model in the embodiment of the present application;

[0027] Figure 4 is a schematic diagram of determining corner point information and mask information in each frame of an image according to an embodiment of the present application;

[0028] Figure 5 is a flow chart of a model training method provided in an embodiment of the present application;

[0029] Figure 6 is a schematic diagram of the training process of the first model in an embodiment of the present application;

[0030] Figure 7 It is a structural schematic diagram of a video content embedding device provided in an embodiment of the present application;

[0031] Figure 8 It is a structural schematic diagram of a model training device provided in an embodiment of the present application;

[0032] Fig. 9 It is a structural schematic diagram of an electronic device provided in an embodiment of the present application. DETAILED DESCRIPTION

[0033] The following will be combined with the drawings in the embodiments of the present application to clearly and completely describe the technical solutions in the embodiments of the present application. Obviously, the described embodiments are part of the embodiments of the present application, not all of the embodiments. Based on the embodiments in the present application, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of this application.

[0034] The terms "first", "second", etc. in the specification and claims of the present application are used to distinguish similar objects, and are not used to describe a specific order or sequence. It should be understood that the data used in this way can be interchangeable under appropriate circumstances, so that the embodiments of the present application can be implemented in an order other than those illustrated or described here, and the objects distinguished by "first", "second", etc. are generally of one type, and the number of objects is not limited. For example, the first object can be one or more. In addition, "and / or" in the specification and claims represents at least one of the connected objects, and the character " / " generally indicates that the objects associated with each other are in an "or" relationship.

[0035] In order to solve the problem of poor authenticity of videos after content is currently implanted, the embodiment of the present application proposes a method for implanting video content, the core of which is to automatically identify the corner points of the panel area in the image, that is, to identify the position information / posture of the panel area, and to identify the segmentation mask of the panel area in the image, that is, to identify the occlusion relationship of the panel area, and to implant video content based on the posture and occlusion relationship of the panel area, such as Figure 1 As shown, compared with the prior art in which video content is only implanted based on the panel position, the naturalness and continuity of the video after the implantation of content is ensured, and the authenticity of the video after the implantation of content is improved. Further, the demand for panel advertising production can be met.

[0036] In the following, in combination with the accompanying drawings, the video content implantation method, model training method, device and electronic device provided in the embodiments of the present application are described in detail through specific embodiments and their application scenarios.

[0037] See also Figure 2 , Figure 2 is a flow chart of a method for embedding video content provided by an embodiment of the present application, the method is applied to electronic devices such as Figure 2 As shown, the method comprises the following steps:

[0038] Step 21: Acquire the first media content and multiple frames of images including the panel area in the target video.

[0039] In this embodiment, the first media content may be advertising content, including but not limited to pictures, images or other forms, which are not limited thereto. The target video may be understood as a video of the content to be replaced, which may be a continuously shot video containing a panel area, and may be sourced from a file or a camera device, etc. The target video may be displayed through a display interface of a terminal (such as a screen of a mobile terminal, an outdoor terminal display), or through a partial interface in a display interface (such as an interface of a player program), etc.

[0040] Optionally, the above-mentioned multiple frames of images may be all or part of the video frames in the target video. For example, if it is necessary to replace the content of all the video frames in the target video, the target video may be decoded after being acquired to obtain frame-by-frame images.

[0041] In a possible implementation scenario, obtaining the first media content may be obtaining an advertising material to be inserted into a target video.

[0042] Step 22: Input the target frame image into the first model for prediction to obtain corner point information and mask information of the panel area in the target frame image.

[0043] In this embodiment, the target frame image is one of the multiple frames, for example, it can be the first frame, i.e., the first frame, or the second frame, etc., without limitation. The mask information is predicted under the supervision of the corner point information, that is, the mask position determined based on the mask information should be within the panel area defined by the corner point.

[0044] Optionally, the above-mentioned first model may include a first sub-model and a second sub-model. The first sub-model is used to predict the corner point information of the panel area, such as the corner point coordinates, to determine the position information and posture of the panel area. The loss function of the first sub-model can be a cross entropy or any other loss function, which can be selected according to different scenarios and is not limited to this. The second sub-model is used to predict the mask information of the panel area, such as the mask coordinates, to determine the occlusion of the panel area. The loss function of the second sub-model uses the corner point information predicted by the first sub-model to supervise / correct the prediction results of the second sub-model to ensure that the mask predicted by the second sub-model is within the area enclosed by the corner points predicted by the first sub-model.

[0045] In some embodiments, the first model may be a multi-task deep learning model, such as Figure 3 As shown, a convolutional neural network (CNN) may be included, and the corner point coordinates and panel mask information may be output.

[0046] In some embodiments, the loss function L of the second sub-model is seg_point Can be:

[0047]

[0048] Where (i, j) is the mask coordinate predicted by the second sub-model. point is the area formed by the corner points corresponding to the corner point coordinates predicted by the first sub-model. dist(ij,A point ) is the coordinate point (i, j) to area A point α is the preset weight coefficient, which can be adjusted according to different scenarios. seg The second sub-model is located in region A at coordinate point (i, j) point The loss function of the inner time can be cross entropy or any other loss function, which can be selected according to different scenarios. seg +α*L seg *dist(ij,A point ) is the second sub-model at coordinate point (i, j) located in area A point The loss function when it is outside the range of L seg A penalty term is added on the basis of the predicted corner point information, and the predicted corner point information is used as additional auxiliary information / supervisory information to ensure that the mask result predicted by the second sub-model does not exceed the area enclosed by the corner points predicted by the first sub-model, making the edge of the mask segmentation more accurate.

[0049] It should be pointed out that the above loss function L seg_point The meanings may include: 1) When the mask predicted by the second sub-model is within the area enclosed by the corner points predicted by the first sub-model, the loss function of the second sub-model is the segmentation loss function L seg ; 2) When the mask predicted by the second sub-model is outside the area enclosed by the corner points predicted by the first sub-model, the loss function of the second sub-model is divided into two parts. The first part is the original segmentation loss function L seg, the second part is a loss function that changes dynamically according to the shortest distance from the currently predicted mask coordinates to the area enclosed by the predicted corner points. The loss function of the second part can effectively punish those mask segmentation results that exceed the area enclosed by the predicted corner points, ensuring that the mask results predicted by the second sub-model do not exceed the area enclosed by the corner points predicted by the first sub-model, making the edge of the mask segmentation more accurate. At the same time, the loss function changes dynamically according to the shortest distance from the currently predicted mask coordinates to the area enclosed by the predicted corner points, which can ensure the stability and convergence of the model training process, strengthen the learning effect of the model, and improve the performance and convergence speed of the model. The above loss function can be applied to any model that simultaneously predicts corner point information and mask information and requires loss function training, that is, the above loss function can be used to train any model that simultaneously predicts corner point information and mask information.

[0050] Step 23: According to the corner point information and mask information of the panel area in the target frame image, determine the corner point information and mask information of the panel area in other frame images in the multiple frame images except the target frame image.

[0051] In this step, based on the obtained corner point information and mask information of the panel area in the target frame image, the corner point information and mask information of the panel area in each frame image of the other frame images can be determined according to the timing information between the target frame image and other frame images other than the target frame image in the multi-frame images, or the corner point information and mask information of the panel area in each frame image of the other frame images can be determined according to the timing information between the multi-frame images, and there is no limitation to this. In this way, with the help of the timing information, it is possible to ensure that a stable and continuous recognition result of the panel area in the multi-frame images is obtained.

[0052] Step 24: implant the first media content into the target video according to the corner point information and mask information of the panel area in each frame of the multiple frames.

[0053] Optionally, when the first media content is embedded in the target video, the first media content may be integrated into each frame of the image according to the corner point information and mask information of the panel area in each frame of the image, thereby integrating the first media content into the target video.

[0054] Optionally, when implanting the first media content into the target video, the first media content may be optimized according to the corner point information of the panel area in each frame of the multiple frames to obtain multiple target media content; and then, the multiple target media content may be implanted into the multiple frames according to the mask information of the panel area in each frame, thereby integrating the first media content into the target video. The above optimization may be an affine change, so that the target media content and the corresponding frame image are in the same vector space, thereby improving the video fusion effect.

[0055] The video content implantation method of the embodiment of the present application, after obtaining the first media content and the multi-frame images containing the panel area in the target video, can input the target frame image into the first model for prediction, obtain the corner point information and mask information of the panel area in the target frame image, and determine the corner point information and mask information of the panel area in other frame images in the multi-frame images except the target frame image based on the corner point information and mask information of the panel area in the target frame image, and implant the first media content into the target video based on the corner point information and mask information of the panel area in each frame image of the multi-frame images. Thus, when implanting content into the video, the position information and occlusion of the panel area in the video can be considered, so as to ensure the naturalness and continuity of the video after the implantation of the content, and improve the authenticity of the video after the implantation of the content.

[0056] Optionally, when determining the corner point information and mask information of the panel area in other frame images except the target frame image in the multi-frame image according to the corner point information and mask information of the panel area in the target frame image and the timing information between the multi-frame images, the obtained corner point information and mask information can be transferred using a timing transfer model based on deep learning to obtain the corner point information and mask information of the panel area in all frame images. When the target frame image is the first frame image of the multi-frame image, such as Figure 4 As shown, the specific process may include:

[0057] The following process is executed cyclically to determine the corner point information and mask information of the panel area in each frame image of other frame images:

[0058] S1: The current frame image I t and the previous frame image I of the current frame image t-1 Input the second model ST for feature extraction to obtain the spatiotemporal feature Feat ST For example, the spatiotemporal features include but are not limited to spatial and temporal features; Figure 4 As shown, the second model ST can be called a spatiotemporal feature extraction model, including a convolutional neural network CNN, which can be trained based on deep learning and actual needs.

[0059] S2: The previous frame image I t-1 Corner point information P t-1 Input the third model Att for feature extraction to obtain the previous frame image I t-1 Corner feature Feat Att , and the previous frame image I t-1 The mask information M t-1 , input the fourth model MM for feature extraction, and obtain the previous frame image I t-1 The mask feature Feat MaskFor example, when inputting the corner point information into the third model Att for feature extraction, the corner point information can be first Gaussian transformed to obtain a corner point heat map, and then the corner point heat map is input into the third model Att for feature extraction to obtain corner point features, which include but are not limited to attention features such as position. The mask features include but are not limited to position features, shape features, etc. Figure 4 As shown, the third model Att can be called a corner attention feature extraction model, including a convolutional neural network CNN, which can be trained based on deep learning and actual needs. The fourth model MM can be called a mask feature extraction model, including a convolutional neural network CNN, which can be trained based on deep learning and actual needs.

[0060] S3: Spatiotemporal features Feat ST and mask feature Feat Mask Fusion is performed to obtain the first fusion feature Feat Mask_fuse The fusion of the spatiotemporal features and the mask features may be to adjust the mask features based on the spatiotemporal features to obtain mask features that are closer to the current frame image.

[0061] S4: For the first fusion feature Feat Mask_fuse And corner feature Feat Att Fusion is performed to obtain the second fusion feature Feat fuse In this way, on the basis of the fusion of spatiotemporal features and mask features, further fusion with corner point features can strengthen the position information of corner points, and at the same time use the corner points as reference information of the mask, thereby highlighting the position and content of effective mask information, making the subsequent prediction effect more accurate.

[0062] S5: The second fusion feature Feat fuse Input the fifth model Pred for prediction and obtain the corner point information P of the current frame image t and mask information M t For example, Figure 4 As shown, the fifth model Pred can be called a prediction model, including a convolutional neural network CNN, which can be trained based on deep learning and actual needs. The fifth model can be trained using the loss function in the above formula (1).

[0063] Wherein, when the above S1 to S5 are executed for the first time, the previous frame image is the target frame image.

[0064] In this way, with the help of the above-mentioned loop process, the corner point information and mask information of the panel area in the subsequent frames of the target frame image (first frame image) can be determined in sequence, thereby determining the corner point information and mask information of the panel area in each frame image.

[0065] Optionally, the above-mentioned process of fusing the first fused feature with the corner feature may include: firstly, performing background matching on the first fused feature according to the mask information of the previous frame image to obtain a third fused feature; then, fusing the third fused feature with the corner feature to obtain a second fused feature. In this way, by performing background matching on the first fused feature with the help of the mask information of the previous frame image, the mask feature in the first fused feature can be strengthened, thereby strengthening the subsequent fusion effect with the corner feature.

[0066] Optionally, the process of fusing the first fusion feature with the corner feature may include:

[0067] The second fusion feature Feat is obtained using the following formula: fuse :

[0068] Feat fuse =Feat mid +Feat Mask_fuse

[0069]

[0070] Among them, Feat Mask_fuse It is the first fusion feature. Mask_fuse,m Feat is the first fusion feature Mask_fuse The mth value of Feat Att,n Feat is the corner feature Att The nth value of , where N is the corner feature Feat Att The number of eigenvalues ​​in . exp() represents the exponential function. mid,mn Feat mid The value of the mth row and nth column in represents the value of Feat Mask_fuse,m With Feat Att,n The product of is normalized again after passing through the exponential function to balance the fusion results of mask features and corner features.

[0071] It is understandable that the above-mentioned mask features, corner features, first fusion features and second fusion features are specifically feature matrices, which can be set based on actual needs. The above-mentioned second model, third model, fourth model and fifth model can be trained using existing model training methods, which is not limited.

[0072] See also Figure 5 , Figure 5 is a flow chart of a model training method provided in an embodiment of the present application, the method is applied to electronic devices, such as Figure 5 As shown, the method comprises the following steps:

[0073] Step 51: Obtain an image sample set to be trained.

[0074] In this embodiment, each image sample in the above image sample set is annotated with a corner point label and a mask label. The corner point label is used to characterize the corner point information of the panel area in the image sample, and the mask label is used to characterize the mask information of the panel area in the image sample.

[0075] In some embodiments, the image samples in the above-mentioned image sample set may be obtained by decoding a sample video. The sample video may come from a file or a camera device.

[0076] Step 52: Input the image sample set into a preset model for training to obtain a first model.

[0077] In this embodiment, the above-mentioned first model may include a first sub-model and a second sub-model. The first sub-model is used to predict the corner point information of the panel area, such as the corner point coordinates, to determine the position information and posture of the panel area. The loss function of the first sub-model may be cross entropy or any other loss function, which may be selected according to different scenarios and is not limited to this. The second sub-model is used to predict the mask information of the panel area, such as the mask coordinates, to determine the occlusion of the panel area. The loss function of the second sub-model may use the corner point information predicted by the first sub-model to supervise / correct the prediction results of the second sub-model, which may be selected according to different scenarios. For example, see Figure 6 As shown, during the training process of the first model, the corner point coordinates predicted by the first sub-model can be used for self-supervision, thereby improving the model training effect.

[0078] In some embodiments, the first model may be selected as a multi-task deep learning model.

[0079] The model training method of the embodiment of the present application uses the corner point information predicted by the first sub-model with the help of the loss function of the second sub-model to supervise / correct the prediction results of the second sub-model. In the model training process, those mask segmentation results that exceed the area enclosed by the predicted corner points can be effectively punished, thereby ensuring the stability and convergence of the model training process, enhancing the learning effect of the model, and improving the performance and convergence speed of the model.

[0080] In some embodiments, the loss function L of the second sub-model is seg_point Can be:

[0081]

[0082] Where (i, j) is the mask coordinate predicted by the second sub-model. point is the area formed by the corner points corresponding to the corner point coordinates predicted by the first sub-model. dist(ij,A point ) is the coordinate point (i, j) to area A pointα is the preset weight coefficient, which can be adjusted according to different scenarios. seg The second sub-model is located in region A at coordinate point (i, j) point The loss function of the inner time can be cross entropy or any other loss function, which can be selected according to different scenarios. seg +α*L seg *dist(ij,A point ) is the second sub-model at coordinate point (i, j) located in area A point The loss function when it is outside the range of L seg A penalty term is added on the basis of the predicted corner point information, and the predicted corner point information is used as additional auxiliary information / supervisory information to ensure that the mask result predicted by the second sub-model does not exceed the area enclosed by the corner points predicted by the first sub-model, making the edge of the mask segmentation more accurate.

[0083] It should be pointed out that the above loss function L seg_point The meanings may include: 1) When the mask predicted by the second sub-model is within the area enclosed by the corner points predicted by the first sub-model, the loss function of the second sub-model is the segmentation loss function L seg ; 2) When the mask predicted by the second sub-model is outside the area enclosed by the corner points predicted by the first sub-model, the loss function of the second sub-model is divided into two parts. The first part is the original segmentation loss function L seg , the second part is a loss function that changes dynamically according to the shortest distance from the current predicted mask coordinates to the area enclosed by the predicted corner points. This second part of the loss function can effectively penalize those mask segmentation results that exceed the area enclosed by the predicted corner points, ensuring that the mask results predicted by the second sub-model do not exceed the area enclosed by the corner points predicted by the first sub-model, making the edge of the mask segmentation more accurate. At the same time, the loss function changes dynamically according to the shortest distance from the current predicted mask coordinates to the area enclosed by the predicted corner points, which can ensure the stability and convergence of the model training process, strengthen the learning effect of the model, and improve the performance and convergence speed of the model.

[0084] It should be noted that the video content implantation method provided in the embodiment of the present application can be performed by a video content implantation device, or a control module in the video content implantation device for performing the video content implantation method. In the embodiment of the present application, the video content implantation device performing the video content implantation method is taken as an example to illustrate the video content implantation device provided in the embodiment of the present application.

[0085] See also Figure 7 , Figure 7 is a structural diagram of a video content embedding device provided in an embodiment of the present application, and the device is applied to electronic devices, such as Figure 7As shown, the video content embedding device 70 includes:

[0086] A first acquisition module 71 is used to acquire a first media content and a plurality of frames of images including a panel area in a target video;

[0087] A prediction module 72 is used to input the target frame image into the first model for prediction, and obtain the corner point information and mask information of the panel area in the target frame image; wherein the target frame image is one frame image in the multiple frame images, and the mask information is predicted under the supervision of the corner point information;

[0088] A determination module 73, configured to determine the corner point information and mask information of the panel area in other frame images in the multiple frame images except the target frame image according to the corner point information and mask information of the panel area in the target frame image;

[0089] The embedding module 74 is used to embed the first media content into the target video according to the corner point information and mask information of the panel area in each frame of the multiple frames of images.

[0090] In some embodiments, according to actual conditions, the target video may be sourced from a file or a camera device.

[0091] In some embodiments, the prediction module 72, determination module 73 and implantation module 74 are integrated with computing modules such as Pytorch, OpenCV and their related dependent third-party libraries to implement related logical functions.

[0092] Optionally, the first model includes a first sub-model and a second sub-model, the first sub-model is used to predict the corner point information of the panel area, the second sub-model is used to predict the mask information of the panel area, and the loss function of the second sub-model uses the corner point information predicted by the first sub-model to supervise the prediction results of the second sub-model.

[0093] Optionally, the loss function L of the second sub-model is seg_point for:

[0094]

[0095] Among them, (i, j) is the mask coordinate predicted by the second sub-model, A point is the area formed by the corner points corresponding to the corner point coordinates predicted by the first sub-model, dist(ij,A point ) is the coordinate point (i, j) to area A point The shortest distance, α is the preset weight coefficient, L seg The second sub-model is located in region A at coordinate point (i, j) pointThe loss function of the inner time, L seg +α*L seg *dist(ij,A point ) is the coordinate point (i, j) of the second sub-model located in area A point The loss function outside of .

[0096] Optionally, the determination module 73 is specifically used for:

[0097] According to the corner point information and mask information of the panel area in the target frame image and the timing information between the multiple frame images, the corner point information and mask information of the panel area in each frame image of the other frame images are determined.

[0098] Optionally, the target frame image is the first frame image of the multiple frame images, and the determination module 73 is specifically used to: loop through the following process to determine the corner point information and mask information of the panel area in each frame image of the other frame images:

[0099] S1: Inputting a current frame image and a previous frame image of the current frame image into a second model for feature extraction to obtain spatiotemporal features;

[0100] S2: inputting the corner point information of the previous frame image into the third model for feature extraction to obtain the corner point features of the previous frame image, and inputting the mask information of the previous frame image into the fourth model for feature extraction to obtain the mask features of the previous frame image;

[0101] S3: fusing the spatiotemporal feature with the mask feature to obtain a first fused feature;

[0102] S4: Fusing the first fusion feature with the corner feature to obtain a second fusion feature;

[0103] S5: inputting the second fusion feature into a fifth model for prediction to obtain corner point information and mask information of the current frame image;

[0104] Wherein, when S1 to S5 are executed for the first time, the previous frame image is the target frame image.

[0105] Optionally, the determination module 73 is further used for:

[0106] The second fusion feature Feat is obtained using the following formula: fuse :

[0107] Feat fuse =Feat mid +Feat Mask_fuse

[0108]

[0109] Among them, Feat Mask_fuse is the first fusion feature, Feat mid,mn Feat mid The value of the mth row and nth column in Feat Mask_fuse,m The first fusion feature Feat Mask_fuse The mth value of Feat Att,n is the corner feature Feat Att The nth value of N is the corner feature Feat Att The number of eigenvalues ​​in , exp() represents the exponential function.

[0110] Optionally, the determination module 73 is further used to: perform background matching on the first fusion feature according to the mask information of the previous frame image to obtain a third fusion feature; and fuse the third fusion feature with the corner feature to obtain the second fusion feature.

[0111] Optionally, the implant module 74 includes:

[0112] an optimization unit, configured to optimize the first media content according to the corner point information of the panel area in each frame of the multiple frames of images, to obtain multiple target media contents;

[0113] The implantation unit is used to implant the multiple target media contents into the multiple frames of images according to the mask information of the panel area in each frame of the image.

[0114] The video content embedding device 70 of the embodiment of the present application can realize the above Figure 2 The various processes of the method embodiment shown can achieve the same technical effect, and will not be described again here to avoid repetition.

[0115] See also Figure 8 , Figure 8 is a schematic diagram of a model training device provided in an embodiment of the present application, which is applied to electronic devices such as Figure 8 As shown, the model training device 80 includes:

[0116] A second acquisition module 81 is used to acquire an image sample set to be trained, wherein each image sample in the image sample set is annotated with a corner point label and a mask label, the corner point label is used to characterize the corner point information of the panel area in the image sample, and the mask label is used to characterize the mask information of the panel area in the image sample;

[0117] The training module 82 is used to input the image sample set into a preset model for training to obtain a first model, wherein the first model includes a first sub-model and a second sub-model, the first sub-model is used to predict the corner point information of the panel area, the second sub-model is used to predict the mask information of the panel area, and the loss function of the second sub-model uses the corner point information predicted by the first sub-model to supervise the prediction result of the second sub-model.

[0118] Optionally, the loss function L of the second sub-model is seg_point for:

[0119]

[0120] Among them, (i, j) is the mask coordinate predicted by the second sub-model, A point is the area formed by the corner points corresponding to the corner point coordinates predicted by the first sub-model, dist(ij,A point ) is the coordinate point (i, j) to area A point The shortest distance, α is the preset weight coefficient, L seg The second sub-model is located in region A at coordinate point (i, j) point The loss function of the inner time, L seg +α*L seg *dist(ij,A point ) is the coordinate point (i, j) of the second sub-model located in area A point The loss function outside of .

[0121] The video content model training device 80 of the embodiment of the present application can achieve the above Figure 5 The various processes of the method embodiment shown can achieve the same technical effect, and will not be described again here to avoid repetition.

[0122] Optional, such as Fig. 9 As shown, the embodiment of the present application also provides an electronic device 90, which mainly includes one or more processors 91, a memory 92, and a program or instruction stored in the memory 92 and executable on the processor 91. When the program or instruction is executed by the processor 91, the various processes of the above-mentioned video content implantation method embodiment are implemented, or the various processes of the above-mentioned model training method embodiment are implemented, and the same technical effect can be achieved. In order to avoid repetition, it will not be repeated here.

[0123] In some embodiments, the processor 91 may be in the form of, but not limited to, a CPU, FPGA, GPU, TPU, NPU, ASIC, SOC, etc., and may be a combination of any one or more thereof.

[0124] In some embodiments, the memory 92 is used to store all long-term and short-term computing data in the relevant computing process. It can exist in the form of physical media such as mobile phone memory, digital camera memory, USB flash drive, mobile hard disk, CD-ROM optical disk storage device, etc. It can be composed of a single or multiple physical media, and its logical media includes Read-Only Memory (ROM) and / or Random Access Memory (RAM) modules.

[0125] The present application also provides a readable storage medium on which a program or instruction is stored. When the program or instruction is executed by a processor, the above Figure 2 or Figure 5 The various processes of the method embodiment shown can achieve the same technical effect, and to avoid repetition, they will not be described again here.

[0126] Computer readable media include permanent and non-permanent, removable and non-removable media, and can be implemented by any method or technology to store information. Information can be computer readable instructions, data structures, program modules or other data. Examples of computer storage media include, but are not limited to, phase change memory (PRAM), static random access memory (SRAM), dynamic random access memory (DRAM), other types of random access memory (RAM), read-only memory (ROM), electrically erasable programmable read-only memory (EEPROM), flash memory or other memory technology, compact disk read-only memory (CD-ROM), digital versatile disk (DVD) or other optical storage, magnetic cassettes, magnetic tape magnetic disk storage or other magnetic storage devices or any other non-transmission media that can be used to store information that can be accessed by a computing device. As defined herein, computer readable media does not include temporary computer readable media (transitory media), such as modulated data signals and carrier waves.

[0127] It should be noted that, in this article, the terms "include", "comprises" or any other variations thereof are intended to cover non-exclusive inclusion, so that a process, method, article or device including a series of elements includes not only those elements, but also other elements not explicitly listed, or also includes elements inherent to such process, method, article or device. In the absence of further restrictions, an element defined by the sentence "comprises a ..." does not exclude the existence of other identical elements in the process, method, article or device including the element.

[0128] The serial numbers of the above-mentioned embodiments of the present application are for description only and do not represent the advantages or disadvantages of the embodiments.

[0129] Through the description of the above implementation methods, those skilled in the art can clearly understand that the above-mentioned embodiment methods can be implemented by means of software plus a necessary general hardware platform, and of course by hardware, but in many cases the former is a better implementation method. Based on this understanding, the technical solution of the present application, or the part that contributes to the prior art, can be embodied in the form of a software product, which is stored in a storage medium (such as ROM / RAM, magnetic disk, optical disk), and includes a number of instructions for enabling a service classification device (which can be a mobile phone, computer, server, air conditioner, or network device, etc.) to execute the methods described in each embodiment of the present application.

[0130] The above is only a preferred implementation of the present application. It should be pointed out that for ordinary technicians in this technical field, several improvements and modifications can be made without departing from the principles of the present application. These improvements and modifications should also be regarded as the scope of protection of the present application.

Claims

1. A method for embedding video content, characterized in that: include: Acquire the first media content and a plurality of frames of images including a panel area in the target video; Inputting the target frame image into the first model for prediction, obtaining corner point information and mask information of the panel area in the target frame image; wherein the target frame image is one frame image in the multiple frame images, and the mask information is predicted under the supervision of the corner point information; Determine the corner point information and mask information of the panel area in other frame images in the multiple frame images except the target frame image according to the corner point information and mask information of the panel area in the target frame image; The first media content is implanted into the target video according to the corner point information and the mask information of the panel area in each frame of the multiple frames of images.

2. The method according to claim 1, characterized in that The first model includes a first sub-model and a second sub-model. The first sub-model is used to predict the corner point information of the panel area, and the second sub-model is used to predict the mask information of the panel area. The loss function of the second sub-model uses the corner point information predicted by the first sub-model to supervise the prediction results of the second sub-model.

3. The method according to claim 2, characterized in that The loss function L of the second sub-model is seg_point for: Among them, (i, j) is the mask coordinate predicted by the second sub-model, A point is the area formed by the corner points corresponding to the corner point coordinates predicted by the first sub-model, dist(ij,A point ) is the coordinate point (i, j) to area A point The shortest distance, α is the preset weight coefficient, L seg The second sub-model is located in region A at coordinate point (i, j) point The loss function of the inner time, L seg +α*L seg *dist(ij,A point ) is the coordinate point (i, j) of the second sub-model located in area A point The loss function outside of .

4. The method according to claim 1, characterized in that: The determining, based on the corner point information and mask information of the panel area in the target frame image, the corner point information and mask information of the panel area in other frame images in the multiple frame images except the target frame image, comprises: According to the corner point information and mask information of the panel area in the target frame image and the timing information between the multiple frame images, the corner point information and mask information of the panel area in each frame image of the other frame images are determined.

5. The method according to claim 4, characterized in that The target frame image is a first frame image of the multiple frame images, and determining the corner point information and mask information of the panel area in each frame image of the other frame images according to the corner point information and mask information of the panel area in the target frame image and the timing information between the multiple frame images includes: The following process is executed in a loop to determine the corner point information and mask information of the panel area in each frame image of the other frame images: S1: Inputting a current frame image and a previous frame image of the current frame image into a second model for feature extraction to obtain spatiotemporal features; S2: inputting the corner point information of the previous frame image into the third model for feature extraction to obtain the corner point features of the previous frame image, and inputting the mask information of the previous frame image into the fourth model for feature extraction to obtain the mask features of the previous frame image; S3: fusing the spatiotemporal feature with the mask feature to obtain a first fused feature; S4: Fusing the first fusion feature with the corner feature to obtain a second fusion feature; S5: inputting the second fusion feature into a fifth model for prediction to obtain corner point information and mask information of the current frame image; Wherein, when S1 to S5 are executed for the first time, the previous frame image is the target frame image.

6. The method according to claim 5, characterized in that The fusing the first fused feature with the corner feature to obtain a second fused feature includes: The second fusion feature Feat is obtained using the following formula: fuse : Feat fuse =Feat mid +Feat Mask_fuse Among them, Feat Mask_fuse is the first fusion feature, Feat mid,mn Feat mid The value of the mth row and nth column in Feat Mask_fuse,m The first fusion feature Feat Mask_fuse The mth value of Feat Att,n is the corner feature Feat Att The nth value of N is the corner feature Feat Att The number of eigenvalues ​​in , exp() represents the exponential function.

7. The method according to claim 5, characterized in that The fusing the first fused feature with the corner feature to obtain a second fused feature includes: Performing background matching on the first fusion feature according to the mask information of the previous frame image to obtain a third fusion feature; The third fusion feature is fused with the corner feature to obtain the second fusion feature.

8. The method according to any one of claims 1 to 7, characterized in that: The step of implanting the first media content into the target video according to the corner point information and the mask information of the panel area in each frame of the multiple frames of images includes: According to the corner point information of the panel area in each frame of the image, the first media content is optimized respectively to obtain a plurality of target media contents; The multiple target media contents are implanted into the multiple frames of images according to the mask information of the panel area in each frame of image.

9. A video content embedding device, characterized in that: include: A first acquisition module is used to acquire the first media content and a plurality of frames of images including a panel area in a target video; A prediction module, used for inputting a target frame image into a first model for prediction, and obtaining corner point information and mask information of a panel area in the target frame image; wherein the target frame image is one frame image in the multiple frame images, and the mask information is predicted under the supervision of the corner point information; A determination module, configured to determine the corner point information and mask information of the panel area in other frame images in the multiple frame images except the target frame image according to the corner point information and mask information of the panel area in the target frame image; The implantation module is used to implant the first media content into the target video according to the corner point information and mask information of the panel area in each frame of the multiple frames of images.

10. An electronic device, characterized in that: The method comprises a processor, a memory, and a program or instruction stored in the memory and executable on the processor, wherein the program or instruction, when executed by the processor, implements the steps of the method for embedding video content as described in any one of claims 1 to 8.

11. A readable storage medium, characterized in that: The readable storage medium stores a program or instruction, and when the program or instruction is executed by a processor, the steps of the video content implantation method according to any one of claims 1 to 8 are implemented.

Images