Live video enhancement method and device, equipment and storage medium

Abstract

The invention discloses a live broadcast video enhancement method, device and equipment and a storage medium, and belongs to the technical field of video processing, and the method comprises the steps: obtaining an original live broadcast video, and extracting multiple frames of to-be-processed pictures from the original live broadcast video according to the frame extraction frequency and the live broadcast duration of the original live broadcast video; performing target detection on the to-be-processed picture to obtain at least one target picture corresponding to the to-be-processed picture; determining a target picture type of each target picture according to candidate picture vectors in a preset picture vector library; generating a target mask corresponding to the target picture according to the target picture type of the target picture; and performing video enhancement processing on the original live video and the target mask corresponding to each target picture by adopting the trained video enhancement model to obtain a new live video. According to the invention, the calculation burden and cost in the live video enhancement process are reduced, and the definition and visual effect of the key content in the live video are improved, so that the overall visual effect of the live video is improved.

Description

Technical Field

[0001] The present invention relates to the field of video processing technology, and in particular to a live video enhancement method, device, equipment and storage medium. Background Art

[0002] With the rapid development of live e-commerce, the quality of live video has become a key factor in attracting users and improving the shopping experience. In actual live broadcast scenarios, since the audience's attention is mainly focused on the anchor and the displayed products, there is no strong demand for enhancing background content.

[0003] However, existing video enhancement technologies often enhance the picture content of the entire live video to the same extent. Although the video quality is improved to a certain extent, it fails to accurately enhance the key picture content in the live video (such as the host, model, and product, etc.) according to the characteristics of the live broadcast scene. This not only increases the live broadcast cost and the computational burden, but also limits the improvement in the visual effect of the live video. Summary of the invention

[0004] The present invention provides a live video enhancement method, device, equipment and storage medium to accurately enhance key picture content in the live video, reduce costs, reduce computing burden, improve the overall visual effect of the live video, and enhance the overall viewing experience of the audience.

[0005] According to one aspect of the present invention, a live video enhancement method is provided, the method comprising:

[0006] The original live video is obtained, and multiple frames of images to be processed are extracted from the original live video according to the frame extraction frequency and the live broadcast duration of the original live video;

[0007] Performing target detection on the image to be processed to obtain at least one target image corresponding to the image to be processed;

[0008] Determine the target image type of each target image according to the candidate image vectors in the preset image vector library;

[0009] Generate a target mask corresponding to the target image according to the target image type of the target image;

[0010] The trained video enhancement model is used to perform video enhancement processing on the original live video and the target masks corresponding to each target image to obtain a new live video.

[0011] According to another aspect of the present invention, a live video enhancement device is provided, the device comprising:

[0012] The to-be-processed picture extraction module is used to obtain the original live video and extract multiple frames of to-be-processed pictures from the original live video according to the frame extraction frequency and the live broadcast duration of the original live video;

[0013] A target image determination module, used to perform target detection on the image to be processed and obtain at least one target image corresponding to the image to be processed;

[0014] A picture type determination module, used to determine a target picture type of each target picture according to candidate picture vectors in a preset picture vector library;

[0015] A target mask generation module, used to generate a target mask corresponding to the target image according to the target image type of the target image;

[0016] The new live video determination module is used to use the trained video enhancement model to perform video enhancement processing on the original live video and the target masks corresponding to each target image to obtain a new live video.

[0017] According to another aspect of the present invention, an electronic device is provided, the electronic device comprising:

[0018] at least one processor; and

[0019] a memory communicatively connected to at least one processor; wherein,

[0020] The memory stores a computer program that can be executed by at least one processor, and the computer program is executed by at least one processor so that the at least one processor can execute the live video enhancement method of any embodiment of the present invention.

[0021] According to another aspect of the present invention, a computer-readable storage medium is provided, wherein the computer-readable storage medium stores computer instructions, and the computer instructions are used to enable a processor to implement the live video enhancement method of any embodiment of the present invention when executed.

[0022] According to another aspect of the present invention, a computer program product is provided, including a computer program, and when the computer program is executed by a processor, the live video enhancement method of any embodiment of the present invention is implemented.

[0023] The technical solution of the embodiment of the present invention is to obtain the original live video, and extract multiple frames of to-be-processed pictures from the original live video according to the frame extraction frequency and the live broadcast duration of the original live video; perform target detection on the pictures to be processed to obtain at least one target picture corresponding to the pictures to be processed; determine the target picture type of each target picture according to the candidate picture vectors in a preset picture vector library; generate a target mask corresponding to the target picture according to the target picture type of the target picture; use a trained video enhancement model to perform video enhancement processing on the original live video and the target masks corresponding to each target picture to obtain a new live video. The above technical scheme reduces the amount of data processing in the live video enhancement process by performing frame extraction processing on the original live video, thereby reducing the computational burden in the live video enhancement process; obtains the object sub-image (i.e., target image) corresponding to each image to be processed by performing target detection on each image to be processed; then, determines the image type of each object sub-image according to the candidate image vector in the preset image vector library, so as to distinguish the criticality of all object sub-images according to the actual live scene requirements, so that the trained video enhancement model can accurately enhance the key picture content in the original live video, thereby avoiding unnecessary enhancement processing of the picture content of the entire live video, reducing the cost of the live video enhancement process, and improving the clarity and visual effect of the key content in the live video, thereby improving the overall visual effect of the live video and improving the overall viewing experience of the audience.

[0024] It should be understood that the contents described in this section are not intended to identify the key or important features of the embodiments of the present invention, nor are they intended to limit the scope of the present invention. Other features of the present invention will become easily understood through the following description. BRIEF DESCRIPTION OF THE DRAWINGS

[0025] In order to more clearly illustrate the technical solutions in the embodiments of the present invention, the following briefly introduces the drawings required for use in the description of the embodiments. Obviously, the drawings described below are only some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without creative work.

[0026] Figure 1 is a flow chart of a live video enhancement method provided according to Embodiment 1 of the present invention;

[0027] Figure 2A is a flow chart of a live video enhancement method provided according to Embodiment 2 of the present invention;

[0028] Figure 2B is a structural diagram of an upsampling sublayer in a core processing layer in a trained video enhancement model provided in Embodiment 2 of the present invention;

[0029] Figure 2C is a schematic diagram of a picture sampling rule provided according to Embodiment 2 of the present invention;

[0030] Figure 3 is a structural schematic diagram of a live video enhancement device provided according to Embodiment 3 of the present invention;

[0031] Figure 4 It is a structural schematic diagram of an electronic device for implementing the live video enhancement method according to an embodiment of the present invention. DETAILED DESCRIPTION

[0032] In order to enable those skilled in the art to better understand the scheme of the present invention, the technical scheme in the embodiments of the present invention will be clearly and completely described below in conjunction with the drawings in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work should fall within the scope of protection of the present invention.

[0033] It should be noted that the terms "target", "candidate", "first" and "second" in the specification and claims of the present invention and the above-mentioned drawings are used to distinguish similar objects, and are not necessarily used to describe a specific order or sequence. It should be understood that the data used in this way can be interchangeable where appropriate, so that the embodiments of the present invention described herein can be implemented in an order other than those illustrated or described herein. In addition, the terms "including" and "having" and any variations thereof are intended to cover non-exclusive inclusions, for example, a process, method, system, product or device that includes a series of steps or units is not necessarily limited to those steps or units that are clearly listed, but may include other steps or units that are not clearly listed or inherent to these processes, methods, products or devices.

[0034] In addition, it should be noted that in the technical solution of the present invention, the collection, storage, use, processing, transmission, provision and disclosure of the similarity thresholds and candidate image vectors in the preset image vector library are in compliance with the provisions of relevant laws and regulations and do not violate public order and good morals.

[0035] Embodiment 1

[0036] Figure 1 This is a flowchart of a live video enhancement method provided in Embodiment 1 of the present invention. This embodiment is applicable to the case of accurately enhancing key picture content in a live video. The method can be performed by a live video enhancement device, which can be implemented in the form of hardware and / or software and can be configured in an electronic device. Figure 1 As shown, the method includes:

[0037] S101, obtaining an original live video, and extracting a plurality of frames of to-be-processed pictures from the original live video according to a frame extraction frequency and a live broadcast duration of the original live video.

[0038] The original live video refers to the unprocessed live video. The frame extraction frequency refers to the number of video frames extracted from the original live video per second; optionally, the frame extraction frequency can be preset according to actual business needs, for example, the frame extraction frequency can be 1 frame / second, which is not specifically limited in the embodiment of the present invention. The live broadcast duration refers to the total live broadcast duration of the original live video.

[0039] Specifically, the original live video can be obtained through the recording device on the terminal device; and the original live video can be extracted according to the frame extraction frequency and the live broadcast duration of the original live video. Frames of images to be processed are used to reduce the amount of data processing and the computational burden in the live video enhancement process. Where L represents the live broadcast duration of the original live video, and f represents the frame extraction frequency.

[0040] S102: Perform target detection on the image to be processed to obtain at least one target image corresponding to the image to be processed.

[0041] The target image refers to an image that contains a detection object in the image to be processed, and is a sub-image of the image to be processed. The detection object refers to an entity in the image to be processed; optionally, the detection object includes but is not limited to anchors, models, and commodities. It should be noted that one image to be processed corresponds to at least one target image, and one target image only contains one detection object in the image to be processed.

[0042] Specifically, for each image to be processed extracted from the original live video, a target detection model can be used to perform target detection on each image to be processed, and at least one target image corresponding to each image to be processed can be obtained. The target detection model can be preset according to actual business needs. For example, the target detection model can be a Yolo (You Only Look Once) model, which is not specifically limited in the embodiment of the present invention.

[0043] S103: Determine a target image type of each target image according to candidate image vectors in a preset image vector library.

[0044] The preset image vector library refers to a database configured in advance according to the actual live broadcast scene, which is used to store image vectors of objects that may appear in the actual live broadcast scene. The target image type refers to the image type to which the target image belongs; optionally, the target image type can be one of a foreground image, a foreign object image, and a background image.

[0045] Among them, the candidate image vector refers to the image vector stored in the preset image vector library; optionally, the candidate image vector includes but is not limited to the foreground image vector and the foreign image vector. The foreign image vector refers to the image vector of the object that needs to be shielded in the actual live broadcast scene. For example, the foreign image vector can be the image vector of the object, text or portrait that violates the regulations of the live broadcast platform. It should be noted that both the foreground image vector and the foreign image vector need to be specified in advance according to the actual live broadcast scene. For example, the image vector of the product, the image vector of the model and the image vector of the anchor are specified in advance as foreground image vectors.

[0046] Specifically, for each target image, graph embedding processing is performed on the target image to obtain a target graph vector corresponding to the target image; based on the target graph vector, a matching graph vector that matches the target graph vector is screened out from candidate graph vectors in a preset graph vector library; based on the vector type of the matching graph vector, the target image type of the target image is determined.

[0047] The target image vector refers to the vector obtained after the image embedding process of the target image, and the matching image vector refers to the candidate image vector in the preset image vector library that matches the target image vector.

[0048] More specifically, for each target image obtained, the target image is input into the image vector embedding model, and after being processed by the image vector embedding model, a target image vector corresponding to the target image is obtained; the target image vector is similarity matched with each candidate image vector in the preset image vector library to obtain a similarity matching result; according to the similarity matching result and the similarity threshold, a matching image vector that matches the target image vector is screened out from the candidate image vectors in the preset image vector library; if there is only one matching image vector, the target image type of the target image is determined according to the vector type of the matching image vector; if there are multiple matching image vectors, the matching image vector with the largest similarity value with the target image vector is screened out from these matching image vectors as the target matching image vector; according to the vector type of the target matching image vector, the target image type of the target image is determined; if there is no matching image vector, the target image type of the target image is determined to be a background image.

[0049] Among them, the graph vector embedding model can be pre-set according to actual business needs. For example, the graph vector embedding model can be a CLIP (Contrastive Language-Image Pre-training) model. For example, the graph vector embedding model can be a ViT (Vision Transformer) model. The embodiments of the present invention do not specifically limit it.

[0050] The similarity matching result includes a similarity value obtained after similarity matching between the target graph vector and each candidate graph vector in the preset graph vector library. The similarity threshold can be preset according to actual business requirements, and the embodiment of the present invention does not specifically limit it.

[0051] Among them, according to the similarity matching result and the similarity threshold, a matching graph vector that matches the target graph vector is screened out from the candidate graph vectors in the preset graph vector library. Specifically, it can be: if the similarity value obtained after the target graph vector is similarly matched with a candidate graph vector in the preset graph vector library is greater than the similarity threshold, then the candidate graph vector is determined as the matching graph vector that matches the target graph vector.

[0052] Among them, according to the vector type of the matching image vector, the target image type of the target image is determined, specifically: if the vector type of the matching image vector is a foreground image vector, the target image type of the target image is determined to be a foreground image; if the vector type of the matching image vector is a foreign image vector, the target image type of the target image is determined to be a foreign image.

[0053] S104: Generate a target mask corresponding to the target image according to the target image type of the target image.

[0054] Among them, the target mask refers to the mask corresponding to the target image. Specifically, the target mask corresponding to the target image can be generated by the mask generation tool according to the target image type of the target image. For example, if the target image type of the target image is a foreground image, the target mask generated by the mask generation tool is a foreground mask. For another example, if the target image type of the target image is a foreign body image, the target mask generated by the mask generation tool is a foreign body mask. Among them, the mask generation tool can be pre-set according to actual business needs, and the embodiment of the present invention does not specifically limit it.

[0055] S105: Using the trained video enhancement model, perform video enhancement processing on the original live video and the target masks corresponding to each target image to obtain a new live video.

[0056] The video enhancement model can be obtained by training a neural network model using a sample live video and a sample mask related to the sample live video. The new live video refers to a live video obtained after the original live video is processed by video enhancement.

[0057] Specifically, the original live video and the target masks corresponding to each target image may be input into a trained video enhancement model, and a new live video may be obtained after being processed by the trained video enhancement model.

[0058] The technical solution of the embodiment of the present invention is to obtain the original live video, and extract multiple frames of to-be-processed pictures from the original live video according to the frame extraction frequency and the live broadcast duration of the original live video; perform target detection on the pictures to be processed to obtain at least one target picture corresponding to the pictures to be processed; determine the target picture type of each target picture according to the candidate picture vectors in a preset picture vector library; generate a target mask corresponding to the target picture according to the target picture type of the target picture; use a trained video enhancement model to perform video enhancement processing on the original live video and the target masks corresponding to each target picture to obtain a new live video. The above technical scheme reduces the amount of data processing in the live video enhancement process by performing frame extraction processing on the original live video, thereby reducing the computational burden in the live video enhancement process; obtains the object sub-image (i.e., target image) corresponding to each image to be processed by performing target detection on each image to be processed; then, determines the image type of each object sub-image according to the candidate image vector in the preset image vector library, so as to distinguish the criticality of all object sub-images according to the actual live scene requirements, so that the trained video enhancement model can accurately enhance the key picture content in the original live video, thereby avoiding unnecessary enhancement processing of the picture content of the entire live video, reducing the cost of the live video enhancement process, and improving the clarity and visual effect of the key content in the live video, thereby improving the overall visual effect of the live video and improving the overall viewing experience of the audience.

[0059] Embodiment 2

[0060] Figure 2A This is a flow chart of a live video enhancement method provided in Example 2 of the present invention. Based on the above-mentioned embodiments, this embodiment further optimizes "using a trained video enhancement model to perform video enhancement processing on the original live video and the target masks corresponding to each target image to obtain a new live video", and provides an optional implementation scheme. It should be noted that the trained video enhancement model includes a coding layer, a decoding layer and at least one core processing layer; the coding layer is connected to at least one core processing layer; the decoding layer is connected to at least one core processing layer. It should also be noted that for the parts not described in detail in the embodiments of the present invention, reference can be made to the relevant statements of other embodiments. Figure 2A As shown, the method includes:

[0061] S201, obtaining an original live video, and extracting a plurality of frames of to-be-processed pictures from the original live video according to a frame extraction frequency and a live broadcast duration of the original live video.

[0062] S202: Perform target detection on the image to be processed to obtain at least one target image corresponding to the image to be processed.

[0063] S203: Determine the target image type of each target image according to the candidate image vectors in the preset image vector library.

[0064] S204: Generate a target mask corresponding to the target image according to the target image type of the target image.

[0065] S205 . Classify the target masks corresponding to the target images according to the mask types of the target masks to obtain a foreground mask set and a foreign body mask set.

[0066] The target mask may be a foreground mask or a foreign body mask. Specifically, the target mask of the foreground mask type is divided into a data set to obtain a foreground mask set; the target mask of the foreign body mask type is divided into a data set to obtain a foreign body mask set.

[0067] S206. Encode the original live video, foreground mask set and foreign object mask set respectively through the coding layer to obtain the video potential representation corresponding to the original live video, the foreground mask potential representation corresponding to the foreground mask set, and the foreign object mask potential representation corresponding to the foreign object mask set.

[0068] Among them, the video potential representation refers to the representation obtained after the original live video is encoded. The foreground mask potential representation refers to the representation obtained after the foreground mask set is encoded. The foreign object mask potential representation refers to the representation obtained after the foreign object mask set is encoded. It should be noted that the video potential representation, the foreground mask potential representation and the foreign object mask potential representation usually exist in the form of a matrix.

[0069] Specifically, the original live video, foreground mask set and foreign object mask set can be encoded respectively through the variational auto-encoder (VAE) in the encoding layer of the trained video enhancement model to obtain the video latent representation corresponding to the original live video, the foreground mask latent representation corresponding to the foreground mask set, and the foreign object mask latent representation corresponding to the foreign object mask set.

[0070] S207. Process the video latent representation, the foreground mask latent representation, and the foreign object mask latent representation through at least one core processing layer to obtain a target video fusion representation.

[0071] The core processing layer includes upsampling sublayer, downsampling sublayer, spatial attention sublayer and temporal attention sublayer; the upsampling sublayer adopts a dilated convolution structure (see Figure 2B ). It should be noted that Figure 2BThe gray squares in the image represent convolution kernels, and the white squares represent spaces. The upsampling sublayer is used to enhance the quality of foreground objects in the image. Its working principle is to subdivide each pixel in the foreground area of ​​the image into a 5*5 pixel sub-image, and make the pixel value of each pixel sub-image equal to the pixel value before subdivision. The downsampling sublayer is used to weaken the quality of foreign objects in the image; optionally, the downsampling sublayer uses a 5*5 average pooling layer. The spatial attention sublayer is used to focus on the key areas in the image, and to enhance the useful spatial position feature representation by calculating the attention weight of each spatial position, while suppressing the invalid spatial position feature representation. The temporal attention sublayer is used to enhance the coherence of the picture during the live video enhancement process. The target video fusion representation refers to the representation obtained after the video potential representation, the foreground mask potential representation, and the foreign object mask potential representation are processed by the core processing layer.

[0072] Specifically, if the trained video enhancement model includes only one core processing layer, the video latent representation and the foreground mask latent representation are input into the upsampling sublayer of the core processing layer, and the first video representation is obtained after processing by the upsampling sublayer; at the same time, the video latent representation and the foreign body mask latent representation are input into the downsampling sublayer of the core processing layer, and the second video representation is obtained after processing by the downsampling sublayer; the first video representation and the second video representation are fused to obtain an initial video fusion representation; the initial video fusion representation is input into the spatial attention sublayer of the core processing layer, and the intermediate video fusion representation is obtained after processing by the spatial attention sublayer; the Gaussian noise latent representation is obtained; the intermediate video fusion representation and the Gaussian noise latent representation are input into the temporal attention sublayer of the core processing layer, and the target video fusion representation is obtained after processing by the temporal attention sublayer.

[0073] Optionally, if the trained video enhancement model includes multiple core processing layers, for example, the trained video enhancement model includes N core processing layers; wherein N is greater than 1 and N is a positive integer; optionally, N can be pre-set according to actual business needs, and the embodiment of the present invention does not specifically limit it. Specifically, the N core processing layers gradually perform video enhancement on the original live video in an autoregressive manner, that is, the video potential representation, the foreground mask potential representation and the foreign body mask potential representation are input into the first core processing layer, and the first target video fusion representation is obtained after being processed by the first core processing layer; the first target video fusion representation, the foreground mask potential representation and the foreign body mask potential representation are input into the second core processing layer, and the second target video fusion representation is obtained after being processed by the second core processing layer; the second target video fusion representation, the foreground mask potential representation and the foreign body mask potential representation are input into the third core processing layer, and the third target video fusion representation is obtained after being processed by the third core processing layer; and so on, the Nth target video fusion representation, that is, the final target video fusion representation, can be obtained. It should be noted that the potential representations of the Gaussian noise added to each core processing layer may be the same or different.

[0074] It should be noted that the upsampling sublayer in the core processing layer is trained by self-supervised training so that the upsampling sublayer in the core processing layer can better enhance the quality of the foreground object in the image. Specifically, an unlabeled sample image is obtained; the unlabeled sample image is pooled through a 5*5 average pooling layer to obtain a blurred image; a preset image sampling rule (see Figure 2C ) samples the blurred image to obtain a sampling result; multiply the sampling result by the weight matrix of the upsampling sublayer to obtain a clear image; based on the MSE (Mean Squared Error) loss function, determine the training loss according to the clear image and the unlabeled sample image; if the training loss is less than the training loss threshold, stop training the upsampling sublayer, and use the upsampling sublayer at the time of stopping training as the upsampling sublayer in the core processing layer; otherwise, adjust the gradient and weight matrix of the upsampling sublayer, and continue to train the upsampling sublayer. The training loss threshold can be pre-set according to actual business needs, and the embodiment of the present invention does not specifically limit it.

[0075] It should be noted that Figure 2C Each square in the represents a pixel point. For each blurred image, the pixel value of each pixel point in the blurred image is equal to the average pixel value obtained by adding the pixel value of the pixel point to the pixel values ​​of surrounding pixels.

[0076] S208. Decode the target video fusion representation through a decoding layer to obtain a new live video.

[0077] Specifically, the target video fusion representation can be decoded through the VAE in the decoding layer of the trained video enhancement model to obtain a new live video.

[0078] The technical solution of the embodiment of the present invention is to obtain the original live video, and extract multiple frames of to-be-processed pictures from the original live video according to the frame extraction frequency and the live broadcast duration of the original live video; perform target detection on the to-be-processed pictures to obtain at least one target picture corresponding to the to-be-processed pictures; determine the target picture type of each target picture according to the candidate picture vectors in the preset picture vector library; generate a target mask corresponding to the target picture according to the target picture type of the target picture; classify the target masks corresponding to each target picture according to the mask type of the target mask to obtain a foreground mask set and a foreign body mask set; respectively encode the original live video, the foreground mask set and the foreign body mask set through the encoding layer to obtain the video potential representation corresponding to the original live video, the foreground mask potential representation corresponding to the foreground mask set, and the foreign body mask potential representation corresponding to the foreign body mask set; process the video potential representation, the foreground mask potential representation and the foreign body mask potential representation through at least one core processing layer to obtain a target video fusion representation; decode the target video fusion representation through the decoding layer to obtain a new live video. The above technical scheme reduces the amount of data processing in the live video enhancement process by extracting frames from the original live video, thereby reducing the computational burden in the live video enhancement process; at the same time, according to the criticality of the picture content in the actual live scene, the obtained target picture is divided into a foreground picture, a foreign object picture and a background picture, so that the subsequent trained video enhancement model can perform targeted enhancement on the foreground picture in the original live video, that is, the subsequent trained video enhancement model can accurately enhance the key picture content in the original live video and weaken the foreign object picture in a targeted manner, thereby avoiding unnecessary enhancement processing on the picture content of the entire live video, reducing the cost of the live video enhancement process, and improving the clarity and visual effect of the key content in the live video, thereby improving the overall visual effect of the live video and improving the overall viewing experience of the audience.

[0079] Embodiment 3

[0080] Figure 3 This is a schematic diagram of the structure of a live video enhancement device provided in Embodiment 3 of the present invention. This embodiment is applicable to the case of accurately enhancing key picture content in a live video. The device can be implemented in the form of hardware and / or software and can be configured in an electronic device. Figure 3 As shown, the device comprises:

[0081] The to-be-processed picture extraction module 301 is used to obtain the original live video and extract multiple frames of to-be-processed pictures from the original live video according to the frame extraction frequency and the live broadcast duration of the original live video;

[0082] A target image determination module 302 is used to perform target detection on the image to be processed to obtain at least one target image corresponding to the image to be processed;

[0083] A picture type determination module 303, used to determine a target picture type of each target picture according to candidate picture vectors in a preset picture vector library;

[0084] A target mask generation module 304, used to generate a target mask corresponding to the target image according to the target image type of the target image;

[0085] The new live video determination module 305 is used to use the trained video enhancement model to perform video enhancement processing on the original live video and the target masks corresponding to each target image to obtain a new live video.

[0086] The technical solution of the embodiment of the present invention is to obtain the original live video, and extract multiple frames of to-be-processed pictures from the original live video according to the frame extraction frequency and the live broadcast duration of the original live video; perform target detection on the pictures to be processed to obtain at least one target picture corresponding to the pictures to be processed; determine the target picture type of each target picture according to the candidate picture vectors in a preset picture vector library; generate a target mask corresponding to the target picture according to the target picture type of the target picture; use a trained video enhancement model to perform video enhancement processing on the original live video and the target masks corresponding to each target picture to obtain a new live video. The above technical scheme reduces the amount of data processing in the live video enhancement process by performing frame extraction processing on the original live video, thereby reducing the computational burden in the live video enhancement process; obtains the object sub-image (i.e., target image) corresponding to each image to be processed by performing target detection on each image to be processed; then, determines the image type of each object sub-image according to the candidate image vector in the preset image vector library, so as to distinguish the criticality of all object sub-images according to the actual live scene requirements, so that the trained video enhancement model can accurately enhance the key picture content in the original live video, thereby avoiding unnecessary enhancement processing of the picture content of the entire live video, reducing the cost of the live video enhancement process, and improving the clarity and visual effect of the key content in the live video, thereby improving the overall visual effect of the live video and improving the overall viewing experience of the audience.

[0087] Optionally, the picture type determination module 303 includes:

[0088] A target graph vector determination unit, configured to perform graph embedding processing on each target image to obtain a target graph vector corresponding to the target image;

[0089] A matching graph vector screening unit, used to screen out a matching graph vector that matches the target graph vector from candidate graph vectors in a preset graph vector library according to the target graph vector;

[0090] The picture type determination unit is used to determine the target picture type of the target picture according to the vector type of the matching picture vector.

[0091] Optionally, a matching graph vector screening unit is used to:

[0092] Perform similarity matching between the target image vector and each candidate image vector in the preset image vector library to obtain a similarity matching result;

[0093] According to the similarity matching result and the similarity threshold, a matching graph vector that matches the target graph vector is screened out from the candidate graph vectors in the preset graph vector library.

[0094] Optionally, the trained video enhancement model includes a coding layer, a decoding layer and at least one core processing layer; the coding layer is connected to the at least one core processing layer; the decoding layer is connected to the at least one core processing layer;

[0095] The new live video determination module 305 is specifically used for:

[0096] According to the mask type of the target mask, the target mask corresponding to each target image is classified to obtain a foreground mask set and a foreign body mask set;

[0097] The original live video, the foreground mask set and the foreign body mask set are respectively encoded through the encoding layer to obtain a video potential representation corresponding to the original live video, a foreground mask potential representation corresponding to the foreground mask set, and a foreign body mask potential representation corresponding to the foreign body mask set;

[0098] Processing the video latent representation, the foreground mask latent representation, and the foreign body mask latent representation through at least one core processing layer to obtain a target video fusion representation;

[0099] The target video fusion representation is decoded through the decoding layer to obtain a new live video.

[0100] Optionally, the core processing layer includes an upsampling sublayer, a downsampling sublayer, a spatial attention sublayer, and a temporal attention sublayer; the upsampling sublayer adopts a dilated convolution structure.

[0101] The live video enhancement device provided in the embodiment of the present invention can execute the live video enhancement method provided in any embodiment of the present invention, and has the corresponding functional modules and beneficial effects for executing each live video enhancement method.

[0102] According to an embodiment of the present invention, the present invention also provides an electronic device, a readable storage medium and a computer program product.

[0103] Embodiment 4

[0104] Figure 4 A schematic diagram of the structure of an electronic device 10 that can be used to implement an embodiment of the present invention is shown. The electronic device is intended to represent various forms of digital computers, such as laptop computers, desktop computers, workstations, personal digital assistants, servers, blade servers, mainframe computers, and other suitable computers. The electronic device can also represent various forms of mobile devices, such as personal digital processing, cellular phones, smart phones, wearable devices (such as helmets, glasses, watches, etc.) and other similar computing devices. The components shown herein, their connections and relationships, and their functions are merely examples and are not intended to limit the implementation of the present invention described and / or required herein.

[0105] like Figure 4 As shown, the electronic device 10 includes at least one processor 11, and a memory connected to the at least one processor 11, such as a read-only memory (ROM) 12, a random access memory (RAM) 13, etc., wherein the memory stores a computer program that can be executed by at least one processor, and the processor 11 can perform various appropriate actions and processes according to the computer program stored in the read-only memory (ROM) 12 or the computer program loaded from the storage unit 18 to the random access memory (RAM) 13. In the RAM 13, various programs and data required for the operation of the electronic device 10 can also be stored. The processor 11, ROM 12 and RAM 13 are connected to each other through a bus 14. An input / output (I / O) interface 15 is also connected to the bus 14.

[0106] A number of components in the electronic device 10 are connected to the I / O interface 15, including: an input unit 16, such as a keyboard, a mouse, etc.; an output unit 17, such as various types of displays, speakers, etc.; a storage unit 18, such as a disk, an optical disk, etc.; and a communication unit 19, such as a network card, a modem, a wireless communication transceiver, etc. The communication unit 19 allows the electronic device 10 to exchange information / data with other devices through a computer network such as the Internet and / or various telecommunication networks.

[0107] The processor 11 may be a variety of general and / or special processing components with processing and computing capabilities. Some examples of the processor 11 include, but are not limited to, a central processing unit (CPU), a graphics processing unit (GPU), various dedicated artificial intelligence (AI) computing chips, various processors running machine learning model algorithms, a digital signal processor (DSP), and any appropriate processor, controller, microcontroller, etc. The processor 11 performs the various methods and processes described above, such as a live video enhancement method.

[0108] In some embodiments, the live video enhancement method may be implemented as a computer program, which is tangibly contained in a computer-readable storage medium, such as a storage unit 18. In some embodiments, part or all of the computer program may be loaded and / or installed on the electronic device 10 via the ROM 12 and / or the communication unit 19. When the computer program is loaded into the RAM 13 and executed by the processor 11, one or more steps of the live video enhancement method described above may be performed. Alternatively, in other embodiments, the processor 11 may be configured to perform the live video enhancement method in any other appropriate manner (e.g., by means of firmware).

[0109] Various implementations of the systems and techniques described above herein can be implemented in digital electronic circuit systems, integrated circuit systems, field programmable gate arrays (FPGAs), application specific integrated circuits (ASICs), application specific standard products (ASSPs), systems on chips (SOCs), load programmable logic devices (CPLDs), computer hardware, firmware, software, and / or combinations thereof. These various implementations can include: being implemented in one or more computer programs that can be executed and / or interpreted on a programmable system including at least one programmable processor, which can be a special purpose or general purpose programmable processor that can receive data and instructions from a storage system, at least one input device, and at least one output device, and transmit data and instructions to the storage system, the at least one input device, and the at least one output device.

[0110] Computer programs for implementing the methods of the present invention may be written in any combination of one or more programming languages. These computer programs may be provided to a processor of a general-purpose computer, a special-purpose computer, or other programmable data processing device, so that when the computer program is executed by the processor, the functions / operations specified in the flow chart and / or block diagram are implemented. The computer program may be executed entirely on the machine, partially on the machine, partially on the machine and partially on a remote machine as a stand-alone software package, or entirely on a remote machine or server.

[0111] In the context of the present invention, a computer-readable storage medium may be a tangible medium that may contain or store a computer program for use by or in combination with an instruction execution system, device or equipment. A computer-readable storage medium may include, but is not limited to, electronic, magnetic, optical, electromagnetic, infrared, or semiconductor systems, devices or equipment, or any suitable combination of the foregoing. Alternatively, a computer-readable storage medium may be a machine-readable signal medium. A more specific example of a machine-readable storage medium may include an electrical connection based on one or more lines, a portable computer disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the foregoing.

[0112] To provide interaction with a user, the systems and techniques described herein may be implemented on an electronic device having: a display device (e.g., a CRT (cathode ray tube) or LCD (liquid crystal display) monitor) for displaying information to the user; and a keyboard and a pointing device (e.g., a mouse or trackball) through which the user can provide input to the electronic device. Other types of devices may also be used to provide interaction with the user; for example, the feedback provided to the user may be any form of sensory feedback (e.g., visual feedback, auditory feedback, or tactile feedback); and input from the user may be received in any form (including acoustic input, voice input, or tactile input).

[0113] The systems and techniques described herein may be implemented in a computing system that includes backend components (e.g., as a data server), or a computing system that includes middleware components (e.g., an application server), or a computing system that includes frontend components (e.g., a user computer with a graphical user interface or a web browser through which a user can interact with implementations of the systems and techniques described herein), or a computing system that includes any combination of such backend components, middleware components, or frontend components. The components of the system may be interconnected by any form or medium of digital data communication (e.g., a communication network). Examples of communication networks include: a local area network (LAN), a wide area network (WAN), a blockchain network, and the Internet.

[0114] A computing system may include a client and a server. The client and the server are generally remote from each other and usually interact through a communication network. The client and server relationship is generated by computer programs running on the corresponding computers and having a client-server relationship with each other. The server may be a cloud server, also known as a cloud computing server or cloud host, which is a host product in the cloud computing service system to solve the defects of difficult management and weak business scalability in traditional physical hosts and VPS services.

[0115] It should be understood that the various forms of processes shown above can be used to reorder, add or delete steps. For example, the steps described in the present invention can be executed in parallel, sequentially or in different orders, as long as the desired results of the technical solution of the present invention can be achieved, and this document does not limit this.

[0116] The above specific implementations do not constitute a limitation on the protection scope of the present invention. It should be understood by those skilled in the art that various modifications, combinations, sub-combinations and substitutions can be made according to design requirements and other factors. Any modification, equivalent substitution and improvement made within the spirit and principle of the present invention should be included in the protection scope of the present invention.

Claims

1. A live video enhancement method, characterized in that: include: Acquire an original live video, and extract multiple frames of to-be-processed pictures from the original live video according to a frame extraction frequency and a live broadcast duration of the original live video; Performing target detection on the image to be processed to obtain at least one target image corresponding to the image to be processed; Determine the target image type of each target image according to the candidate image vectors in the preset image vector library; Generate a target mask corresponding to the target image according to the target image type of the target image; The trained video enhancement model is used to perform video enhancement processing on the original live video and the target masks corresponding to each target image to obtain a new live video.

2. The method according to claim 1, characterized in that The step of determining the target image type of each target image according to the candidate image vectors in the preset image vector library includes: For each target image, perform graph embedding processing on the target image to obtain a target graph vector corresponding to the target image; According to the target graph vector, a matching graph vector matching the target graph vector is selected from candidate graph vectors in a preset graph vector library; A target image type of the target image is determined according to the vector type of the matching image vector.

3. The method according to claim 2, characterized in that The step of selecting a matching graph vector that matches the target graph vector from candidate graph vectors in a preset graph vector library according to the target graph vector includes: Performing similarity matching between the target image vector and each candidate image vector in a preset image vector library to obtain a similarity matching result; According to the similarity matching result and the similarity threshold, a matching graph vector that matches the target graph vector is screened out from the candidate graph vectors in the preset graph vector library.

4. The method according to claim 1, characterized in that: The trained video enhancement model includes a coding layer, a decoding layer and at least one core processing layer; the coding layer is connected to the at least one core processing layer; the decoding layer is connected to the at least one core processing layer; The trained video enhancement model is used to perform video enhancement processing on the original live video and the target masks corresponding to each target image to obtain a new live video, including: According to the mask type of the target mask, the target mask corresponding to each target image is classified to obtain a foreground mask set and a foreign body mask set; The original live video, the foreground mask set and the foreign object mask set are respectively encoded by the encoding layer to obtain a video potential representation corresponding to the original live video, a foreground mask potential representation corresponding to the foreground mask set, and a foreign object mask potential representation corresponding to the foreign object mask set; Processing the video potential representation, the foreground mask potential representation and the foreign object mask potential representation through the at least one core processing layer to obtain a target video fusion representation; The target video fusion representation is decoded through the decoding layer to obtain a new live video.

5. The method according to claim 4, characterized in that The core processing layer includes an upsampling sublayer, a downsampling sublayer, a spatial attention sublayer and a temporal attention sublayer; the upsampling sublayer adopts a dilated convolution structure.

6. A live video enhancement device, characterized in that: include: The to-be-processed picture extraction module is used to obtain the original live video and extract multiple frames of to-be-processed pictures from the original live video according to the frame extraction frequency and the live broadcast duration of the original live video; A target image determination module, configured to perform target detection on the image to be processed to obtain at least one target image corresponding to the image to be processed; A picture type determination module, used to determine a target picture type of each target picture according to candidate picture vectors in a preset picture vector library; A target mask generation module, used to generate a target mask corresponding to the target image according to the target image type of the target image; The new live video determination module is used to use the trained video enhancement model to perform video enhancement processing on the original live video and the target masks corresponding to each target image to obtain a new live video.

7. The device according to claim 6, characterized in that The picture type determination module includes: A target graph vector determination unit, configured to perform graph embedding processing on each target image to obtain a target graph vector corresponding to the target image; A matching graph vector screening unit, configured to screen out a matching graph vector that matches the target graph vector from candidate graph vectors in a preset graph vector library according to the target graph vector; The picture type determination unit is used to determine the target picture type of the target picture according to the vector type of the matching picture vector.

8. An electronic device, characterized in that: The electronic device comprises: at least one processor; and a memory communicatively connected to the at least one processor; wherein, The memory stores a computer program that can be executed by the at least one processor, and the computer program is executed by the at least one processor so that the at least one processor can execute the live video enhancement method described in any one of claims 1-5.

9. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores computer instructions, and the computer instructions are used to enable a processor to implement the live video enhancement method described in any one of claims 1-5 when executed.

10. A computer program product, comprising a computer program, wherein when the computer program is executed by a processor, the live video enhancement method according to any one of claims 1 to 5 is implemented.

Images