Video cover replacement method and device

Abstract

The embodiment of the invention provides a video cover replacement method. The video cover replacement method comprises the following steps: acquiring an original cover of a video; determining whether the original cover meets a preset condition or not, wherein the preset condition is used for determining the visual quality of the original cover; and under the condition that the original cover meets the preset condition, generating a new cover of the video by adopting generative artificial intelligence, and replacing the original cover with the new cover. According to the technical scheme provided by the embodiment of the invention, the generation mode of the new cover can be reasonably determined, so that the new cover brings a relatively large visual increment, the visual quality of the new cover is higher than that of the original cover, and the putting effect of the video advertisement is effectively improved.

Description

Technical Field

[0001] The embodiments of the present application relate to the field of multimedia technology, and in particular, to a video cover replacement method, apparatus, computer equipment, computer-readable storage medium, and computer program product. Background Art

[0002] Video cover replacement is usually one of the ways in advertising systems to quickly enrich advertising materials and improve advertising creative supply.

[0003] Currently, video cover replacements are either simply modifying the basic elements of the video cover (such as background color, text color), or extracting frames from the video. However, it is difficult to bring visual increments by modifying the basic elements of the video cover to generate a new cover. And it is difficult to ensure that the visual quality of the new cover generated by extracting frames is better than the original cover. These methods of video cover replacement are difficult to effectively improve the delivery effect of video advertising.

[0004] It should be noted that the above content is not necessarily prior art, nor is it intended to limit the scope of patent protection of this application. Summary of the invention

[0005] The embodiments of the present application provide a video cover replacement method, apparatus, computer device, computer-readable storage medium, and computer program product to solve or alleviate one or more of the technical problems raised above.

[0006] An aspect of an embodiment of the present application provides a video cover replacement method, the method comprising:

[0007] Get the original cover of the video;

[0008] Determining whether the original cover meets a preset condition, wherein the preset condition is used to determine the visual quality of the original cover;

[0009] When the original cover meets the preset conditions, generative artificial intelligence is used to generate a new cover for the video, and the new cover replaces the original cover.

[0010] Optionally, determining whether the original cover meets a preset condition includes:

[0011] Obtaining target features of the original cover, wherein the target features include at least some features of image features, text features, a saliency map, an aesthetic score, a clarity score, and an image description;

[0012] The target feature is input into a pre-trained binary classification model, and the binary classification model is used to determine whether the original cover meets a preset condition based on the target feature.

[0013] Optionally, the adopting generative artificial intelligence to generate a new cover for the video includes:

[0014] Obtaining the original text of the original cover, and determining whether the original text meets the first condition;

[0015] When the original text does not meet the first condition, a large language model is used to generate a first text;

[0016] Generate a new cover for the video using generative artificial intelligence based on the first text and the original cover.

[0017] Optionally, the step of using a large language model to generate a first copy includes:

[0018] Acquire first information of the video, the first information including text information, description information and at least part of the title information of the video, the text information is text information of a video frame in the video, the text information is obtained according to voice conversion of the video, and the description information is generated according to a video detailed description method;

[0019] The first text is generated using a large language model according to the first information.

[0020] Optionally, the step of generating a new cover for the video using generative artificial intelligence according to the first text and the original cover includes:

[0021] Determine whether the original cover has a target subject, wherein the target subject includes a person and / or a target product;

[0022] In the case that the target subject does not exist on the original cover, using generative artificial intelligence to generate a first picture similar to the original cover;

[0023] Generate a new cover for the video according to the first text and the first picture.

[0024] Optionally, the method further comprises:

[0025] In a case where the target subject exists in the original cover, determining a first background in the original cover;

[0026] Generate a second background according to the generative artificial intelligence, and replace the first background with the second background to obtain a second picture;

[0027] Generate a new cover for the video based on the second picture and the first text.

[0028] Optionally, the method further comprises:

[0029] When the original text meets the first condition, obtaining second information of the original text, wherein the second information includes at least part of the information of the position, font and content of the original text;

[0030] The original text is removed, and a new cover for the video is generated using generative artificial intelligence based on the second information and the original cover.

[0031] Optionally, the step of generating a new cover for the video using generative artificial intelligence according to the second information and the original cover includes:

[0032] Predicting a target position for generating the Chinese copy for the new cover;

[0033] Generate a new cover for the video using generative artificial intelligence based on the target location, the second information and the original cover.

[0034] Another aspect of an embodiment of the present application provides a video cover replacement device, the device comprising:

[0035] The acquisition module is used to obtain the original cover of the video;

[0036] A determination module, used to determine whether the original cover meets a preset condition, wherein the preset condition is used to determine the visual quality of the original cover;

[0037] A replacement module is used to generate a new cover for the video using generative artificial intelligence when the original cover meets the preset conditions, and replace the original cover with the new cover.

[0038] Another aspect of an embodiment of the present application provides a computer device, comprising: at least one processor; and a memory communicatively connected to the at least one processor; wherein: the memory stores instructions executable by the at least one processor, and the instructions are executed by the at least one processor to enable the at least one processor to execute the method described above.

[0039] Another aspect of an embodiment of the present application provides a computer-readable storage medium, wherein the computer-readable storage medium stores computer instructions, and when the computer instructions are executed by a processor, the method described above is implemented.

[0040] Another aspect of an embodiment of the present application provides a computer program product, including a computer program, which implements the method described above when executed by a processor.

[0041] The above technical solution adopted in the embodiment of the present application may have the following advantages:

[0042] By obtaining the original cover of the video, determining the visual quality of the original cover according to preset conditions, and using generative artificial intelligence to generate a new cover for the video when the original cover meets the preset conditions, the generation method of the new cover can be reasonably determined based on the visual quality of the original cover; at the same time, using generative artificial intelligence to generate a new cover and replacing the original cover with the new cover of the video can make the new cover bring a larger visual increment compared to the original cover, and can also make the visual quality of the new cover higher than the original cover, thereby effectively improving the delivery effect of video advertising. BRIEF DESCRIPTION OF THE DRAWINGS

[0043] The accompanying drawings exemplarily illustrate the embodiments and constitute a part of the specification, and together with the text description of the specification, are used to explain the exemplary implementation of the embodiments. The embodiments shown are for illustrative purposes only and do not limit the scope of the claims. In all drawings, the same reference numerals refer to similar but not necessarily identical elements.

[0044] Figure 1 The following schematically shows the operating environment of the video cover replacement method according to the first embodiment of the present application;

[0045] Figure 2 The flowchart of the video cover replacement method according to the first embodiment of the present application is schematically shown;

[0046] Figure 3 Schematically shows Figure 2 Flow chart of sub-steps of step S202;

[0047] Figure 4 The binary classification model in the video cover replacement method according to the first embodiment of the present application is schematically shown;

[0048] Figure 5 Schematically shows Figure 2 Flow chart of sub-steps of step S204;

[0049] Figure 6 Schematically shows Figure 5 Flow chart of sub-steps of step S402;

[0050] Figure 7 The schematic diagram shows the principle diagram of generating the first text in the video cover replacement method according to the first embodiment of the present application;

[0051] Figure 8 Schematically shows Figure 5 Flow chart of sub-steps of step S404;

[0052] Fig. 9 The newly added process of the video cover replacement method according to the first embodiment of the present application is schematically shown;

[0053] Fig.10 Another newly added process of the video cover replacement method according to the first embodiment of the present application is schematically shown;

[0054] Fig.11 Schematically shows Fig.10 Flow chart of sub-steps of step S802;

[0055] Fig.12 The schematic diagram shows a principle diagram of a video cover replacement method according to the first embodiment of the present application;

[0056] Fig.13 A partial application flow chart of the video cover replacement method according to the first embodiment of the present application is schematically shown;

[0057] Fig.14 A block diagram schematically shows a video cover replacement device according to Embodiment 2 of the present application; and

[0058] Fig.15 The hardware architecture diagram of the computer device according to the third embodiment of the present application is schematically shown. DETAILED DESCRIPTION

[0059] In order to make the purpose, technical solutions and advantages of the present application more clearly understood, the present application is further described in detail below in conjunction with the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are only used to explain the present application and are not intended to limit the present application. Based on the embodiments in the present application, all other embodiments obtained by ordinary technicians in the field without making creative work are within the scope of protection of the present application.

[0060] It should be noted that the descriptions involving "first", "second", etc. in the embodiments of the present application are only for descriptive purposes and cannot be understood as indicating or implying their relative importance or implicitly indicating the number of technical features indicated. Therefore, the features defined as "first" and "second" may explicitly or implicitly include at least one of the features. In addition, the technical solutions between the various embodiments can be combined with each other, but they must be based on the ability of ordinary technicians in the field to implement them. When the combination of technical solutions is contradictory or cannot be implemented, it should be deemed that such combination of technical solutions does not exist and is not within the scope of protection required by this application.

[0061] In the description of the present application, it should be understood that the numerical labels before the steps do not indicate the order in which the steps are executed, but are only used to facilitate the description of the present application and to distinguish each step, and therefore should not be understood as a limitation on the present application.

[0062] First, the following terms are explained:

[0063] Optical Character Recognition (OCR): is a technology that is specifically designed to automatically recognize and extract text from unstructured documents such as images, screenshots, and physical paper documents. OCR technology determines the shape of text by detecting dark and light patterns in an image, and then uses character recognition methods to translate these shapes into computer-readable text.

[0064] Automatic Speech Recognition (ASR) is a technology that converts human speech into text. It can process real-time audio streams or recorded audio files and convert them into readable text form.

[0065] Image Caption (image annotation or image description): refers to the automatic generation of a descriptive text from a given image. This process requires not only that the computer can detect objects in the image, but also that it understand the relationship between objects and convert this information into human-readable text descriptions in combination with natural language processing technology. Image Caption is an intersection task of computer vision and natural language processing, and its goal is to enable computers to "understand" visual images and automatically generate accurate text descriptions for them.

[0066] Video Dense Caption: It is to detect and describe multiple overlapping events from long uncut videos, and generate multiple captions or descriptions related to the time positions of these events. This is a complex task because it requires the model to not only accurately identify objects, scenes, and actions in the video, but also understand the relationship between these elements and convert these understandings into coherent and natural text descriptions.

[0067] Image Inpainting: refers to the technology of restoring or filling the missing, damaged or occluded parts of an image by utilizing the existing information in the image.

[0068] Content-aware layout prediction: refers to a technology or method that can automatically arrange the layout intelligently according to the content (such as text, images, etc.). This technology aims to improve the efficiency and quality of design. By analyzing and understanding the characteristics of the content, it automatically adjusts the arrangement, size, spacing, etc. of elements to create a layout that is both beautiful and meets the requirements of the content.

[0069] Generative AI (Artificial Intelligence Generated Content, AIGC for short): is a technology that uses artificial intelligence to automatically generate content such as text, images, audio and video. This technology uses machine learning and deep learning algorithms to enable computer models to understand and generate human language, thereby automatically generating valuable content.

[0070] Secondly, in order to facilitate those skilled in the art to understand the technical solutions provided in the embodiments of the present application, the relevant technologies are described below:

[0071] In the related art, replacing a video cover is either simply modifying the basic elements of the video cover, or extracting frames from the video to generate a new cover; however, it is difficult to bring visual increments by simply modifying the basic elements of the video cover; and it is difficult to ensure that the visual quality of the new cover generated by extracting frames is better than the original cover, resulting in these video cover replacement methods being difficult to improve the effectiveness of video advertising.

[0072] To this end, the embodiment of the present application provides a technical solution for replacing a video cover. In this technical solution, the new cover can bring a larger visual increment, and the visual quality of the new cover can be higher than the original cover, thereby effectively improving the delivery effect of the video advertisement. See below for details.

[0073] Finally, for ease of understanding, an exemplary operating environment is provided below.

[0074] like Figure 1 As shown, the environment diagram includes a service platform 2, a network 4, and a client 6, wherein:

[0075] The service platform 2 may be composed of a single or multiple computing devices. The multiple computing devices may include virtualized computing instances. Virtualized computing instances may include virtual machines, such as simulations of computer systems, operating systems, servers, etc. The computing device may load a virtual machine based on a virtual image and / or other data defining specific software (e.g., operating system, dedicated application, server) for simulation. As the demand for different types of processing services changes, different virtual machines may be loaded and / or terminated on one or more computing devices. A hypervisor may be implemented to manage the use of different virtual machines on the same computing device.

[0076] The service platform 2 may be configured to communicate with the client 6 or the like via a network 4. The network 4 includes various network devices, such as routers, switches, multiplexers, hubs, modems, bridges, repeaters, firewalls, proxy devices, and / or the like. The network 4 may include physical links, such as coaxial cable links, twisted pair cable links, optical fiber links, combinations thereof, and the like, or wireless links, such as cellular links, satellite links, Wi-Fi links, and the like.

[0077] The service platform 2 can provide storage, reading, writing, querying, deleting and other services, such as providing video viewing services for clients.

[0078] Client 6 can be a Windows, Android TM ) or iOS operating systems, such as smart phones, tablet devices, laptop computers, virtual reality devices, gaming devices, set-top boxes, car terminals, and smart TVs. Based on the above operating systems, various applications can be run, such as an application for watching videos.

[0079] The client 6 may provide / configure a user access page for manipulating the service platform 2 or uploading an object, etc.

[0080] It should be noted that the above devices are exemplary, and the number and type of devices are adjustable in different scenarios or according to different needs.

[0081] The following describes the technical solution of the present application through multiple embodiments, taking the service platform as the execution subject. It should be noted that these embodiments can be implemented in a variety of different forms and should not be interpreted as being limited to the embodiments described here. In addition, the execution subject of the video cover replacement method in the embodiments of the present application can also be a single client.

[0082] Embodiment 1

[0083] Figure 2 The flowchart of the video cover replacement method according to the first embodiment of the present application is schematically shown.

[0084] like Figure 2 As shown, the video cover replacement method may include steps S200 to S204, wherein:

[0085] Step S200, obtaining the original cover of the video.

[0086] Step S202, determining whether the original cover meets a preset condition, where the preset condition is used to determine the visual quality of the original cover.

[0087] Step S204, when the original cover meets the preset conditions, generative artificial intelligence is used to generate a new cover for the video, and the new cover replaces the original cover.

[0088] The video cover replacement method provided in this embodiment obtains the original cover of the video, determines the visual quality of the original cover according to preset conditions, and uses generative artificial intelligence to generate a new cover for the video when the original cover meets the preset conditions. The generation method of the new cover can be reasonably determined according to the visual quality of the original cover. At the same time, generative artificial intelligence is used to generate a new cover, and the original cover is replaced by the new cover of the video. The new cover can bring a larger visual increment compared to the original cover, and the visual quality of the new cover can be higher than that of the original cover, thereby effectively improving the delivery effect of video advertising.

[0089] The following combination Figure 2 , each step in steps S200 to S204 and other optional steps are described in detail.

[0090] Step S200 , get the original cover of the video.

[0091] The video may be an advertisement video or a video that will be used for advertising.

[0092] Step S202 , determine whether the original cover meets preset conditions, and the preset conditions are used to determine the visual quality of the original cover.

[0093] The service platform may pre-train a model for judging the visual quality of the original cover, so as to determine that the original cover meets the preset conditions when the visual quality of the original cover is good, and otherwise does not meet the preset conditions.

[0094] In an optional embodiment, the determination in step S202 whether the original cover meets a preset condition, such as Figure 3 As shown, it may include steps S300 to S302:

[0095] Step S300, obtaining target features of the original cover, wherein the target features include image features, text features, saliency map, aesthetic score, clarity score and at least some features of the image description.

[0096] Step S302, input the target features into a pre-trained binary classification model, and determine whether the original cover meets the preset conditions based on the target features through the binary classification model.

[0097] Image features can be obtained through traditional image processing methods, such as extracting the color histogram of the original cover to obtain color features; they can also be extracted through artificial intelligence models, such as using convolutional neural networks to extract image features of the original cover.

[0098] The copy features may include the normalized position of the copy, and may also include other copy features, such as typesetting features, emotional tendency features, etc. The copy features may be extracted using OCR. For example, OCR may be used to extract the boundary coordinates of the copy, and the normalized position of the copy may be calculated based on the normalization standard. Optionally, the copy features may also first use OCR to detect and extract the copy of the original cover, and then obtain the corresponding copy features based on the copy. For example, after obtaining the copy, natural language processing is used to analyze the emotion of the copy to obtain the emotional tendency features of the copy.

[0099] The saliency map can be obtained by subject detection on the original cover (such as detection using a subject detection model). After the subject is identified, the boundary coordinates of the subject are obtained, and then the saliency map of the original cover is generated according to the boundary coordinates of the subject.

[0100] The aesthetic score can be a score of the color, composition, creativity, prominence of the theme, visual impact and emotional communication of the original cover. When performing the aesthetic score, a tool can be used to score, for example, a color analysis tool can be used to score the color saturation, contrast, color balance, etc. of the original cover. Optionally, an artificial intelligence model (such as a deep learning model) can also be used to comprehensively score the aesthetics of the original cover.

[0101] The clarity rating may include, but is not limited to, rating the image resolution, detail retention, noise control, contrast, etc. of the original cover. When performing the clarity rating, a script may be written based on a clarity evaluation algorithm (such as the Laplacian gradient function), and then the script may be used to rate the clarity of the original cover. In addition, an interface or evaluation tool capable of rating clarity may also be used to rate the clarity of the original cover.

[0102] The image description can be obtained by using Image Caption, that is, using Image Caption to automatically generate descriptive text for the original cover to obtain the image description of the original cover.

[0103] The binary classification model can be a transformer, which can determine whether the original cover meets the preset conditions based on image features, text features, saliency maps, aesthetic scores, clarity scores, and several features in the image description. In addition, in addition to using transformers, the binary classification model can also be implemented using a multimodal fusion model or a hybrid model.

[0104] Please refer to Figure 4, which is a specific example diagram of a binary classification model. As shown in the figure, image feature extraction can be performed on the original cover to obtain image features; OCR detection can be performed on the original cover to determine the normalized position of the text on the image; the original cover can be subject detected to generate a saliency map; the original cover can be aesthetically scored to obtain an aesthetic score; the original cover can be clarity detected to obtain a clarity score; Image Caption is used to obtain an image description based on the original cover; the image features, the normalized position of the text on the image, the saliency map, the aesthetic score, the clarity score and the image description are input into the transformer to determine whether the original cover meets the preset conditions, so as to determine whether to use generative artificial intelligence to generate a new cover to replace the original cover. For example, if the original cover has a lot of text and a large proportion, the saliency map does not show an obvious subject, the aesthetic score is low, and the clarity score is low, it can be determined that the original cover does not meet the preset conditions. For another example, if the number of words and the proportion of the text on the original cover are reasonable, the saliency map can show an obvious subject, the aesthetic score is high, and the clarity score is high, it can be determined that the original cover meets the preset conditions. It should be understood that the above examples are only illustrative. In fact, it is determined whether the preset conditions are met based on a comprehensive judgment of various characteristics. Therefore, it is not necessary for each characteristic to meet specific conditions.

[0105] In this optional embodiment, by obtaining target features of the original cover, the target features include image features, text features, saliency maps, aesthetic scores, clarity scores and at least some features in the image description, the target features are input into a pre-trained binary classification model, and the binary classification model is used to determine whether the original cover meets preset conditions based on the target features. The visual quality of the original cover of the video can be effectively evaluated, and the accuracy of the visual quality evaluation of the original cover can be improved, thereby facilitating accurate and reasonable determination of the generation method of the new video cover.

[0106] Step S204 ,When the original cover meets the preset conditions, generative artificial intelligence is used to generate a new cover for the video, and the new cover replaces the original cover.

[0107] Optionally, when the original cover does not meet the preset conditions, a new cover for the video can be generated based on a frame extraction image of the video, and then the new cover replaces the original cover.

[0108] In an optional embodiment, the new cover of the video is generated by using generative artificial intelligence in step S204, such as Figure 5 As shown, steps S400 to S404 may be included:

[0109] Step S400, obtaining the original text of the original cover, and determining whether the original text meets the first condition.

[0110] Step S402: when the original text does not meet the first condition, a large language model is used to generate a first text.

[0111] Step S404, generating a new cover for the video using generative artificial intelligence based on the first text and the original cover.

[0112] The service platform can use OCR to detect the original cover to obtain the original copy. The first condition can include restrictions such as the number of words in the image is within a certain range, the proportion of the words in the image is within a certain range, etc. The specific range can be configured according to the actual situation, and no specific restrictions are made here. For example, if the number of words corresponding to the first condition is 5-20, and the number of words in the original copy is 0, it can be determined that the original copy does not meet the first condition.

[0113] When a large language model is used to generate the first copy, the first copy may be generated based on the original cover, or may be generated based on information such as the title of the video.

[0114] In step S404, since the original cover meets the preset conditions, the visual quality of the original cover is good. When generative artificial intelligence is used to generate a new cover for the video, generative artificial intelligence can be used to generate a new background for the original cover to replace the original background, and then the main content of the original cover and the first text are combined to perform graphic synthesis to obtain a new cover for the video.

[0115] In this optional embodiment, the original text of the original cover is obtained, and it is determined whether the original text meets the first condition. If the original text does not meet the first condition, a large language model is used to generate a first text. A new cover of the video is generated based on the first text and the original cover using generative artificial intelligence. If the original text is not good enough, a large language model can be used to generate a new text to improve the quality of the text corresponding to the new cover of the video.

[0116] In an optional embodiment, in step S402, a large language model is used to generate the first copy of the original cover, such as Figure 6 As shown, steps S500 to S502 may be included:

[0117] Step S500, obtaining first information of the video, the first information includes text information, description information and at least part of the title information of the video, the text information is the text information of the video frame in the video, the text information is obtained according to the voice conversion of the video, and the description information is generated according to the video detailed description method.

[0118] Step S502: Generate a first copy using a large language model according to the first information.

[0119] Text information can be obtained by acquiring text in the video frame through OCR; text information can be obtained by transcribing the speech of the video through ASR; and description information can be generated based on the video using a video detailed description method.

[0120] When the first text is generated by using the large language model according to the first information, the generated first text may be one or more.

[0121] Please refer to Figure 7 , which is a schematic diagram of the principle of generating the first copy. As shown in the figure, optical character recognition (OCR) detection can be performed on the video to obtain the text information in the video frame; automatic speech recognition (ASR) can be performed on the voice in the video to obtain the text information of the video; the video is described in detail using the video to obtain the description information; and the text information, text information, description information and video title information are input into the large language model text generation model to generate several on-picture copy (i.e., the first copy).

[0122] In this optional embodiment, by obtaining first information of the video, the first information includes text information, description information and at least part of the title information of the video, and using a large language model to generate a first copy based on the first information, a copy corresponding to a new cover can be generated based on information related to the video itself and the text, thereby improving the accuracy of the newly generated copy.

[0123] In an optional embodiment, in step S404, a new cover of the video is generated using generative artificial intelligence based on the first copy and the original cover, such as Figure 8 As shown, steps S600 to S604 may be included:

[0124] Step S600, determining whether the original cover has a target subject, where the target subject includes a person and / or a target product.

[0125] Step S602, when the target subject does not exist in the original cover, a first picture similar to the original cover is generated using generative artificial intelligence.

[0126] Step S604, generating a new cover for the video according to the first text and the first picture.

[0127] Specifically, the service platform can detect the target subject on the original cover to determine whether the target subject exists on the original cover. If the target subject does not exist on the original cover, the service platform can use generative artificial intelligence to generate a first picture similar to the original cover, and then perform image-text synthesis based on the first text and the first picture to obtain a new cover for the video. When using generative artificial intelligence to generate a first picture similar to the original cover, Image Caption can be used to generate descriptive text for the original cover, and then the first picture is generated based on the descriptive text.

[0128] In this optional embodiment, by determining whether the original cover has a target subject, if the original cover does not have a target subject, generative artificial intelligence is used to generate a first image similar to the original cover, and a new cover for the video is generated according to the first copy and the first image. If the original cover does not have a target subject, generative artificial intelligence can be used to generate an image corresponding to the new cover, so that the generated new cover brings more visual increments, thereby improving the delivery effect of video advertising.

[0129] In an optional embodiment, if Fig. 9 As shown, the video cover replacement method may further include steps S700 to S704:

[0130] Step S700: when the target subject exists in the original cover, determine the first background in the original cover.

[0131] Step S702, generating a second background according to generative artificial intelligence, and replacing the first background with the second background to obtain a second picture.

[0132] Step S704, generating a new cover for the video based on the second picture and the first text.

[0133] Specifically, when the service platform determines that the target subject exists in the original cover, it determines the first background excluding the target subject in the original cover, uses generative artificial intelligence to generate a second background that can replace the first background, and replaces the first background with the second background to obtain a second picture that combines the target subject and the second background; then the second picture is combined with the first text to obtain a new cover for the video.

[0134] In this optional embodiment, by determining the first background in the original cover when the target subject exists in the original cover, generating a second background according to generative artificial intelligence, replacing the first background with the second background to obtain a second picture, and generating a new cover for the video according to the second picture and the first text, when the target subject exists in the original cover, generative artificial intelligence can be used to replace the background of the original cover while retaining the target subject, thereby allowing the new cover to bring a larger visual increment and improve the effectiveness of video advertising.

[0135] In an optional embodiment, if Fig.10 As shown, the video cover replacement method may further include steps S800 to S802:

[0136] Step S800: When the original text meets the first condition, second information of the original text is obtained, where the second information includes at least part of the information of the position, font and content of the original text.

[0137] Step S802, remove the original text, and use generative artificial intelligence to generate a new cover for the video based on the second information and the original cover.

[0138] Through the above-mentioned embodiments, it can be known that when the image corresponding to the new cover of the video is generated by generative artificial intelligence based on the original cover, a completely new image can be generated, or an image with a changed background can be generated; in this case, it may not be appropriate to keep the original text in the original position, and it is not convenient to flexibly adjust the position of the text. Therefore, in this embodiment, the original text is first removed, and then the new cover of the video is generated by generative artificial intelligence based on the second information and the original cover. Among them, Image Inpainting can be used to erase the original text.

[0139] When the second information includes the position of the original copy, the position of the copy corresponding to the new cover may preferentially adopt the position of the original copy, or the position obtained by fine-tuning the position of the original copy; when the second information includes the font of the original copy, the font of the copy corresponding to the new cover may preferentially adopt the font of the original copy or a font similar to the font; when the second information includes the content of the original copy, the content of the copy corresponding to the new cover may preferentially adopt the content of the original copy, or the content obtained by fine-tuning the content of the original copy. Optionally, the second information may necessarily include the content of the original copy, that is, the second information includes at least part of the information in the position and font of the original copy in addition to the content of the original copy.

[0140] In this optional embodiment, by obtaining second information of the original copy when the original copy meets the first condition, the second information includes at least part of the information of the position, font and content of the original copy, removing the original copy, and generating a new cover for the video based on the second information and the original cover using generative artificial intelligence. On the basis of removing the original copy, a new cover and text corresponding to the new cover can be generated based on relevant information of the original copy, thereby improving the generation quality of the new cover and facilitating necessary adjustments to the text corresponding to the new cover.

[0141] In an optional embodiment, in step S802, a new cover of the video is generated using generative artificial intelligence based on the second information and the original cover, such as Fig.11 As shown, steps S900 to S902 may be included:

[0142] Step S900, predicting the target position for generating the Chinese copy for the new cover.

[0143] Step S902, generating a new cover for the video using generative artificial intelligence according to the target position, the second information and the original cover.

[0144] The service platform can use content-aware layout prediction to predict the target location for generating the Chinese copy for the new cover.

[0145] When the second information does not include the position of the original copy, the target position is directly used as the position of the copy corresponding to the new cover; when the second information includes the position of the original copy, the position of the original copy can be corrected according to the target position to obtain the position of the copy corresponding to the new cover, or one of the target position or the original copy position can be randomly used.

[0146] In this optional embodiment, by predicting the target position for generating the text on the new cover, a new cover for the video is generated using generative artificial intelligence based on the target position, the second information and the original cover. The position of the text on the new cover can be determined by predicting the position of the text, thereby optimizing the generation quality of the new cover and improving the delivery effect of the video advertisement.

[0147] In order to make this application easier to understand, the following Fig.12 and Fig.13 The video cover replacement method is further explained.

[0148] like Fig.12 As shown, the original cover of the video is judged by a discriminant model to determine whether the original cover can be replaced generatively. If the result obtained by the discriminant model is negative, a new cover of the video is generated by methods such as video-based frame extraction; if the result obtained by the discriminant model is positive, a new cover of the video is generated by using vision-based and generative technology. Vision can refer to computer vision, specifically the aforementioned image feature extraction and OCR detection technologies.

[0149] Fig.13 The following is a flowchart of how to generate a new cover when it is determined that a generative replacement can be performed. As shown in the figure, the following contents may be included:

[0150] 1. Perform OCR detection on the original cover of the video to determine whether there is text on the original cover or whether the text ratio is reasonable. If there is no text on the picture or the text ratio is too small, use the following method: Figure 7 Generate the text corresponding to the new cover according to the process of generating text on the image shown in the figure; if there is text on the image and the proportion of the text is reasonable, record the font and content of the text, and use Image Inpainting to erase the text on the original cover;

[0151] 2. Perform subject detection on the original cover to determine whether there are people or target products. If there is no subject on the original cover, use generative artificial intelligence to generate a similar picture based on the image description obtained by Image Caption; if there are people or target products on the original cover, use generative artificial intelligence to generate a picture with the same subject but different background;

[0152] 3. Use content-aware layout prediction to predict the corresponding position of the new cover copy;

[0153] 4. Use the predicted position and the text obtained in step 1 to synthesize the image and text to get a new cover for the video.

[0154] Embodiment 2

[0155] Fig.14 The block diagram of the video cover replacement device according to the second embodiment of the present application is schematically shown. The device can be divided into one or more program modules, one or more program modules are stored in a storage medium, and are executed by one or more processors to complete the embodiment of the present application. The program module referred to in the embodiment of the present application refers to a series of computer program instruction segments that can complete specific functions. The following description will specifically introduce the functions of each program module in this embodiment. Fig.14 As shown, the apparatus 1000 may include: an acquisition module 1010, a determination module 1020 and a replacement module 1030, wherein:

[0156] The acquisition module 1010 is used to acquire the original cover of the video;

[0157] A determination module 1020, configured to determine whether the original cover meets a preset condition, wherein the preset condition is used to determine the visual quality of the original cover;

[0158] The replacement module 1030 is used to generate a new cover for the video using generative artificial intelligence when the original cover meets the preset conditions, and replace the original cover with the new cover.

[0159] In an optional embodiment, the determination module 1020 is further configured to:

[0160] Obtaining target features of the original cover, wherein the target features include at least some features of image features, text features, a saliency map, an aesthetic score, a clarity score, and an image description;

[0161] The target feature is input into a pre-trained binary classification model, and the binary classification model is used to determine whether the original cover meets a preset condition based on the target feature.

[0162] In an optional embodiment, the replacement module 1030 is further configured to:

[0163] Obtaining the original text of the original cover, and determining whether the original text meets the first condition;

[0164] When the original text does not meet the first condition, a large language model is used to generate a first text;

[0165] Generate a new cover for the video using generative artificial intelligence based on the first text and the original cover.

[0166] In an optional embodiment, the replacement module 1030 is further configured to:

[0167] Acquire first information of the video, the first information including text information, description information and at least part of the title information of the video, the text information is text information of a video frame in the video, the text information is obtained according to voice conversion of the video, and the description information is generated according to a video detailed description method;

[0168] The first text is generated using a large language model according to the first information.

[0169] In an optional embodiment, the replacement module 1030 is further configured to:

[0170] Determine whether the original cover has a target subject, wherein the target subject includes a person and / or a target product;

[0171] In the case that the target subject does not exist on the original cover, using generative artificial intelligence to generate a first picture similar to the original cover;

[0172] Generate a new cover for the video according to the first text and the first picture.

[0173] In an optional embodiment, the replacement module 1030 is further configured to:

[0174] In a case where the target subject exists in the original cover, determining a first background in the original cover;

[0175] Generate a second background according to the generative artificial intelligence, and replace the first background with the second background to obtain a second picture;

[0176] Generate a new cover for the video based on the second picture and the first text.

[0177] In an optional embodiment, the replacement module 1030 is further configured to:

[0178] When the original text meets the first condition, obtaining second information of the original text, wherein the second information includes at least part of the information of the position, font and content of the original text;

[0179] The original text is removed, and a new cover for the video is generated using generative artificial intelligence based on the second information and the original cover.

[0180] In an optional embodiment, the replacement module 1030 is further configured to:

[0181] Predicting a target position for generating the Chinese copy for the new cover;

[0182] Generate a new cover for the video using generative artificial intelligence based on the target location, the second information and the original cover.

[0183] Embodiment 3

[0184] Fig.15 The schematic diagram of the hardware architecture of a computer device 10000 suitable for implementing the video cover replacement method according to the third embodiment of the present application is shown schematically. In some embodiments, the computer device 10000 may be a terminal device such as a smart phone, a wearable device, a tablet computer, a personal computer, a vehicle terminal, a game console, a virtual device, a workbench, a digital assistant, a set-top box, a robot, etc. In other embodiments, the computer device 10000 may be a rack server, a blade server, a tower server, or a cabinet server (including an independent server, or a server cluster composed of multiple servers), etc. Fig.15 As shown, the computer device 10000 includes but is not limited to: a memory 10010, a processor 10020, and a network interface 10030 that can communicate with each other through a system bus. Among them:

[0185] The memory 10010 includes at least one type of computer-readable storage medium, and the readable storage medium includes a flash memory, a hard disk, a multimedia card, a card-type memory (such as an SD or DX memory), a random access memory (RAM), a static random access memory (SRAM), a read-only memory (ROM), an electrically erasable programmable read-only memory (EEPROM), a programmable read-only memory (PROM), a magnetic memory, a magnetic disk, an optical disk, etc. In some embodiments, the memory 10010 may be an internal storage module of the computer device 10000, such as a hard disk or a memory of the computer device 10000. In other embodiments, the memory 10010 may also be an external storage device of the computer device 10000, such as a plug-in hard disk equipped on the computer device 10000, a smart memory card (Smart Media Card, SMC), a secure digital (Secure Digital, SD) card, a flash card (Flash Card), etc. Of course, the memory 10010 may also include both the internal storage module of the computer device 10000 and its external storage device. In this embodiment, the memory 10010 is generally used to store the operating system and various application software installed in the computer device 10000, such as the program code of the video cover replacement method, etc. In addition, the memory 10010 can also be used to temporarily store various data that have been output or will be output.

[0186] In some embodiments, the processor 10020 may be a central processing unit (CPU), a controller, a microcontroller, a microprocessor, or other chips. The processor 10020 is generally used to control the overall operation of the computer device 10000, such as performing control and processing related to data interaction or communication with the computer device 10000. In this embodiment, the processor 10020 is used to run the program code stored in the memory 10010 or process data.

[0187] The network interface 10030 may include a wireless network interface or a wired network interface, and the network interface 10030 is generally used to establish a communication link between the computer device 10000 and other computer devices. For example, the network interface 10030 is used to connect the computer device 10000 to an external terminal through a network, and to establish a data transmission channel and a communication link between the computer device 10000 and the external terminal. The network may be a wireless or wired network such as an intranet, the Internet, the Global System of Mobile communication (GSM), Wideband Code Division Multiple Access (WCDMA), 4G network, 5G network, Bluetooth, Wi-Fi, etc.

[0188] It should be pointed out that Fig.15 Only a computer device having components 10010 - 10030 is shown, but it should be understood that implementation of all of the components shown is not a requirement, and more or fewer components may alternatively be implemented.

[0189] In this embodiment, the video cover replacement method stored in the memory 10010 can also be divided into one or more program modules and executed by one or more processors (such as processor 10020) to complete the embodiment of the present application.

[0190] Embodiment 4

[0191] An embodiment of the present application further provides a computer-readable storage medium having a computer program stored thereon, wherein when the computer program is executed by a processor, the steps of the video cover replacement method in the embodiment are implemented.

[0192] In this embodiment, the computer-readable storage medium includes flash memory, hard disk, multimedia card, card-type memory (for example, SD or DX memory, etc.), random access memory (RAM), static random access memory (SRAM), read-only memory (ROM), electrically erasable programmable read-only memory (EEP ROM), programmable read-only memory (PROM), magnetic memory, disk, optical disk, etc. In some embodiments, the computer-readable storage medium can be an internal storage unit of a computer device, such as a hard disk or memory of the computer device. In other embodiments, the computer-readable storage medium can also be an external storage device of a computer device, such as a plug-in hard disk equipped on the computer device, a smart memory card (Smart Media Card, SMC), a secure digital (Secure Digital, SD) card, a flash card (Flash Card), etc. Of course, the computer-readable storage medium can also include both the internal storage unit of the computer device and its external storage device. In this embodiment, the computer-readable storage medium is generally used to store an operating system and various application software installed on the computer device, such as the program code of the video cover replacement method in the embodiment. In addition, the computer-readable storage medium can also be used to temporarily store various types of data that have been output or are to be output.

[0193] Embodiment 5

[0194] An embodiment of the present application also provides a computer program product, including a computer program, which implements the method in the above embodiment when executed by a processor.

[0195] Obviously, those skilled in the art should understand that the modules or steps of the above-mentioned embodiments of the present application can be implemented by general-purpose computer devices, they can be concentrated on a single computer device, or distributed on a network composed of multiple computer devices, optionally, they can be implemented by executable program codes of computer devices, so that they can be stored in a storage device and executed by the computer device, and in some cases, the steps shown or described can be executed in a different order from that herein, or they can be made into individual integrated circuit modules, or multiple modules or steps therein can be made into a single integrated circuit module for implementation. In this way, the embodiments of the present application are not limited to any specific combination of hardware and software.

[0196] It should be noted that the above are only preferred embodiments of the present application, and the patent protection scope of the present application is not limited thereto. Any equivalent structure or equivalent process transformation made using the contents of the specification and drawings of the present application, or directly or indirectly applied in other related technical fields, are also included in the patent protection scope of the present application.

Claims

1. A video cover replacement method, characterized in that: The method comprises: Get the original cover of the video; Determining whether the original cover meets a preset condition, wherein the preset condition is used to determine the visual quality of the original cover; When the original cover meets the preset conditions, generative artificial intelligence is used to generate a new cover for the video, and the new cover replaces the original cover.

2. The method according to claim 1, characterized in that The determining whether the original cover meets a preset condition comprises: Obtaining target features of the original cover, wherein the target features include at least some features of image features, text features, a saliency map, an aesthetic score, a clarity score, and an image description; The target feature is input into a pre-trained binary classification model, and the binary classification model is used to determine whether the original cover meets a preset condition based on the target feature.

3. The method according to claim 2, characterized in that The method of using generative artificial intelligence to generate a new cover for the video includes: Obtaining the original text of the original cover, and determining whether the original text meets the first condition; When the original text does not meet the first condition, a large language model is used to generate a first text; Generate a new cover for the video using generative artificial intelligence based on the first text and the original cover.

4. The method according to claim 3, characterized in that The step of using a large language model to generate a first copy includes: Acquire first information of the video, the first information including text information, description information and at least part of the title information of the video, the text information is text information of a video frame in the video, the text information is obtained according to voice conversion of the video, and the description information is generated according to a video detailed description method; The first text is generated using a large language model according to the first information.

5. The method according to claim 3, characterized in that: The step of generating a new cover for the video using generative artificial intelligence according to the first text and the original cover includes: Determine whether the original cover has a target subject, wherein the target subject includes a person and / or a target product; In the case that the target subject does not exist on the original cover, using generative artificial intelligence to generate a first picture similar to the original cover; Generate a new cover for the video according to the first text and the first picture.

6. The method according to claim 5, characterized in that The method further comprises: In a case where the target subject exists in the original cover, determining a first background in the original cover; Generate a second background according to the generative artificial intelligence, and replace the first background with the second background to obtain a second picture; Generate a new cover for the video based on the second picture and the first text.

7. The method according to any one of claims 3 to 6, characterized in that: The method further comprises: When the original text meets the first condition, obtaining second information of the original text, wherein the second information includes at least part of the information of the position, font and content of the original text; The original text is removed, and a new cover for the video is generated using generative artificial intelligence based on the second information and the original cover.

8. The method according to claim 7, characterized in that The step of generating a new cover for the video using generative artificial intelligence according to the second information and the original cover includes: Predicting a target position for generating the Chinese copy for the new cover; Generate a new cover for the video using generative artificial intelligence based on the target location, the second information and the original cover.

9. A video cover replacement device, characterized in that: The device comprises: The acquisition module is used to obtain the original cover of the video; A determination module, used to determine whether the original cover meets a preset condition, wherein the preset condition is used to determine the visual quality of the original cover; A replacement module is used to generate a new cover for the video using generative artificial intelligence when the original cover meets the preset conditions, and replace the original cover with the new cover.

10. A computer device, characterized in that: include: at least one processor; and a memory communicatively connected to the at least one processor; wherein: The memory stores instructions that can be executed by the at least one processor, and the instructions are executed by the at least one processor to enable the at least one processor to perform the method according to any one of claims 1 to 8.

11. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores computer instructions, and when the computer instructions are executed by a processor, the method according to any one of claims 1 to 8 is implemented.

12. A computer program product, comprising a computer program, characterized in that When the computer program is executed by a processor, the steps of the method according to any one of claims 1 to 8 are implemented.

Images