Dynamic cover generation method and device, electronic equipment and program product

Abstract

The embodiment of the invention provides a dynamic cover generation method and device, electronic equipment and a computer program product, and relates to the technical field of computers. The method comprises the following steps: acquiring an initial static image corresponding to a target media resource; determining the image category of the initial static image; and performing dynamic video generation processing on the initial static image according to the image category to obtain a dynamic video cover corresponding to the target media resource. According to the method, the corresponding dynamic video cover can be generated by using the single static image, the user experience can be effectively improved, the platform attraction is enhanced, and the propagation channel of media resources is further expanded.

Description

Background Art

[0002] This section is intended to provide a background or context to the embodiments of the disclosure that are recited in the claims. No description herein is admitted to be prior art by inclusion in this section.

[0003] With the rapid development of the Internet, more and more people are beginning to obtain media resources through the Internet, such as videos, music, news, etc. For example, music service platforms provide a wide range of music content for users to play, download or collect online, including various types of songs, albums, artists, radio stations and playlists, etc., to meet the music needs of different users.

[0004] In the process of user experience of music, visual presentation and perception also play an important role. In the visual information of songs, the song cover plays a vital role. It is not only the first impression of the song, but also a supplement and explanation of the deep emotions and themes that the author wants to convey. Through a well-designed cover, the audience can understand the connotation of the song more intuitively and have a deeper emotional resonance with the author. Summary of the invention

[0005] However, the static cover used in the related solutions does not fully consider the emotion and theme of the song, resulting in the generated cover not corresponding to the overall style of the song and the low accuracy of the generated cover. In the solution of using traditional video production methods to create dynamic cover, generating dynamic video has extremely high requirements on image data, computing resources and production costs.

[0006] To this end, the present invention proposes an improved dynamic cover generation method to generate a corresponding dynamic video cover using a single static image of a media resource. The dynamic video cover can convey the emotions and themes of the media resource through the dynamic changes of visual elements, which can effectively improve the user experience, enhance the attractiveness of the platform, and further expand the dissemination channels of media resources.

[0007] In this context, the embodiments of the present disclosure are intended to provide a dynamic cover generation method, a dynamic cover generation device, a computer-readable storage medium and electronic device, and a computer program product.

[0008] In the first aspect of the embodiments of the present disclosure, a dynamic cover generation method is provided, including: obtaining an initial static image corresponding to a target media resource; determining an image category of the initial static image; performing dynamic video generation processing on the initial static image according to the image category to obtain a dynamic video cover corresponding to the target media resource.

[0009] In one embodiment of the present disclosure, determining the image category of the initial static image includes: obtaining a preconfigured image classification model, the image classification model including one or more of a human body detection model, a face detection model, a text recognition model and an image detection model; using the image classification model to perform image classification processing on the initial static image to obtain the image category.

[0010] In one embodiment of the present disclosure, the image classification model is used to perform image classification processing on the initial static image to obtain the image category, including: using the human body detection model or the face detection model to perform image classification detection on the initial static image, and determining a portrait image according to the obtained classification detection result; using the text recognition model to perform image classification detection on the initial static image, and determining a text image according to the obtained classification detection result; using the image detection model to perform image classification detection on the initial static image, and determining a regular image according to the obtained classification detection result, and the regular image includes one or more of a general category image, a border category image, and an invalid category image.

[0011] In one embodiment of the present disclosure, the image category includes portrait images, and the dynamic video cover includes a first portrait dynamic cover and a second portrait dynamic cover; the dynamic video generation processing of the initial static image according to the image category to obtain the dynamic video cover corresponding to the target media resource includes: determining the area ratio of the portrait area in the portrait image; dividing the portrait image into a first portrait image and a second portrait image according to the area ratio, the first portrait image being an image whose area ratio is greater than a preset ratio threshold; performing dynamic video generation processing on the first portrait image to obtain the first portrait dynamic cover; performing dynamic video generation processing on the second portrait image to obtain the second portrait dynamic cover.

[0012] In one embodiment of the present disclosure, the dynamic video generation processing is performed on the first portrait image to obtain the first portrait dynamic cover, including: determining the image color category of the first portrait image, the image color category including the full image color category and the edge color category; determining the dynamic special effects template of the first portrait image according to the full image color category and the edge color category; and generating the first portrait dynamic cover based on the dynamic special effects template.

[0013] In one embodiment of the present disclosure, determining the image color category of the first portrait image includes: determining the first channel pixels of each color channel in the first portrait image; determining the first histogram vector of each color channel based on multiple first channel pixels; concatenating the first histogram vectors of the multiple color channels to obtain a full-image histogram vector; and determining the full-image color category based on the full-image histogram vector.

[0014] In one embodiment of the present disclosure, determining the image color category of the first portrait image further includes: obtaining an edge area image of the first portrait image, and determining the second channel pixels of each color channel in the edge area image; determining a second histogram vector of each color channel based on a plurality of the second channel pixels; concatenating the second histogram vectors of the plurality of color channels to obtain an edge histogram vector; and determining the edge color category based on the edge histogram vector.

[0015] In one embodiment of the present disclosure, determining the dynamic special effects template of the first portrait image based on the full-image color category and the edge color category includes: obtaining a candidate special effects template corresponding to the candidate image cover, the candidate special effects template including a candidate full-image special effects template and a candidate border special effects template; determining a matching mapping relationship between the image color category and the candidate special effects template; determining a target image category corresponding to the first portrait image based on the full-image color category and the edge color category; determining the dynamic special effects template from the candidate special effects templates based on the matching mapping relationship and the target image category; the candidate full-image special effects template is obtained by clustering the candidate image full-image histogram vectors of a plurality of the candidate image covers, and the candidate border special effects template is obtained by clustering the candidate edge area histogram vectors of a plurality of the candidate image covers.

[0016] In one embodiment of the present disclosure, the dynamic video generation processing of the second portrait image to obtain the second portrait dynamic cover includes: determining the portrait area of ​​the second portrait image, masking the portrait area to obtain a masked portrait image, wherein the masked portrait image includes a masked portrait area and an area outside the mask; determining the portrait area ratio of the masked portrait area in the masked portrait image, and determining dynamic video generation parameters according to the portrait area ratio; based on the dynamic video generation parameters, performing dynamic video generation processing on the image of the area outside the mask to obtain the second portrait dynamic cover.

[0017] In one embodiment of the present disclosure, the image category includes text images, and the dynamic video cover includes a text dynamic cover; the dynamic video generation processing is performed on the initial static image according to the image category to obtain the dynamic video cover corresponding to the target media resource, including: obtaining a pre-configured text video generation model, the text video generation model is trained based on text video samples; using the text video generation model, the dynamic video generation processing is performed on the text image to obtain the text dynamic cover.

[0018] In one embodiment of the present disclosure, the image category includes a border category image, and the dynamic video cover includes a border image dynamic cover; the dynamic video generation processing of the initial static image according to the image category to obtain the dynamic video cover corresponding to the target media resource includes: determining the border point coordinates of the border category image, and determining the border area based on the border point coordinates; determining the image within the border area based on the border area, performing dynamic video generation processing on the image within the border area to obtain the initial dynamic image; filling the initial dynamic image into the border area to obtain the border image dynamic cover.

[0019] In one embodiment of the present disclosure, the border point coordinates include first border point coordinates and second border point coordinates, and determining the border point coordinates of the border category image includes: performing horizontal pixel accumulation processing and vertical pixel accumulation processing on the border category image respectively to obtain a horizontal pixel accumulation vector and a vertical pixel accumulation vector; calculating the horizontal vector gradient of the horizontal pixel accumulation vector from left to right; taking the pixel points whose horizontal gradient values ​​in the horizontal vector gradient are greater than the gradient threshold as the first border point coordinates; calculating the vertical vector gradient of the vertical pixel accumulation vector from right to left; taking the pixel points whose vertical gradient values ​​in the vertical vector gradient are greater than the gradient threshold as the second border point coordinates.

[0020] In the second aspect of the embodiments of the present disclosure, a dynamic cover generation device is provided, including: an initial image acquisition module, used to acquire an initial static image corresponding to a target media resource; an image category determination module, used to determine the image category of the initial static image; and a dynamic cover generation module, used to perform dynamic video generation processing on the initial static image according to the image category to obtain a dynamic video cover corresponding to the target media resource.

[0021] In one embodiment of the present disclosure, the image category determination module includes an image category determination unit, which is used to: obtain a pre-configured image classification model, the image classification model includes one or more of a human body detection model, a face detection model, a text recognition model and an image detection model; use the image classification model to perform image classification processing on the initial static image to obtain the image category.

[0022] In one embodiment of the present disclosure, the image category determination unit includes an image category determination subunit, which is used to: use the human body detection model or the face detection model to perform image classification detection on the initial static image, and determine a portrait image based on the obtained classification detection result; use the text recognition model to perform image classification detection on the initial static image, and determine a text image based on the obtained classification detection result; use the image detection model to perform image classification detection on the initial static image, and determine a regular image based on the obtained classification detection result, and the regular image includes one or more of a general category image, a border category image, and an invalid category image.

[0023] In one embodiment of the present disclosure, the image category includes portrait images, and the dynamic video cover includes a first portrait dynamic cover and a second portrait dynamic cover; the dynamic cover generation module includes a portrait cover generation unit, which is used to: determine the area ratio of the portrait area in the portrait image; divide the portrait image into a first portrait image and a second portrait image according to the area ratio, and the first portrait image is an image whose area ratio is greater than a preset ratio threshold; perform dynamic video generation processing on the first portrait image to obtain the first portrait dynamic cover; perform dynamic video generation processing on the second portrait image to obtain the second portrait dynamic cover.

[0024] In one embodiment of the present disclosure, the portrait cover generation unit includes a first portrait cover generation sub-unit, which is used to: determine the image color category of the first portrait image, the image color category including the full image color category and the edge color category; determine the dynamic special effects template of the first portrait image based on the full image color category and the edge color category; and generate the first portrait dynamic cover based on the dynamic special effects template.

[0025] In one embodiment of the present disclosure, the first portrait cover generation subunit includes a first color category generation subunit, which is used to: determine the first channel pixels of each color channel in the first portrait image; determine the first histogram vector of each color channel based on multiple first channel pixels; splice the first histogram vectors of multiple color channels to obtain a full-image histogram vector; and determine the full-image color category based on the full-image histogram vector.

[0026] In one embodiment of the present disclosure, the first portrait cover generation subunit also includes a second color category generation subunit, which is used to: obtain an edge area image of the first portrait image, and determine the second channel pixels of each color channel in the edge area image; determine the second histogram vector of each color channel based on multiple second channel pixels; splice the second histogram vectors of multiple color channels to obtain an edge histogram vector; and determine the edge color category based on the edge histogram vector.

[0027] In one embodiment of the present disclosure, the first portrait cover generation subunit includes a special effects generation subunit, which is used to: obtain candidate special effects templates corresponding to the candidate image cover, the candidate special effects templates including candidate full-image special effects templates and candidate border special effects templates; determine the matching mapping relationship between the image color category and the candidate special effects template; determine the target image category corresponding to the first portrait image according to the full-image color category and the edge color category; determine the dynamic special effects template from the candidate special effects templates according to the matching mapping relationship and the target image category; the candidate full-image special effects template is obtained by clustering the candidate image full-image histogram vectors of multiple candidate image covers, and the candidate border special effects template is obtained by clustering the candidate edge area histogram vectors of multiple candidate image covers.

[0028] In one embodiment of the present disclosure, the portrait cover generation unit includes a second portrait cover generation sub-unit, which is used to: determine the portrait area of ​​the second portrait image, perform mask processing on the portrait area, and obtain a masked portrait image, wherein the masked portrait image includes a masked portrait area and an area outside the mask; determine the portrait area ratio of the masked portrait area in the masked portrait image, and determine dynamic video generation parameters based on the portrait area ratio; based on the dynamic video generation parameters, perform dynamic video generation processing on the image of the area outside the mask to obtain the second portrait dynamic cover.

[0029] In one embodiment of the present disclosure, the image category includes text images, and the dynamic video cover includes a text dynamic cover; the dynamic cover generation module includes a text cover generation unit, which is used to: obtain a pre-configured text video generation model, and the text video generation model is trained based on text video samples; use the text video generation model to perform dynamic video generation processing on the text image to obtain the text dynamic cover.

[0030] In one embodiment of the present disclosure, the image category includes a border category image, and the dynamic video cover includes a border image dynamic cover; the dynamic cover generation module includes a border cover generation unit, which is used to: determine the border point coordinates of the border category image, and determine the border area based on the border point coordinates; determine the image within the border area based on the border area, perform dynamic video generation processing on the image within the border area to obtain an initial dynamic image; fill the initial dynamic image into the border area to obtain the border image dynamic cover.

[0031] In one embodiment of the present disclosure, the border point coordinates include first border point coordinates and second border point coordinates, and the border cover generation unit includes a border coordinate determination subunit, which is used to: perform horizontal pixel accumulation processing and vertical pixel accumulation processing on the border category image respectively to obtain a horizontal pixel accumulation vector and a vertical pixel accumulation vector; calculate the horizontal vector gradient of the horizontal pixel accumulation vector in order from left to right; use the pixel points whose horizontal gradient values ​​in the horizontal vector gradient are greater than the gradient threshold as the first border point coordinates; calculate the vertical vector gradient of the vertical pixel accumulation vector in order from right to left; use the pixel points whose vertical gradient values ​​in the vertical vector gradient are greater than the gradient threshold as the second border point coordinates.

[0032] In a third aspect of the embodiments of the present disclosure, a computer-readable storage medium is provided, on which a computer program is stored. When the computer program is executed by a processor, the dynamic cover generation method as described in any one of the above is implemented.

[0033] In a fourth aspect of the embodiments of the present disclosure, an electronic device is provided, comprising: a processor; and a memory, wherein the memory stores computer-readable instructions, and when the computer-readable instructions are executed by the processor, the dynamic cover generation method as described in any one of the above is implemented.

[0034] According to a fifth aspect of an embodiment of the present disclosure, a computer program product is provided, comprising a computer program, which, when executed by a processor, implements any one of the above-described methods for generating a dynamic cover.

[0035] According to the technical solution of the disclosed implementation mode, on the one hand, the dynamic video cover generated by a single static image of a media resource can, through the dynamic changes of visual elements, give each media resource a unique personalized label, convey the emotion and theme of the media resource, better attract the user's attention, and effectively improve the user experience. On the other hand, the generated dynamic video cover of the media resource provides the media resource service platform with richer and more diverse visual materials, enhancing the platform's attractiveness. On another hand, the dynamic video cover of the media resource can be shared and disseminated on various platforms, further expanding the dissemination channels of the media resource. BRIEF DESCRIPTION OF THE DRAWINGS

[0036] The above and other objects, features and advantages of the exemplary embodiments of the present disclosure will become readily understood by reading the detailed description below with reference to the accompanying drawings. In the accompanying drawings, several embodiments of the present disclosure are shown in an exemplary and non-limiting manner, in which:

[0037] Figure 1 A block diagram schematically illustrates a system architecture of an exemplary application scenario according to some embodiments of the present disclosure;

[0038] Figure 2 A flowchart of a dynamic cover generation method according to some embodiments of the present disclosure is schematically shown;

[0039] Figure 3 Schematically shows a flow chart of classifying cover images according to some embodiments of the present disclosure;

[0040] Figure 4 A schematic diagram schematically shows a portrait cover image according to some embodiments of the present disclosure;

[0041] Figure 5 A schematic diagram schematically shows a text-type cover image according to some embodiments of the present disclosure;

[0042] Figure 6 A process diagram schematically illustrates a process of generating a dynamic video cover of a text-type cover image according to some embodiments of the present disclosure;

[0043] Figure 7 A schematic diagram schematically shows a frame-type cover image according to some embodiments of the present disclosure;

[0044] Figure 8 A schematic diagram schematically shows pixel accumulation processing for a frame-type cover image according to some embodiments of the present disclosure;

[0045] Fig. 9A schematic block diagram schematically shows a dynamic cover generation device according to some embodiments of the present disclosure;

[0046] Fig.10 A schematic diagram schematically illustrates a storage medium according to an example embodiment of the present disclosure;

[0047] Fig.11 A block diagram of an electronic device according to an example embodiment of the invention is schematically shown.

[0048] In the drawings, the same or corresponding reference numerals represent the same or corresponding parts. DETAILED DESCRIPTION

[0049] The principles and spirit of the present disclosure will be described below with reference to several exemplary embodiments. It should be understood that these embodiments are provided only to enable those skilled in the art to better understand and implement the present disclosure, and are not intended to limit the scope of the present disclosure in any way. On the contrary, these embodiments are provided to make the present disclosure more thorough and complete, and to fully convey the scope of the present disclosure to those skilled in the art.

[0050] Those skilled in the art will appreciate that the embodiments of the present disclosure may be implemented as a system, device, apparatus, method or computer program product. Therefore, the present disclosure may be specifically implemented in the following forms, namely: complete hardware, complete software (including firmware, resident software, microcode, etc.), or a combination of hardware and software.

[0051] According to the embodiments of the present disclosure, a dynamic cover generation method, a dynamic cover generation device, a computer-readable storage medium and electronic device, and a computer program product are proposed.

[0052] In this article, it is important to understand that the terms involved, such as dynamic cover, are a technique for converting a static image into a dynamic video, which uses a series of related consecutive frames to create a dynamic visual effect.

[0053] In addition, any number of elements in the drawings is for illustration and not limitation, and any naming is only for distinction and does not have any limiting meaning.

[0054] The principle and spirit of the present disclosure are explained in detail below with reference to several representative embodiments of the present disclosure. SUMMARY OF THE INVENTION

[0056] For various media resources with different forms of expression, the resource theme can be expressed through the cover of the media resource. For example, for the visual information of a song, the song cover is not only the first impression of the song, but also a supplement and explanation of the deep emotions and theme that the author wants to convey.

[0057] In one technical solution, the number of times each note appears in the song is counted, and the background notes and pattern notes are determined based on these numbers, and then the song cover is generated based on these notes. However, this solution does not fully consider the emotional and thematic content of the song, which may result in the generated cover not being completely consistent with the overall artistic style of the song, which is not user-friendly.

[0058] In another technical solution, the lyrics of the currently playing music are obtained, specific high-frequency words are extracted, and the cover image is matched in the album of the mobile terminal to generate a personalized music cover. However, the selection of images cannot reflect the content of the song to a large extent, and although the static cover is important, it is difficult to fully express the dynamic emotions of the song.

[0059] When generating dynamic covers, traditional video production solutions can be used, which usually use a series of continuous image frames to create dynamic effects. When using this solution to create dynamic covers, each second of video contains dozens or even hundreds of frames of images, and each frame needs to be carefully processed and rendered, which means that when generating a video of several minutes, tens of thousands of image frames may need to be processed. Such a workload undoubtedly places extremely high demands on image data, computing resources, and production costs.

[0060] Based on the above content, the basic idea of ​​the present disclosure is to obtain the initial static image corresponding to the target media resource; determine the image category of the initial static image; perform dynamic video generation processing on the initial static image according to the image category to obtain the dynamic video cover corresponding to the target media resource. The dynamic video cover can convey the emotion and theme of the media resource through the dynamic changes of visual elements, which can effectively improve the user experience, enhance the platform's attractiveness, and further expand the dissemination channels of media resources.

[0061] After introducing the basic principles of the present disclosure, various non-limiting embodiments of the present disclosure are described in detail below.

[0062] Application Scenario Overview

[0063] First reference Figure 1 , Figure 1 A schematic block diagram of a system architecture showing an exemplary application scenario in which a dynamic cover generation method and apparatus according to an embodiment of the present disclosure can be applied.

[0064] like Figure 1As shown, the system architecture 100 may include one or more of terminal devices 101, 102, 103, a network 104, and a server 105. The network 104 is used to provide a medium for communication links between the terminal devices 101, 102, 103 and the server 105. The network 104 may include various connection types, such as wired, wireless communication links or optical fiber cables, etc. The terminal devices 101, 102, 103 may be various electronic devices with display screens, including but not limited to desktop computers, portable computers, smart phones, tablet computers, etc. It should be understood that Figure 1 The number of terminal devices, networks and servers in the embodiment is only for illustration. According to the implementation requirements, there may be any number of terminal devices, networks and servers. For example, the server 105 may be a server cluster composed of multiple servers.

[0065] The dynamic cover generation method provided in the embodiment of the present disclosure is generally executed by the server 105, and accordingly, the dynamic cover generation device is generally set in the server 105. However, it is easy for those skilled in the art to understand that the dynamic cover generation method provided in the embodiment of the present disclosure can also be executed by the terminal devices 101, 102, and 103, and accordingly, the dynamic cover generation device can also be set in the terminal devices 101, 102, and 103, which is not particularly limited in this exemplary embodiment. For example, in an exemplary embodiment, a staff member can upload the initial static image of the target media resource to the server 105 through the terminal devices 101, 102, and 103, and the server performs dynamic video generation processing on the initial static image through the dynamic cover generation method provided in the embodiment of the present disclosure to obtain a dynamic video cover, and transmits the dynamic video cover to the terminal devices 101, 102, 103, etc. so that the terminal devices 101, 102, and 103 display the dynamic video cover to the user.

[0066] It should be understood that Figure 1 The application scenario shown is only an example in which the embodiments of the present disclosure can be implemented. The scope of application of the embodiments of the present disclosure is not limited in any aspect by the application scenario.

[0067] Exemplary Methods

[0068] Combine the following Figure 1 For application scenarios, refer to Figure 2 To describe the dynamic cover generation method according to an exemplary embodiment of the present disclosure. It should be noted that the above application scenarios are only shown to facilitate understanding of the spirit and principles of the present disclosure, and the embodiments of the present disclosure are not limited in this regard. On the contrary, the embodiments of the present disclosure can be applied to any applicable scenario.

[0069] The present disclosure first provides a dynamic cover generation method. The execution subject of the method can be a terminal device or a server. The present disclosure does not make any special limitation on this. In this example embodiment, the method is described by taking the server executing the method as an example.

[0070] Reference Figure 2 As shown, the dynamic cover generation method may include the following steps S210 to S230:

[0071] Step S210: obtaining an initial static image corresponding to the target media resource.

[0072] In one embodiment of the present disclosure, the target media resource may be various media resources provided on the network resource service platform, for example, the target media resource may include but is not limited to music, songs, videos, news, etc. The initial static image may be a static image determined according to the theme and content of the target media resource.

[0073] For the target media resource, a static image that matches the style and theme expressed by the target media resource is selected as the initial static image. Taking the music media service platform as an example, the presentation of a song is often prepared by the song creator in advance with a static image as the song cover. This cover image is not only an artistic expression of the song, but also a symbol of its identity and characteristics, to show the unique connotation of the song. As a visual representation of the song, the cover image plays a vital role in conveying the theme, emotion and artist style of the song.

[0074] In order to bring users a more vivid and three-dimensional visual experience through the cover of the target media resource, the present disclosure proposes to convert the static cover of the target media resource into a dynamic cover to enhance the visual appeal and emotional transmission and improve the user experience.

[0075] Step S220: determining the image category of the initial static image.

[0076] In one embodiment of the present disclosure, the image category may be a specific category determined according to the element content included in the initial static image.

[0077] After obtaining the initial static image of the target media resource, the image category of the initial static image can be determined first, and then the corresponding dynamic video generation processing operation can be performed according to different image categories. Taking the target media resource as a song as an example, since the style types of a song cover vary greatly, and the cover is the author's display of the direct content of the song, the expression form of the cover is also quite different, both in terms of content and style, such as real-life photos, cartoons, abstract art, template design, text, color gradients, etc. After determining the image category of the initial static image, the dynamic video generation processing operation of the corresponding category can be called to execute.

[0078] Step S230, performing dynamic video generation processing on the initial static image according to the image category to obtain a dynamic video cover corresponding to the target media resource.

[0079] In one embodiment of the present disclosure, the dynamic video generation process may be a process of generating a corresponding dynamic video according to the image content of the initial static image. The dynamic video cover may be a dynamic cover obtained by performing the dynamic video generation process on the initial static image. The dynamic video cover may bring a more vivid and three-dimensional visual experience, and convey the theme and emotion of the target media resource through the dynamic changes of visual elements.

[0080] After determining the image category, the initial static image is processed for dynamic video generation according to the image category to generate a video with dynamic effects as the dynamic video cover corresponding to the target media resource. Due to the large differences in image categories, when generating static videos from a single static image, it is necessary to first analyze tens of millions of large-scale static cover images. Based on the analyzed image category characteristics, different strategies are used for different categories of images to ensure that cover images of different types and styles can achieve the required visual effects when generating dynamic videos.

[0081] In the dynamic cover generation method provided in this example implementation, on the one hand, the dynamic video cover generated by a single static image of a media resource can, through the dynamic changes of visual elements, give each media resource a unique personalized label, convey the emotion and theme of the media resource, better attract the user's attention, and effectively improve the user experience. On the other hand, the generated dynamic video cover of the media resource provides the media resource service platform with richer and more diverse visual materials, enhancing the platform's attractiveness. On another hand, the dynamic video cover of the media resource can be shared and disseminated on various platforms, further expanding the dissemination channels of the media resource.

[0082] Next, the above steps of this exemplary embodiment are described in more detail.

[0083] In practical applications, the initial static image of the target media resource is input into the system implemented based on the present disclosure through the image input module. Then, the deep learning module can perform deep learning processing on this static image to generate a video with dynamic effects as a dynamic video cover.

[0084] In one embodiment of the present disclosure, for step S220, determining the image category of the initial static image includes: obtaining a pre-configured image classification model, the image classification model includes one or more of a human body detection model, a face detection model, a text recognition model and an image detection model; using the image classification model to perform image classification processing on the initial static image to obtain the image category.

[0085] Among them, the image classification model can be a model used to classify the image features of the static cover image, and the image classification model can integrate multiple different types of models. The human body detection model can be used to detect people and identify the people contained in the image. The face detection model can be a detection model used to identify the faces contained in the image. The text recognition model can be a model used to extract text from an image. The image detection model can be a detection model used to identify the content elements contained in the image. For example, the image detection model can detect whether the image is an ordinary image, whether it contains special elements such as borders, whether it is an invalid image, etc. The image classification process can be a process of determining the specific category of the initial static image.

[0086] After obtaining the initial static image of the target media resource, the initial static image is first classified. Images with similar styles or themes can be grouped together. Based on the common characteristics of images in each category, the next part is to generate vertical category videos, which is conducive to improving the yield rate of video generation. For target media resources on the Internet, compared with conventionally shot images, they usually contain more artistic elements, and the difference between different types of cover images is large. Therefore, before the static image is processed for dynamic video generation, the core elements of the static cover image can be quickly and accurately analyzed and classified through image recognition and image understanding algorithms. This capability plays a very critical role in the subsequent category-based image dynamic video generation process.

[0087] For example, taking the target media resource as a song, the static cover image of the song is also different from the conventional photographed image, and usually contains more artistic elements. The static cover of the song is first categorized according to the characteristics of the song cover, and then the cover images of different categories are processed into videos respectively. According to the elements contained in the static cover of the song, the static cover image can be mainly divided into the following major categories: portrait cover, text cover, general category cover, bordered cover, and invalid image cover.

[0088] Due to the large differences in song styles, especially for the tens of millions or even hundreds of millions of song covers on music service platforms, it is difficult to use a unified algorithm strategy to complete the generation of dynamic song covers from static song covers of different styles and types. Based on this, for the special application scenario of song covers, when realizing the generation of dynamic video covers from a single static image, we first analyze and classify song cover images of tens of millions or even larger scales.

[0089] For example, the initial static image is input into a pre-configured image classification model. Since the image classification model integrates one or more image recognition detection models among the human body detection model, face detection model, text recognition model and image detection model, after the classification processing of the image classification model, the image classification model will output the image category of the initial static image accordingly. Then, based on the analyzed characteristics of different image categories, different strategies are used for processing images of different categories to ensure that cover images of different types and styles can achieve the best effect when generating dynamic videos.

[0090] In one embodiment of the present disclosure, an image classification model is used to perform image classification processing on an initial static image to obtain an image category, including: using a human body detection model or a face detection model to perform image classification detection on the initial static image, and determining a portrait image based on the obtained classification detection result; using a text recognition model to perform image classification detection on the initial static image, and determining a text image based on the obtained classification detection result; using an image detection model to perform image classification detection on the initial static image, and determining a regular image based on the obtained classification detection result, wherein the regular image includes one or more of a general category image, a border category image, and an invalid category image.

[0091] Among them, a portrait image may be an initial static image containing a portrait element. A text image may be an initial static image containing a text element. A regular image may be an initial static image containing regular category elements. For example, a regular image may be an image that does not contain special elements such as portraits or text. A general category image may be an image that only contains regular category elements. A border category image may be an initial static image that contains image border elements. An invalid category image may be an image that only contains elements such as solid colors or gradient colors. For example, an initial static image that only consists of elements without obvious meanings such as all black, all white, or only gradient colors may be classified as an invalid category image.

[0092] The initial static image is input into the image classification model. The initial static image will be entered into any detection and recognition model in the image classification model in parallel, and the matching detection and recognition model will output the corresponding classification result. Figure 3 , Figure 3 The flowchart of classifying cover images according to some embodiments of the present disclosure is schematically shown.

[0093] refer to Figure 3 It can be seen that the image classification model includes a human body detection model or a face detection model. When the initial static image passes through the human body detection model or the face detection model, the human body detection model or the face detection model will perform image classification detection on the initial static image, and then determine the portrait image according to the classification detection result. For example, the detected face and human body results are further refined and classified according to the presence or absence of faces and the number and area ratio of faces in the image, and are divided into the following categories: big head, half body, full body, multiple people, back view, etc. The human body detection model or the face detection model can be implemented based on any human body detection algorithm and face detection algorithm, and the present disclosure does not make any special limitation on the human body detection algorithm and face detection algorithm adopted.

[0094] The text recognition model uses a text recognition algorithm to detect text in the entire image and extract text from the image. When the initial static image flows through the text recognition model, the text recognition model can detect whether the initial static image contains text and output the corresponding classification detection result, and then classify the initial static image into a text-type image based on the obtained classification detection result. For example, an image in which the maximum area of ​​the rectangular box of the detected text area accounts for more than 20% of the area of ​​the entire image is a text-type cover image. Similarly, the text recognition model can be implemented using any text recognition algorithm, and the present disclosure does not make any special limitations.

[0095] Portrait images and text images are images that contain special elements. For regular images that do not contain the above elements, the present disclosure uses an image detection model to realize the recognition and detection of regular images. For example, regular images can include general category images, border category images, and invalid category images. For the classification of these three categories of images, in order to improve the efficiency of processing tens of millions of cover images and for the uniqueness of these categories, this embodiment designs an end-to-end multi-task cover image classification algorithm, which can simultaneously output these required categories, which not only significantly improves the execution efficiency, but also further optimizes the accuracy of classification.

[0096] After classifying the image covers of tens of millions or hundreds of millions of song covers, it can be found that there are obvious differences between the images of each category, and the way of dynamicizing static images in each image category is also different. The specific implementation process of dynamic processing of static images will be described in more detail below.

[0097] For invalid category images, such as song covers that only contain all black, all white, or only gradient colors and have no obvious meaning, it is also meaningless to dynamize their meaningless content. Therefore, the song covers that fall under the invalid category of images can be directly ignored to avoid generating meaningless dynamic video generation tasks.

[0098] In one embodiment of the present disclosure, for step S230, dynamic video generation processing is performed on the initial static image according to the image category to obtain a dynamic video cover corresponding to the target media resource, including: determining the area ratio of the portrait area in the portrait image; dividing the portrait image into a first portrait image and a second portrait image according to the area ratio, the first portrait image being an image with an area ratio greater than a preset ratio threshold; performing dynamic video generation processing on the first portrait image to obtain a first portrait dynamic cover; performing dynamic video generation processing on the second portrait image to obtain a second portrait dynamic cover.

[0099] Among them, the portrait area can be the area where the portrait elements are located in the portrait image. The area ratio can be the ratio of the portrait area to the entire portrait image. The first portrait image can be a portrait image in which the ratio of the portrait area to the entire portrait image is greater than the ratio threshold. For example, the first portrait image can be a big-head portrait image. The second portrait image can be a portrait image in which the ratio of the portrait area to the entire portrait image is less than or equal to the ratio threshold. For example, the second portrait image can be a portrait image containing elements such as half body, full body, multiple people, back view, etc. The first portrait dynamic cover can be a dynamic video cover dynamically generated based on the first portrait image. The second portrait dynamic cover can be a dynamic video cover dynamically generated based on the second portrait image.

[0100] For portrait images, the animation of the portrait is relatively sensitive. This embodiment retains the portrait area and only performs video animation on the non-background area for dynamic processing. This disclosure determines the area ratio of the portrait area in the portrait image, and further divides the portrait image into a first portrait image and a second portrait image according to the area ratio. Figure 4 , Figure 4 A schematic diagram of a portrait cover image according to some embodiments of the present disclosure is schematically shown. Figure 4 The left image in may be a first portrait image, Figure 4 The right image in the figure may be a second portrait image, and the first portrait image may be a head image, that is, the first portrait image may be an image in which the area ratio of the portrait area in the entire image is greater than a preset ratio threshold.

[0101] Furthermore, the second portrait images can be further divided, such as into portrait images of large head, half body, full body, multiple people, back view, etc. Then, dynamic processing is performed according to the characteristics of different portrait types. For example, for the first portrait images (large head portrait images), dynamic video generation processing is performed using an animation strategy adapted to such images to obtain a first portrait dynamic cover; for the second portrait images (other portrait categories except head portrait images), dynamic video generation processing is performed to obtain a second portrait dynamic cover, so as to ensure that the static images under each image category can achieve the best dynamic video effect.

[0102] In one embodiment of the present disclosure, a first portrait image is subjected to dynamic video generation processing to obtain a first portrait dynamic cover, including: determining an image color category of the first portrait image, the image color category including a full-image color category and an edge color category; determining a dynamic special effects template for the first portrait image based on the full-image color category and the edge color category; and generating the first portrait dynamic cover based on the dynamic special effects template.

[0103] The image color category may be the color category to which the first portrait image belongs. The full image color category may be the color category corresponding to the full image of the first portrait image. The edge color category may be the color category corresponding to the edge of the first portrait image. The dynamic special effects template, also referred to as a dynamic special effects template, may be a special effects template adapted to the first portrait image determined from a dynamic special effects template library according to the image color category.

[0104] For the first portrait image, since the portrait occupies a relatively large proportion of the image, after removing the portrait area that cannot be animated, the remaining area of ​​the background image is small and cannot be animated. Therefore, for the animation strategy of this type of portrait image, this embodiment adopts the method of adaptively selecting dynamic template special effects to generate a dynamic special effects video for the first portrait image.

[0105] In the pre-configured dynamic template library, since there is a lot of data for dynamic special effects templates, and the most important factor affecting the adaptability between templates and images is color, the first portrait image is color-classified to perform adaptive selection of the best template. Specifically, the image color category of the first portrait image is determined. Here, the color calculation of the first portrait image has two values, one is the full image color category (i.e., the full image color classification), and the other is the edge color category (i.e., the edge image color classification).

[0106] Then, according to the classification results of the full image color category and the edge color category, the best matching template is selected from the special effect template library as the dynamic special effect template of the first portrait image; then, based on the dynamic special effect template, the first portrait image is processed for dynamic video generation to obtain the first portrait dynamic cover. The first portrait dynamic cover is generated by adopting an animation strategy adapted to the first portrait image, which can achieve the best dynamic video effect.

[0107] In one embodiment of the present disclosure, determining the image color category of a first portrait image includes: determining first channel pixels of each color channel in the first portrait image; determining a first histogram vector of each color channel based on multiple first channel pixels; concatenating the first histogram vectors of multiple color channels to obtain a full-image histogram vector; and determining the full-image color category based on the full-image histogram vector.

[0108] The first channel pixel may be pixel information corresponding to each color channel of the full image in the first portrait image. The first histogram vector may be a color histogram vector calculated based on the first channel pixels of each color channel. The full image histogram vector may be a histogram vector obtained by splicing the first histogram vectors of multiple color channels.

[0109] When determining the full-image color category of the first portrait image, you can first obtain the full Figure 3The pixel values ​​corresponding to each color channel, for example, the pixel values ​​of the three color channels of blue, green, and red (blue green red, BGR), are used as the first channel pixels. The color histogram of each color channel is calculated according to the first channel pixels corresponding to each of the three color channels, and the color histogram of each channel is divided into 8 bins to form a 1*8-dimensional histogram vector to obtain the first histogram vector; the first histogram vectors of the three color channels are spliced ​​together to form a 1*24-dimensional histogram vector in total, that is, the full-image histogram vector is obtained.

[0110] The obtained full-image histogram vector can be used to determine the full-image color category of the first portrait image, and the full-image color category is used to match the dynamic special effect template applicable to the full-image type. For example, the dynamic special effect template applicable to the full-image type may include full-screen particle effects, flashing, falling, etc. Through the above operation, the full-image color category of the first portrait image can be determined to adaptively match the best dynamic template.

[0111] In one embodiment of the present disclosure, determining the image color category of the first portrait image also includes: obtaining an edge area image of the first portrait image, determining the second channel pixels of each color channel in the edge area image; determining a second histogram vector of each color channel based on multiple second channel pixels; concatenating the second histogram vectors of multiple color channels to obtain an edge histogram vector; and determining the edge color category based on the edge histogram vector.

[0112] The edge area image may be an image outside the edge area border (such as a blue frame) in the first portrait image. The second channel pixel may be pixel information corresponding to each color channel in the edge area of ​​the first portrait image. The second histogram vector may be a color histogram vector calculated based on the second channel pixels of each color channel. The edge histogram vector may be a histogram vector obtained by splicing the second histogram vectors of multiple color channels.

[0113] After determining the full-image color category of the first portrait image, the edge color category of the first portrait image is calculated in the same way. Specifically, the edge region of the first portrait image, such as the region outside the image frame in the first portrait image, is first determined as the edge region image. The pixel values ​​corresponding to the three color channels in the edge region image are obtained as the second channel pixels; the color histograms of the three color channels are calculated respectively, and the color histogram of each color channel is divided into 8 bins to form a 1*8-dimensional histogram vector as the second histogram vector.

[0114] The second histogram vectors of the three color channels are spliced ​​to form a 1*24 color histogram b as an edge histogram vector, so as to determine the edge color category according to the edge histogram vector and use it to select a matching border-type special effects template. The above steps provide a processing process for calculating the color category of the edge image area. Before matching the dynamic special effects template of the first portrait image, all the first portrait image covers are calculated according to the above calculation method.

[0115] In one embodiment of the present disclosure, a dynamic special effects template for a first portrait image is determined based on a full-image color category and an edge color category, including: obtaining a candidate special effects template corresponding to a candidate image cover, the candidate special effects template including a candidate full-image special effects template and a candidate border special effects template; determining a matching mapping relationship between the image color category and the candidate special effects template; determining a target image category corresponding to the first portrait image based on the full-image color category and the edge color category; determining a dynamic special effects template from the candidate special effects templates based on the matching mapping relationship and the target image category; the candidate full-image special effects template is obtained by clustering candidate image full-image histogram vectors of multiple candidate image covers, and the candidate border special effects template is obtained by clustering candidate edge area histogram vectors of multiple candidate image covers.

[0116] Among them, the candidate special effects template may be a dynamic special effects template pre-stored in the special effects template library. The candidate full-image special effects template may be a dynamic special effects template in the special effects template library for adapting to the full-image color category of the first portrait image. The candidate border special effects template may be a dynamic special effects template in the special effects template library for adapting to the edge color category of the first portrait image. The matching mapping relationship may be a mapping relationship between the image color category and the candidate special effects template. The target image category may be the image category to which the first portrait image belongs. The dynamic special effects template may be a special effects template selected from the candidate special effects templates and matching the first portrait image.

[0117] For the first portrait image, the determined full-image color category and edge color category can be used as feature a and feature b, respectively, and all features a and features b are clustered to cluster a certain number of cluster centers, such as 50 cluster centers, as category classification. For example, the clustering algorithm can adopt a K-Nearest Neighbor (KNN) classification algorithm, etc., and the present disclosure does not make any special limitation on the specific clustering algorithm.

[0118] At the same time, the dynamic special effects templates in the dynamic special effects template library are subjected to histogram feature extraction in the same way and classified to the nearest cluster center as category mapping. The candidate special effects templates include candidate full-image special effects templates and candidate border special effects templates. The candidate full-image special effects templates are obtained by clustering the candidate full-image histogram vectors of multiple candidate image covers, and the candidate border special effects templates are obtained by clustering the candidate edge area histogram vectors of multiple candidate image covers, thereby completing the matching mapping relationship between the dynamic special effects templates and the categories of static images.

[0119] Subsequently, as long as the image category to which the static image belongs is calculated, a suitable special effect template can be selected from the candidate special effect templates of the same category as the dynamic special effect template. The matched dynamic special effect template is added to the first portrait image, and the video generation processing of the first portrait image is realized by this strategy.

[0120] In one embodiment of the present disclosure, a second portrait image is subjected to dynamic video generation processing to obtain a second portrait dynamic cover, including: determining a portrait area of ​​the second portrait image, masking the portrait area to obtain a masked portrait image, the masked portrait image including a masked portrait area and an area outside the mask; determining a portrait area ratio of the masked portrait area in the masked portrait image, and determining dynamic video generation parameters according to the portrait area ratio; based on the dynamic video generation parameters, performing dynamic video generation processing on the image of the area outside the mask to obtain a second portrait dynamic cover.

[0121] Among them, the masked portrait image can be an image obtained by masking the image elements of the portrait area in the second portrait image. The masked portrait area can be the area where the masked portrait image is located in the second portrait image. The masked external area can be the area outside the masked portrait image in the second portrait image. The portrait area ratio can be the ratio of the masked portrait area to the entire second portrait image. The dynamic video generation parameter can be a motion parameter used to control the second portrait image to generate a dynamic video cover, and the parameter can be used to control the degree of motion of the picture.

[0122] For the second portrait image, that is, the remaining portrait categories, including portrait images of half body, full body, multiple people, back view and other categories, the portrait area is first segmented out using a portrait segmentation algorithm, and the portrait area is masked to generate a binary mask of the portrait area. At this time, a masked portrait image can be obtained, which includes a masked portrait area (mask) that has been masked and a masked external area that has not been masked.

[0123] The mask is sent to the image generation model to keep the image in the mask area from being animated, and only animate the image outside the mask area. In the present invention, an image generation video model, such as a Segment Any RGBD (SAD) model, is used to generate dynamic cover videos. Since the proportion of the size of the portrait in the image is different in the above-mentioned portrait image types, different motion parameters will be used to control the degree of motion of the picture when performing dynamic video generation of images with different types of portrait proportions.

[0124] For example, the portrait area ratio of the masked portrait area in the masked portrait image is determined, and the motion parameters used in dynamic video generation, i.e., dynamic video generation parameters, are determined according to the portrait area ratio. The dynamic video generation parameters are used to perform dynamic video generation processing on the image of the masked area, and the degree of picture motion of the image of the masked area is controlled to obtain a second portrait dynamic cover. Through the above steps, the dynamic video generation process of the remaining portrait categories can be implemented to obtain the corresponding dynamic video covers.

[0125] In one embodiment of the present disclosure, for step S230, dynamic video generation processing is performed on the initial static image according to the image category to obtain a dynamic video cover corresponding to the target media resource, including: obtaining a pre-configured text video generation model, the text video generation model is trained based on text video samples; using the text video generation model, dynamic video generation processing is performed on the text image to obtain a text dynamic cover.

[0126] The text video generation model may be a model that generates a text-specific animation video using a text image as input. For example, the text video generation model may generate animation effects such as text rotation, zooming in and out, color flashing, fading in and out, etc. based on the text image. The text dynamic cover may be a dynamic video cover with a video animation effect containing text.

[0127] For text images, refer to Figure 5 , Figure 5 The following schematic diagrams show a text cover image according to some embodiments of the present disclosure. Since text accounts for a relatively large proportion of an image and is a prominent target in a text image, the target that needs to be animated should be the text in the process of animating a static image of text. Based on the conventional knowledge of image animating, it can be known that the video animation effect of text is quite different from the animation effect of ordinary image content.

[0128] In order to adapt the video animation effect of text, the present disclosure pre-trains a text video generation model to use the text video generation model to perform dynamic video generation processing on text images to generate corresponding text dynamic covers. The text video generation model is used to realize dynamic processing of text images to obtain an adapted text dynamic cover.

[0129] refer to Figure 6 , Figure 6 The process diagram of generating a dynamic video cover of a text cover image according to some embodiments of the present disclosure is schematically shown. The text video generation model can be trained by the following steps: obtaining a text cover image, and a pre-made text animation video corresponding to the text cover image (i.e., a video sample of the text type), combining the two, and generating a training data set. On the basis of the image-generated video model, the training data set is used to train this category model to generate special animation modes for text, such as text rotation, zooming in and out, tonal flashing, fading in and out, and other animation effects. During use, the test text cover is input into the trained model, and the corresponding dynamic cover video is generated by the model. For example, the image-generated video model in the present disclosure can be a Stable Video Diffusion (SVD) model, and the present disclosure does not make any special limitation on the specific model structure of the image-generated video model.

[0130] In one embodiment of the present disclosure, for step S230, dynamic video generation processing is performed on the initial static image according to the image category to obtain a dynamic video cover corresponding to the target media resource, including: determining the border point coordinates of the border category image, and determining the border area based on the border point coordinates; determining the image within the border area based on the border area, performing dynamic video generation processing on the image within the border area to obtain an initial dynamic image; filling the initial dynamic image into the border area to obtain a border image dynamic cover.

[0131] Among them, the border category image can be an image containing border decoration elements, and the border elements in the border category image mainly play a role of beautification and decoration. The border point coordinates can be the coordinate points corresponding to the border elements in the border category image. The border area can be the image area determined by the border point coordinates. The image in the border area can be the image in the border category image that is in the border area. The initial dynamic image can be a dynamic image obtained by performing dynamic video generation processing on the image in the border area. The border image dynamic cover can be a dynamic video cover obtained by filling the dynamically generated initial dynamic image into the position of the image in the original border area.

[0132] For border images, the borders in the image mainly serve to beautify and decorate. Figure 7 , Figure 7A schematic diagram of a border-type cover image according to some embodiments of the present disclosure is schematically shown. When generating a dynamic video cover of a border-category image, the static decorative effect of the border should be maintained to ensure that the image border does not interfere with the visual effect of the dynamic cover. However, when these cover images are fed into the generation model, the model may mistakenly identify the border as meaningful content, thereby affecting the generation of normal animation. Therefore, before such static images are animated, accurate border detection is required, followed by removal of the border, and then the image is animated.

[0133] The step of border detection and removal is crucial to improving the quality of dynamic transformation of static cover. Based on this, the present disclosure proposes an efficient and accurate image border detection algorithm, which determines the border point coordinates of the border category image and then determines the border area based on the border point coordinates. After determining the border area, the image within the border area is obtained based on the border area, while the image part outside the border area is retained.

[0134] The image in the border area is processed for dynamic video generation to obtain the initial dynamic image corresponding to the image in the border area, and then the initial dynamic image is filled into the border area to obtain the dynamic video cover corresponding to the border category image, that is, the border image dynamic cover. Since the image border detection algorithm can accurately detect the border area, it can significantly improve the efficiency and accuracy of cover image cropping, thereby optimizing the entire dynamic process.

[0135] In one embodiment of the present disclosure, determining the coordinates of border points of a border category image includes: performing horizontal pixel accumulation processing and vertical pixel accumulation processing on the border category image respectively to obtain a horizontal pixel accumulation vector and a vertical pixel accumulation vector; calculating the horizontal vector gradient of the horizontal pixel accumulation vector from left to right; taking the pixel points whose horizontal gradient values ​​in the horizontal vector gradient are greater than the gradient threshold as the first border point coordinates; calculating the vertical vector gradient of the vertical pixel accumulation vector from right to left; taking the pixel points whose vertical gradient values ​​in the vertical vector gradient are greater than the gradient threshold as the second border point coordinates.

[0136] Among them, the horizontal pixel accumulation vector can be a vector obtained by respectively accumulating the pixel values ​​in the same horizontal row in the border category image. The vertical pixel accumulation vector can be a vector obtained by respectively accumulating the pixel values ​​in the same vertical column in the border category image. The horizontal vector gradient can be the gradient corresponding to the horizontal pixel accumulation vector. The horizontal gradient value can be the specific numerical value of the horizontal vector gradient. The first border point coordinates can be the pixel point whose horizontal gradient value in the horizontal vector gradient is greater than the gradient threshold, that is, the pixel point whose gradient value in the horizontal vector gradient suddenly changes. The vertical vector gradient can be the gradient corresponding to the vertical pixel accumulation vector. The vertical gradient value can be the specific numerical value of the vertical vector gradient. The second border point coordinates can be the pixel point whose vertical gradient value in the vertical vector gradient is greater than the gradient threshold, that is, the pixel point whose gradient value in the vertical vector gradient suddenly changes.

[0137] The specific implementation process of the image border detection algorithm is as follows: For the entire image, if you want to detect the border area, you need to detect the coordinates of the border points in the image, such as calculating the coordinates of the upper left corner (x1, y1) and the lower right corner (x2, y2) of the border. Figure 8 , Figure 8 The schematic diagram of pixel accumulation processing for border-type cover images according to some embodiments of the present disclosure is schematically shown. The border-type image is subjected to horizontal pixel accumulation processing and vertical pixel accumulation processing respectively to obtain a horizontal pixel accumulation vector and a vertical pixel accumulation vector. For example, the image size of the border-type image is 1080 in height and 1920 in width. Through the above processing, a 1*1080 horizontal pixel accumulation vector is obtained to calculate y1 and y2; at the same time, a 1*1920 vertical accumulation vector is obtained to calculate x1 and x2.

[0138] For these two accumulated vectors, the points where the values ​​suddenly change are calculated from left to right, that is, the horizontal vector gradient of the horizontal pixel accumulated vector is calculated by sliding the frame from left to right. If the gradient value change of the horizontal vector gradient is greater than the pre-configured gradient threshold, the pixel point can be basically determined as the starting point x1 or y1 of the border, that is, the coordinates of the first border point. Then, the vertical vector gradient of the vertical pixel accumulated vector is calculated by sliding the frame from right to left. If the gradient value change of the vertical vector gradient is greater than the gradient threshold, the pixel point can be basically determined as the ending point x2 or y2 of the border, that is, the coordinates of the second border point.

[0139] For the images in the [x1, y1, x2, y2] area after the calculation is completed, the dynamic cover of the corresponding category is generated according to the previous cover image classification method. For the generated video, fill it into the original static image [x1, y1, x2, y2] to complete the animation of the video with the border image. In this way, the validity of the border is guaranteed, and the picture after animation processing is prevented from exceeding the border and forming an erroneous visual picture, thereby ensuring the visual effect of the generated dynamic video.

[0140] Furthermore, when using static cover animation technology to create dynamic video covers, in order to overcome the problem that the generated dynamic video may not be good due to poor image quality or improper animation. In the process of dynamic video generation, this embodiment conducts in-depth analysis on the unique properties of static image covers, automatically adjusts the dynamic effects, and ensures that each static cover, regardless of its style, color or element composition, can show excellent animation effects. Through precise algorithm and strategy optimization, this disclosure is committed to allowing tens of millions of static image covers to achieve satisfactory visual effects when dynamically presented.

[0141] Exemplary Devices

[0142] After introducing the method of the exemplary embodiment of the present disclosure, next, refer to Fig. 9 A dynamic cover generation device according to an exemplary embodiment of the present disclosure is described.

[0143] exist Fig. 9 In the embodiment, the dynamic cover generation device 900 may include: an initial image acquisition module 910, an image category determination module 920 and a dynamic cover generation module 930. Among them: the initial image acquisition module 910 is used to acquire the initial static image corresponding to the target media resource; the image category determination module 920 is used to determine the image category of the initial static image; the dynamic cover generation module 930 is used to perform dynamic video generation processing on the initial static image according to the image category to obtain the dynamic video cover corresponding to the target media resource.

[0144] In one embodiment of the present disclosure, the image category determination module 920 includes an image category determination unit, which is used to: obtain a pre-configured image classification model, the image classification model includes one or more of a human body detection model, a face detection model, a text recognition model and an image detection model; use the image classification model to perform image classification processing on the initial static image to obtain an image category.

[0145] In one embodiment of the present disclosure, the image category determination unit includes an image category determination subunit, which is used to: use a human body detection model or a face detection model to perform image classification detection on the initial static image, and determine a portrait image based on the obtained classification detection result; use a text recognition model to perform image classification detection on the initial static image, and determine a text image based on the obtained classification detection result; use an image detection model to perform image classification detection on the initial static image, and determine a regular image based on the obtained classification detection result, and the regular image includes one or more of a general category image, a border category image, and an invalid category image.

[0146] In one embodiment of the present disclosure, the image category includes portrait images, and the dynamic video cover includes a first portrait dynamic cover and a second portrait dynamic cover; the dynamic cover generation module 930 includes a portrait cover generation unit, which is used to: determine the area ratio of the portrait area in the portrait image; divide the portrait image into a first portrait image and a second portrait image according to the area ratio, and the first portrait image is an image with an area ratio greater than a preset ratio threshold; perform dynamic video generation processing on the first portrait image to obtain a first portrait dynamic cover; perform dynamic video generation processing on the second portrait image to obtain a second portrait dynamic cover.

[0147] In one embodiment of the present disclosure, a portrait cover generation unit includes a first portrait cover generation sub-unit, which is used to: determine the image color category of the first portrait image, the image color category includes the full image color category and the edge color category; determine the dynamic special effects template of the first portrait image according to the full image color category and the edge color category; and generate the first portrait dynamic cover based on the dynamic special effects template.

[0148] In one embodiment of the present disclosure, the first portrait cover generation subunit includes a first color category generation subunit, which is used to: determine the first channel pixels of each color channel in the first portrait image; determine the first histogram vector of each color channel based on multiple first channel pixels; splice the first histogram vectors of multiple color channels to obtain a full-image histogram vector; and determine the full-image color category based on the full-image histogram vector.

[0149] In one embodiment of the present disclosure, the first portrait cover generation subunit also includes a second color category generation subunit, which is used to: obtain an edge area image of the first portrait image, determine the second channel pixels of each color channel in the edge area image; determine the second histogram vector of each color channel based on multiple second channel pixels; splice the second histogram vectors of multiple color channels to obtain an edge histogram vector; and determine the edge color category based on the edge histogram vector.

[0150] In one embodiment of the present disclosure, the first portrait cover generation subunit includes a special effects generation subunit, which is used to: obtain candidate special effects templates corresponding to the candidate image cover, the candidate special effects templates include candidate full-image special effects templates and candidate border special effects templates; determine the matching mapping relationship between the image color category and the candidate special effects templates; determine the target image category corresponding to the first portrait image based on the full-image color category and the edge color category; determine the dynamic special effects template from the candidate special effects templates based on the matching mapping relationship and the target image category; the candidate full-image special effects template is obtained by clustering the candidate image full-image histogram vectors of multiple candidate image covers, and the candidate border special effects template is obtained by clustering the candidate edge area histogram vectors of multiple candidate image covers.

[0151] In one embodiment of the present disclosure, the portrait cover generation unit includes a second portrait cover generation sub-unit, which is used to: determine the portrait area of ​​the second portrait image, mask the portrait area, and obtain a masked portrait image, the masked portrait image includes the masked portrait area and the masked external area; determine the portrait area ratio of the masked portrait area in the masked portrait image, and determine the dynamic video generation parameters according to the portrait area ratio; based on the dynamic video generation parameters, perform dynamic video generation processing on the image of the masked external area to obtain a second portrait dynamic cover.

[0152] In one embodiment of the present disclosure, the image category includes text images, and the dynamic video cover includes a text dynamic cover; the dynamic cover generation module 930 includes a text cover generation unit, which is used to: obtain a pre-configured text video generation model, the text video generation model is obtained based on text video sample training; use the text video generation model to perform dynamic video generation processing on the text image to obtain a text dynamic cover.

[0153] In one embodiment of the present disclosure, the image category includes a border category image, and the dynamic video cover includes a border image dynamic cover; the dynamic cover generation module 930 includes a border cover generation unit, which is used to: determine the border point coordinates of the border category image, and determine the border area based on the border point coordinates; determine the image in the border area based on the border area, perform dynamic video generation processing on the image in the border area to obtain an initial dynamic image; fill the initial dynamic image into the border area to obtain a border image dynamic cover.

[0154] In one embodiment of the present disclosure, the border point coordinates include first border point coordinates and second border point coordinates, and the border cover generation unit includes a border coordinate determination subunit, which is used to: perform horizontal pixel accumulation processing and vertical pixel accumulation processing on the border category image respectively to obtain a horizontal pixel accumulation vector and a vertical pixel accumulation vector; calculate the horizontal vector gradient of the horizontal pixel accumulation vector in order from left to right; use the pixel points whose horizontal gradient values ​​in the horizontal vector gradient are greater than the gradient threshold as the first border point coordinates; calculate the vertical vector gradient of the vertical pixel accumulation vector in order from right to left; use the pixel points whose vertical gradient values ​​in the vertical vector gradient are greater than the gradient threshold as the second border point coordinates.

[0155] Since the various functional modules of the dynamic cover generation device of the example embodiment of the present disclosure correspond to the steps of the example embodiment of the dynamic cover generation method described above, for details not disclosed in the embodiment of the device of the present disclosure, please refer to the embodiment of the dynamic cover generation method described above in the present disclosure, and no further details will be given here.

[0156] It should be noted that although several modules or units of the dynamic cover generation device are mentioned in the above detailed description, such division is not mandatory. In fact, according to the embodiments of the present disclosure, the features and functions of two or more modules or units described above can be embodied in one module or unit. Conversely, the features and functions of one module or unit described above can be further divided into multiple modules or units for embodiment.

[0157] In a third aspect of the embodiments of the present disclosure, a computer-readable storage medium is provided, on which a computer program is stored. When the computer program is executed by a processor, the dynamic cover generation method as described in the first aspect above is implemented.

[0158] Exemplary Media

[0159] After introducing the device of the exemplary embodiment of the present disclosure, next, refer to Fig.10 A storage medium according to an exemplary embodiment of the present disclosure is described.

[0160] In some embodiments, various aspects of the present disclosure may also be implemented as a medium on which program code is stored, which, when executed by a processor of a device, is used to implement the steps in the dynamic cover generation method according to various exemplary embodiments of the present disclosure described in the above “Exemplary Method” section of this specification.

[0161] For example, when the processor of the device executes the program code, the following may be implemented: Figure 2Step S210 described in the above, obtain the initial static image corresponding to the target media resource. Step S220, determine the image category of the initial static image. Step S230, perform dynamic video generation processing on the initial static image according to the image category to obtain the dynamic video cover corresponding to the target media resource.

[0162] refer to Fig.10 As shown, a program product 1000 for implementing the above-mentioned dynamic cover generation method or implementing the above-mentioned dynamic cover generation method according to an embodiment of the present disclosure is described, which can adopt a portable compact disk read-only memory (CD-ROM) and include program code, and can be run on a terminal device, such as a personal computer. However, the program product of the present disclosure is not limited thereto.

[0163] The program product may use any combination of one or more readable media. The readable medium may be a readable signal medium or a readable storage medium. The readable storage medium may be, for example, but not limited to, an electrical, magnetic, optical, electromagnetic, infrared, or semiconductor system, device or device, or any combination of the above. More specific examples (non-exhaustive list) of readable storage media include: an electrical connection with one or more wires, a portable disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the above.

[0164] Computer readable signal media may include data signals propagated in baseband or as part of a carrier wave, wherein readable program code is carried. Such propagated data signals may take a variety of forms, including but not limited to electromagnetic signals, optical signals, or any suitable combination thereof. Readable signal media may also be any readable medium other than a readable storage medium.

[0165] Program code for performing the operations of the present disclosure may be written in any combination of one or more programming languages, including object-oriented programming languages ​​such as Java, C++, etc., and conventional procedural programming languages ​​such as "C" or similar programming languages. The program code may be executed entirely on the user computing device, partially on the user device, partially on a remote computing device, or entirely on a remote computing device or server. In the case of a remote computing device, the remote computing device may be connected to the user computing device through any type of network, including a local area network (LAN) or a wide area network (WAN).

[0166] Exemplary Computing Devices

[0167] After introducing the dynamic cover generation method, dynamic cover generation device and storage medium of the exemplary embodiment of the present disclosure, next, refer to Fig.11 An electronic device according to an exemplary embodiment of the present disclosure is described.

[0168] Those skilled in the art will appreciate that various aspects of the present disclosure may be implemented as systems, methods or program products. Therefore, various aspects of the present disclosure may be specifically implemented in the following forms, namely: complete hardware embodiments, complete software embodiments (including firmware, microcode, etc.), or embodiments combining hardware and software aspects, which may be collectively referred to herein as "circuits", "modules" or "systems".

[0169] In some possible embodiments, the electronic device according to the present disclosure may include at least one processing unit and at least one storage unit. The storage unit stores program code, and when the program code is executed by the processing unit, the processing unit executes the steps of the dynamic cover generation method according to various exemplary embodiments of the present disclosure described in the above "Exemplary Method" section of this specification. For example, the processing unit may execute the following steps: Figure 2 In step S210, an initial static image corresponding to the target media resource is obtained. In step S220, an image category of the initial static image is determined. In step S230, a dynamic video generation process is performed on the initial static image according to the image category to obtain a dynamic video cover corresponding to the target media resource.

[0170] Refer to the following Fig.11 1100 according to this embodiment of the present disclosure is described. Fig.11 The electronic device 1100 shown is merely an example and should not bring any limitation to the functions and scope of use of the embodiments of the present disclosure.

[0171] like Fig.11 As shown, the electronic device 1100 is in the form of a general computing device. The components of the electronic device 1100 may include, but are not limited to: the at least one processing unit 1110, the at least one storage unit 1120, a bus 1130 connecting different system components (including the storage unit 1120 and the processing unit 1110), and a display unit 1140.

[0172] The storage unit stores program codes, which can be executed by the processing unit 1110, so that the processing unit 1110 executes the steps according to various exemplary embodiments of the present disclosure described in the above “Exemplary Method” section of this specification.

[0173] The storage unit 1120 may include a readable medium in the form of a volatile storage unit, such as a random access memory unit (RAM) 1121 and / or a cache memory unit 1122 , and may further include a read-only memory unit (ROM) 1123 .

[0174] The storage unit 1120 may also include a program / utility 1124 having a set (at least one) of program modules 1125, such program modules 1125 including but not limited to: an operating system, one or more application programs, other program modules, and program data, each of which or some combination may include an implementation of a network environment.

[0175] Bus 1130 may represent one or more of several types of bus structures, including a memory unit bus or memory unit controller, a peripheral bus, an accelerated graphics port, a processing unit, or a local bus using any of a variety of bus architectures.

[0176] The electronic device 1100 may also communicate with one or more external devices 1170 (e.g., keyboards, pointing devices, Bluetooth devices, etc.), may also communicate with one or more devices that enable a user to interact with the electronic device 1100, and / or communicate with any device that enables the electronic device 1100 to communicate with one or more other computing devices (e.g., routers, modems, etc.). Such communication may be performed via an input / output (I / O) interface 1150. Furthermore, the electronic device 1100 may also communicate with one or more networks (e.g., a local area network (LAN), a wide area network (WAN), and / or a public network, such as the Internet) via a network adapter 1160. As shown, the network adapter 1160 communicates with other modules of the electronic device 1100 via a bus 1130. It should be understood that, although not shown in the figure, other hardware and / or software modules may be used in conjunction with the electronic device 1100, including but not limited to: microcode, device drivers, redundant processing units, external disk drive arrays, RAID systems, tape drives, and data backup storage systems.

[0177] It should be noted that, although several units / modules or sub-units / modules of the dynamic cover generation device are mentioned in the above detailed description, such division is merely exemplary and not mandatory. In fact, according to the embodiments of the present disclosure, the features and functions of two or more units / modules described above may be embodied in one unit / module. Conversely, the features and functions of one unit / module described above may be further divided to be embodied by multiple units / modules.

[0178] In addition, although the operations of the disclosed method are described in a specific order in the drawings, this does not require or imply that the operations must be performed in this specific order, or that all the operations shown must be performed to achieve the desired results. Additionally or alternatively, some steps may be omitted, multiple steps may be combined into one step, and / or one step may be decomposed into multiple steps.

[0179] Although the spirit and principle of the present disclosure have been described with reference to several specific embodiments, it should be understood that the present disclosure is not limited to the disclosed specific embodiments, and the division of various aspects does not mean that the features in these aspects cannot be combined to benefit, and such division is only for the convenience of expression. The present disclosure is intended to cover various modifications and equivalent arrangements included in the spirit and scope of the attached claims.

Claims

1. A dynamic cover generation method, characterized in that: include: Obtaining an initial static image corresponding to the target media resource; Determining an image category of the initial static image; The initial static image is processed for dynamic video generation according to the image category to obtain a dynamic video cover corresponding to the target media resource.

2. The method according to claim 1, characterized in that The image category includes portrait images, and the dynamic video cover includes a first portrait dynamic cover and a second portrait dynamic cover; The performing dynamic video generation processing on the initial static image according to the image category to obtain a dynamic video cover corresponding to the target media resource includes: Determining the area ratio of the portrait area in the portrait image; Dividing the portrait image into a first portrait image and a second portrait image according to the area ratio, wherein the first portrait image is an image whose area ratio is greater than a preset ratio threshold; Performing dynamic video generation processing on the first portrait image to obtain the first portrait dynamic cover; The second portrait image is processed for dynamic video generation to obtain the second portrait dynamic cover.

3. The method according to claim 2, characterized in that The step of performing dynamic video generation processing on the first portrait image to obtain the first portrait dynamic cover includes: Determine an image color category of the first portrait image, where the image color category includes an overall image color category and an edge color category; Determining a dynamic special effects template for the first portrait image according to the full-image color category and the edge color category; Based on the dynamic special effects template, the first portrait dynamic cover is generated.

4. The method according to claim 3, characterized in that The step of determining the dynamic special effects template of the first portrait image according to the full image color category and the edge color category includes: Obtain candidate special effect templates corresponding to the candidate image cover, wherein the candidate special effect templates include candidate full-image special effect templates and candidate border special effect templates; Determine a matching mapping relationship between the image color category and the candidate special effect template; Determining a target image category corresponding to the first portrait image according to the full image color category and the edge color category; Determining the dynamic special effect template from the candidate special effect templates according to the matching mapping relationship and the target image category; The candidate full-image special effect template is obtained by clustering the candidate full-image histogram vectors of multiple candidate image covers, and the candidate border special effect template is obtained by clustering the candidate edge area histogram vectors of multiple candidate image covers.

5. The method according to claim 2, characterized in that: The step of performing dynamic video generation processing on the second portrait image to obtain the second portrait dynamic cover includes: Determine a portrait region of the second portrait image, perform mask processing on the portrait region, and obtain a masked portrait image, wherein the masked portrait image includes the masked portrait region and a masked external region; Determining a portrait area ratio of the masked portrait area in the masked portrait image, and determining a dynamic video generation parameter according to the portrait area ratio; Based on the dynamic video generation parameters, dynamic video generation processing is performed on the image in the outer area of ​​the mask to obtain the second portrait dynamic cover.

6. The method according to claim 1, characterized in that The image category includes text images, and the dynamic video cover includes a text dynamic cover; The performing dynamic video generation processing on the initial static image according to the image category to obtain a dynamic video cover corresponding to the target media resource includes: Obtaining a preconfigured text-video generation model, wherein the text-video generation model is trained based on text-video samples; The text video generation model is used to perform dynamic video generation processing on the text image to obtain the text dynamic cover.

7. The method according to claim 1, characterized in that The image category includes a frame category image, and the dynamic video cover includes a frame image dynamic cover; The performing dynamic video generation processing on the initial static image according to the image category to obtain a dynamic video cover corresponding to the target media resource includes: Determine the frame point coordinates of the frame category image, and determine the frame area based on the frame point coordinates; Determine an image within the frame area based on the frame area, and perform dynamic video generation processing on the image within the frame area to obtain an initial dynamic image; The initial dynamic image is filled into the frame area to obtain the frame image dynamic cover.

8. A dynamic cover generation device, characterized in that: include: An initial image acquisition module, used to acquire an initial static image corresponding to a target media resource; An image category determination module, used to determine the image category of the initial static image; The dynamic cover generation module is used to perform dynamic video generation processing on the initial static image according to the image category to obtain a dynamic video cover corresponding to the target media resource.

9. An electronic device, characterized in that: include: processor; as well as A memory having computer-readable instructions stored thereon, wherein the computer-readable instructions, when executed by the processor, implement the dynamic cover generation method as described in any one of claims 1 to 7.

10. A computer program product, comprising a computer program, characterized in that When the computer program is executed by a processor, the dynamic cover generation method according to any one of claims 1 to 7 is implemented.

Images