Image generation method and device, electronic equipment, storage medium and program product

Abstract

The invention provides an image generation method and device, electronic equipment, a storage medium and a program product, the image generation method is used for generating a description image of multimedia data, the image generation method comprises the following steps: receiving multimedia data and user information of a current user, the current user being an audience user of the multimedia data; determining a description text of the multimedia data according to the multimedia data and the user information; and generating a description image of the multimedia data according to the description text. According to the method, the multimedia data description text matched with the current user is determined based on the targeted multimedia data and the user information of the current user, and the description image of the multimedia data is generated based on the description text, so that the description image can be better matched with the watching demand of the current user, and the watching experience of the current user is improved. Therefore, the randomness of the automatically generated image is reduced, and the quality of the automatically generated image is improved.

Description

Technical Field

[0001] The present disclosure generally relates to the field of artificial intelligence technology, and more specifically, to an image generation method, device, electronic device, storage medium and program product. Background Art

[0002] At present, when domestic electronic devices such as televisions or mobile phones introduce videos to be played, their video details pages usually look like this: Figure 1 As shown, different components are only arranged in a fixed format, and at most a simple text introduction is added (for example Figure 1 The main reason is that the available copyrighted resources are not enough to support high-quality video detail pages. If such devices want to use high-quality pictures, they need to spend a lot of manpower to maintain and operate. Although with the development of artificial intelligence technology, AIGC (Artificial Intelligence Generated Content) can generate pictures in batches, the content generated by AIGC is relatively random and cannot be used directly on the page.

[0003] Therefore, how to automatically generate high-quality images has become a problem that needs to be solved urgently. Summary of the invention

[0004] The present disclosure provides an image generation method, an apparatus, an electronic device, a storage medium, and a program product, for solving at least one of the above problems.

[0005] According to a first aspect of an embodiment of the present disclosure, there is provided an image generating method for generating a description image of multimedia data, the image generating method comprising: receiving the multimedia data and user information of a current user, wherein the current user is an audience user of the multimedia data; determining a description text of the multimedia data based on the multimedia data and the user information; and generating a description image of the multimedia data based on the description text.

[0006] Optionally, generating a description image of the multimedia data based on the description text includes: receiving an image template, the image template including multiple preset elements; determining an element description text for each of the multiple preset elements based on the description text; generating an element image of each preset element based on the element description text of each preset element; and generating a description image of the multimedia data based on the element image of each preset element.

[0007] Optionally, generating a description image of the multimedia data based on the element image of each preset element includes: determining a dynamic element image and a static element image from the element images of the multiple preset elements; fusing the dynamic element image and the static element image into a dynamic layer and a static layer respectively, and merging the dynamic layer and the static layer into the description image of the multimedia data, wherein the dynamic layer can be regenerated.

[0008] Optionally, determining the dynamic element image and the static element image from the element images of the multiple preset elements includes: obtaining scene information, wherein the scene information is related to the current user and the current user's viewing behavior of the multimedia data; and determining the dynamic element image and the static element image from the element images of the multiple preset elements according to the scene information, wherein the dynamic layer can be regenerated when the scene information changes.

[0009] Optionally, the image generation method also includes: in response to receiving updated scene information, determining whether to trigger an update scene mode or a reset scene mode based on the updated scene information; when it is determined that the update scene mode is triggered, adjusting the dynamic element image according to the updated scene information to obtain an updated dynamic element image, fusing the updated dynamic element image into an updated dynamic layer, and merging the updated dynamic layer and the static layer into an updated description image of the multimedia data; when it is determined that the reset scene mode is triggered, re-executing the step from receiving the multimedia data and user information of the current user to generating a description image of the multimedia data based on the description text.

[0010] Optionally, the scene information includes at least one of the following: multimedia type, user portrait, camera sensor data, playback progress, and playback mode; wherein the multimedia type includes multimedia sets and independent multimedia, the multimedia set indicates that the multimedia data contains multiple multimedia files, and the independent multimedia indicates that the multimedia data contains only one multimedia file; the user portrait includes at least one of the following: viewing preference, gender, and age; the camera sensor data includes at least one of the following: user identification, viewing distance, background noise, and ambient light; the playback mode includes at least one of the following: children's mode, elderly mode, standard mode, and office mode.

[0011] Optionally, each of the multiple preset elements has element configuration information, the element configuration information includes an element name and an element display area, the element display area is used to represent the display area covered by the corresponding element image in the entire description image, wherein the element image of each preset element is generated according to the element description text of each preset element, including: generating a complete image of each preset element according to the element description text and the element configuration information of each preset element, wherein the size of the complete image matches the size of the description image; generating the element image of each preset element based on the complete image of each preset element and the element display area.

[0012] Optionally, generating the element image of each preset element based on the complete image and the element display area of ​​each preset element includes: based on the fusion effect of the complete images of the multiple preset elements, adjusting the complete image of at least one preset element among the multiple preset elements to obtain a corrected complete image of each preset element among the multiple preset elements; generating the element image of each preset element based on the corrected complete image and the element display area of ​​each preset element.

[0013] Optionally, determining the description text of the multimedia data based on the multimedia data and the user information includes: using a large language model to process the multimedia data and the user information to determine the description text of the multimedia data.

[0014] According to a second aspect of an embodiment of the present disclosure, an image generating device is provided, which is used to generate a description image of multimedia data, and the image generating device includes: a receiving unit, configured to receive the multimedia data and user information of a current user, wherein the current user is an audience user of the multimedia data; a description unit, configured to determine a description text of the multimedia data based on the multimedia data and the user information; and a generation unit, configured to generate a description image of the multimedia data based on the description text.

[0015] Optionally, the generation unit is further configured to: receive an image template, the image template comprising multiple preset elements; based on the description text, determine an element description text for each of the multiple preset elements; generate an element image of each preset element according to the element description text of each preset element; and generate a description image of the multimedia data according to the element image of each preset element.

[0016] Optionally, the generation unit is further configured to: determine a dynamic element image and a static element image from the element images of the multiple preset elements; merge the dynamic element image and the static element image into a dynamic layer and a static layer respectively, and merge the dynamic layer and the static layer into a description image of the multimedia data, wherein the dynamic layer can be regenerated.

[0017] Optionally, the generation unit is further configured to: obtain scene information, wherein the scene information is related to the current user and the current user's viewing behavior of the multimedia data; determine the dynamic element image and the static element image from the element images of the multiple preset elements according to the scene information, wherein the dynamic layer can be regenerated when the scene information changes.

[0018] Optionally, the image generating device further includes: a triggering unit, configured to determine, in response to receiving updated scene information, whether to trigger an update scene mode or a reset scene mode according to the updated scene information; an updating unit, configured to, when determining that the update scene mode is triggered, adjust the dynamic element image according to the updated scene information to obtain an updated dynamic element image, fuse the updated dynamic element image into an updated dynamic layer, and merge the updated dynamic layer and the static layer into an updated description image of the multimedia data; the receiving unit, the describing unit and the generating unit are further configured to, when determining that the reset scene mode is triggered, re-execute the step from the step of receiving the multimedia data and the user information of the current user to the step of generating a description image of the multimedia data according to the description text.

[0019] Optionally, the scene information includes at least one of the following: multimedia type, user portrait, camera sensor data, playback progress, and playback mode; wherein the multimedia type includes multimedia sets and independent multimedia, the multimedia set indicates that the multimedia data contains multiple multimedia files, and the independent multimedia indicates that the multimedia data contains only one multimedia file; the user portrait includes at least one of the following: viewing preference, gender, and age; the camera sensor data includes at least one of the following: user identification, viewing distance, background noise, and ambient light; the playback mode includes at least one of the following: children's mode, elderly mode, standard mode, and office mode.

[0020] Optionally, each of the multiple preset elements has element configuration information, the element configuration information includes an element name and an element display area, the element display area is used to represent the display area covered by the corresponding element image in the entire description image, and the generation unit is further configured to: generate a complete image of each preset element according to the element description text and element configuration information of each preset element, wherein the size of the complete image matches the size of the description image; generate an element image of each preset element based on the complete image of each preset element and the element display area.

[0021] Optionally, the generation unit is further configured to: based on the fusion effect of the complete images of the multiple preset elements, adjust the complete image of at least one of the multiple preset elements to obtain a corrected complete image of each of the multiple preset elements; and generate an element image of each preset element based on the corrected complete image of each preset element and the element display area.

[0022] Optionally, the description unit is further configured to process the multimedia data and the user information using a large language model to determine a description text of the multimedia data.

[0023] According to a third aspect of an embodiment of the present disclosure, an electronic device is provided, comprising: at least one processor; and at least one memory storing computer executable instructions, wherein the computer executable instructions, when executed by the at least one processor, prompt the at least one processor to execute an image generation method according to an exemplary embodiment of the present disclosure.

[0024] According to a fourth aspect of an embodiment of the present disclosure, a computer-readable storage medium is provided. When instructions in the computer-readable storage medium are executed by at least one processor, the at least one processor is prompted to execute an image generating method according to an exemplary embodiment of the present disclosure.

[0025] According to a fifth aspect of an embodiment of the present disclosure, a computer program product is provided, comprising computer instructions, which, when executed by at least one processor, prompt the at least one processor to execute an image generating method according to an exemplary embodiment of the present disclosure.

[0026] The technical solution provided by the embodiments of the present disclosure brings at least the following beneficial effects: according to the image generation method and device, electronic device, and storage medium of the present disclosure, by determining a multimedia data description text that matches the current user based on the targeted multimedia data and the user information of the current user, and generating a description image of the multimedia data based on the description text, the description image can be made to be more in line with the viewing needs of the current user, thereby reducing the randomness of the automatically generated images, improving the quality of the automatically generated images, and supporting various types of pages related to multimedia data at a lower operating cost.

[0027] It is to be understood that the foregoing general description and the following detailed description are exemplary and explanatory only and are not restrictive of the present disclosure. BRIEF DESCRIPTION OF THE DRAWINGS

[0028] The drawings herein are incorporated into and constitute a part of the specification, illustrate embodiments consistent with the present disclosure, and together with the description are used to explain the principles of the present disclosure, and do not constitute improper limitations on the present disclosure.

[0029] Figure 1 It is a schematic diagram of a video details page of the related technology.

[0030] Figure 2 is a flowchart of an image generating method according to an exemplary embodiment of the present disclosure.

[0031] Figure 3 is a mask diagram of an image template according to an exemplary embodiment of the present disclosure.

[0032] Figure 4 It is a flowchart of an image generating method according to a specific embodiment of the present disclosure.

[0033] Figure 5 It is a flowchart of determining description text according to a specific embodiment of the present disclosure.

[0034] Figure 6 It is a description image of the TV series "XXX" according to a specific embodiment of the present disclosure.

[0035] Figure 7 is another description image of the TV series "XXX" according to a specific embodiment of the present disclosure.

[0036] Figure 8 It is a flowchart describing text redirection according to a specific embodiment of the present disclosure.

[0037] Fig. 9 It is a flowchart of generating element images according to a specific embodiment of the present disclosure.

[0038] Fig.10It is a schematic diagram of the principle of generating element images according to a specific embodiment of the present disclosure.

[0039] Fig.11 It is a schematic diagram of the execution flow of the scene monitoring module according to a specific embodiment of the present disclosure.

[0040] Fig.12 It is a schematic diagram of a process of fusing element images according to a specific embodiment of the present disclosure.

[0041] Fig.13 It is a description image of the movie "XX" according to a specific embodiment of the present disclosure.

[0042] Fig.14 is another description image of the movie “XX” according to a specific embodiment of the present disclosure.

[0043] Fig.15 It is another description image of the movie "XX" according to a specific embodiment of the present disclosure.

[0044] Fig.16 This is a schematic diagram of a video details page of a TV series "XXX" in a mobile phone according to a specific embodiment of the present disclosure.

[0045] Fig.17 It is a schematic diagram of a video details page of a TV series "XXX" in a tablet computer according to the related art.

[0046] Fig.18 It is a schematic diagram of a video details page of a TV series "XXX" in a tablet computer according to a specific embodiment of the present disclosure.

[0047] Fig.19 It is a page diagram of a mobile phone search desktop widget according to a specific embodiment of the present disclosure.

[0048] Fig. 20 is a block diagram of an image generating apparatus according to an exemplary embodiment of the present disclosure.

[0049] Fig.21 is a block diagram of an electronic device according to an exemplary embodiment of the present disclosure. DETAILED DESCRIPTION

[0050] In order to enable ordinary persons in the art to better understand the technical solutions of the present disclosure, the technical solutions in the embodiments of the present disclosure will be clearly and completely described below in conjunction with the accompanying drawings.

[0051] It should be noted that the terms "first", "second", etc. in the specification and claims of the present disclosure and the above-mentioned drawings are used to distinguish similar objects, and are not necessarily used to describe a specific order or sequence. It should be understood that the data used in this way can be interchanged where appropriate, so that the embodiments of the present disclosure described herein can be implemented in an order other than those illustrated or described herein. The implementation methods described in the following examples do not represent all implementation methods consistent with the present disclosure. Instead, they are merely examples of devices and methods consistent with some aspects of the present disclosure as detailed in the attached claims.

[0052] It should be noted that the phrase "at least one of the items" in the present disclosure includes three types of parallel situations: "any one of the items", "a combination of any number of the items", and "all of the items". For example, "including at least one of A and B" includes the following three parallel situations: (1) including A; (2) including B; (3) including A and B. Another example is "executing at least one of step 1 and step 2" which means the following three parallel situations: (1) executing step 1; (2) executing step 2; (3) executing step 1 and step 2.

[0053] Hereinafter, an image generating method and apparatus, an electronic device, and a storage medium according to exemplary embodiments of the present disclosure will be described in detail with reference to the accompanying drawings.

[0054] Figure 2 It is a flowchart of an image generation method according to an exemplary embodiment of the present disclosure. The image generation method is used to generate a description image of multimedia data, and the generated description image can be applied to the details page of multimedia data, the screen saver page of electronic devices, the multimedia recommendation component page of electronic devices, etc., so as to introduce multimedia data in a richer form on these pages. The image generation method can be implemented in a device with sufficient computing power, such as a mobile phone, a television, and other electronic devices that can play multimedia data, and a server (the server can communicate with the electronic device through a communication network and send the generated description image to the electronic device).

[0055] Reference Figure 2 In step S201, multimedia data and user information of the current user are received.

[0056] Multimedia data is data that needs to generate a description image, such as a TV series, a movie, or a song that can be played. Specifically, it may include the video image, audio, lines, etc., and may also include its promotional posters. The current user is the viewer of the multimedia data, that is, the user who is using the above-mentioned electronic device. For example, the user information can be determined through the user account logged in on the electronic device, or through other reasonable methods. The present disclosure does not limit this. It should be noted that the user information involved in the present disclosure (including but not limited to user device information, user personal information, etc.) is all information authorized by the user or fully authorized by all parties.

[0057] In step S202, a description text of the multimedia data is determined based on the multimedia data and the user information. This step can determine the description text based on natural language processing technology, which will be described in detail later and will not be expanded here. It should be understood that the description text may include at least one form of text in the form of keywords, sentences, and paragraphs, and the present disclosure does not limit this.

[0058] In step S203, a description image of the multimedia data is generated according to the description text. This step can be based on AIGC technology to achieve image generation combined with text conditions.

[0059] According to the image generation method of the exemplary embodiment of the present disclosure, by determining the multimedia data description text matching the current user based on the targeted multimedia data and the user information of the current user, and generating a description image of the multimedia data based on the description text, the description image can be made to better match the viewing needs of the current user, thereby reducing the randomness of the automatically generated image, improving the quality of the automatically generated image, and supporting various types of pages related to multimedia data at a lower operating cost.

[0060] It should be understood that for different execution entities of the image generation method according to the exemplary embodiment of the present disclosure, the transmission of data will be slightly different. For electronic devices that can play multimedia data, multimedia data in a multimedia server and user information in a user management server can be received through a communication network (the multimedia server and the user management server can be the same server or different servers belonging to the same server cluster, or they can be unrelated different servers), and then the description text of the multimedia data is determined and a description image is generated based on this, and the description image can also be displayed on an appropriate page. For a server, a multimedia server or a user management server can be directly borrowed, or it can be a server configured separately. For the former, the multimedia data or user information stored by itself can be used, and multimedia data or user information that it does not have can be received from other servers. For the latter, like an electronic device, it is necessary to receive multimedia data from a multimedia server and user information from a user management server, and then determine the description text of the multimedia data and generate a description image, and the description image can also be sent to the electronic device for the electronic device to display the description image on an appropriate page.

[0061] Next, an image generating method according to an exemplary embodiment of the present disclosure is further introduced.

[0062] Regarding step S203, in some embodiments, optionally, step S203 includes: using the description text as a keyword, generating an image that matches the keyword as a description image of the multimedia data.

[0063] In other embodiments, optionally, step S203 includes: receiving an image template, the image template including multiple preset elements; determining an element description text for each of the multiple preset elements based on the description text; generating an element image of each preset element according to the element description text of each preset element; generating a description image of the multimedia data according to the element image of each preset element. By preparing an image template including multiple preset elements (such as titles, introduction text, and background images) in advance, element images of each preset element can be generated according to the template, and then these element images are combined into a description image, thereby ensuring that the arrangement form of the generated description image is unified, further reducing the randomness of the automatically generated description image, and helping to ensure the accuracy and stability of the automatically generated description image. At the same time, based on the previously obtained description text, determining the corresponding element description text for each preset element can provide clearer and more specific instructions and improve the quality of the description image.

[0064] As an example, when determining the element description text for each of the plurality of preset elements based on the description text, for each preset element, the text suitable for the preset element can be extracted from all the description texts as the element description text of the preset element. It should be understood that this means that when determining the element description text, it is equivalent to redirecting the description text based on the preset element without changing the content of the description text; it also means that the element description texts of different preset elements may overlap, that is, the same text may be reused by different preset elements.

[0065] In some other embodiments described above, further optionally, each of the multiple preset elements has element configuration information, and the element configuration information includes an element name (such as a title, introduction text, background image, etc.) and an element display area, and the element display area is used to represent the display area covered by the corresponding element image in the entire description image, so as to reliably combine the element images of each preset element. Of course, the element configuration information may also include other information to enrich the description of the preset elements, and the present disclosure does not limit this. As an example, a mask may be constructed for each preset element to represent the element display area, Figure 3 An example of a mask of an image template is shown, and the image template includes three preset elements, namely, a title, an introduction text, and a background image. A mask is designed for each preset element. The dark area in the mask is the area that is not filled with an image, and the transparent area is the area that needs to be filled with an image, that is, the element display area. Accordingly, the operation of generating an element image of each preset element according to the element description text of each preset element in step S203 includes: generating a complete image of each preset element according to the element description text and element configuration information of each preset element, wherein the size of the complete image matches the size of the description image (it should be understood that matching means that the size can be exactly the same or in a certain proportion, so that the complete image can be scaled proportionally to be consistent with the size of the description image); generating an element image of each preset element based on the complete image of each preset element and the element display area. By first generating a complete image for the entire display area, and then generating an element image in combination with the element display area, for example but not limited to cropping out a portion of the non-filled area (i.e., the dark area) according to the mask, and retaining only a portion of the filled area (transparent area) as the element image, the complex task of generating element images of different sizes and different display areas can be decomposed into two simple tasks, namely, the task of generating a complete image of uniform size according to the text, and the task of cropping the complete image to obtain the element image, which helps to reduce the difficulty of the task and improve the efficiency of task execution. As an example, Figure 3As shown, the element image of the title will fill a rectangular area in the center of the entire display area, the element image of the introduction text will be divided into several strips and filled in multiple scattered small-sized long strip areas in the display area, and the element image of the background image will fill the entire real area.

[0066] Further optionally, for the above-mentioned operation of generating an element image of each preset element based on the complete image and the element display area of ​​each preset element, in addition to the embodiment of directly referring to the element display area to crop the complete image, the operation can also be implemented as including: adjusting the complete image of at least one preset element among the multiple preset elements based on the fusion effect of the complete images of multiple preset elements to obtain a corrected complete image of each preset element among the multiple preset elements; generating an element image of each preset element based on the corrected complete image and the element display area of ​​each preset element. Considering that the element display areas of different preset elements may overlap, and the complete images of each preset element are generated independently, if the element image is directly cropped and the element images are overlapped in sequence, the overlapping area after fusion may not be displayed clearly. In view of this, by adjusting at least one complete image based on the fusion effect of the complete image (for example, but not limited to taking the content display clarity of each element image as a measure of the fusion effect), and then cropping the element image based on the adjusted corrected complete image, the display effect of the final description image can be guaranteed and the image generation quality can be improved. As an example, the adjustment of the complete image may be, for example, moving the position of part of the image or adjusting the image color of the overlapping area. Of course, it may be other reasonable adjustment methods, and the present disclosure does not limit this. In addition, with respect to correcting the complete image, it should be understood that, for the convenience of unified description, regardless of whether the complete image of a preset element will be adjusted, the complete image ultimately used to generate the element image will be referred to as a corrected complete image, which means that for the preset element whose complete image is not adjusted, its complete image and the corrected complete image are the same image. Of course, when the fusion effect is not good, in addition to adopting the current implementation method of adjusting the complete image to improve the fusion effect, an implementation method of adjusting the element display area of ​​at least one preset element may also be adopted, and other reasonable implementation methods may also be adopted, and these implementation methods may be used in combination without contradiction, and the present disclosure does not limit this.

[0067] In some other embodiments described above, further optionally, for the operation of generating a description image of multimedia data according to the element image of each preset element in step S203, in addition to the embodiment of directly combining several element images together, the operation can also be implemented as: determining a dynamic element image and a static element image from the element images of multiple preset elements; fusing the dynamic element image and the static element image into a dynamic layer and a static layer respectively, and merging the dynamic layer and the static layer into a description image of multimedia data, wherein the dynamic layer can be regenerated. By distinguishing multiple element images into dynamic element images that can be regenerated and static element images that remain unchanged, and fusing them into dynamic layers and static layers respectively, a new description image can be generated by changing the dynamic layer, that is, changing at least part of the dynamic element image when the demand changes, which can not only reserve space for the description image to change and improve its flexibility, but also reduce the amount of calculation required to generate a new description image, and can reduce the randomness of image generation that follows, and improve the stability of the description image with the help of the static layer, thereby taking into account both the flexibility and stability of the description image generation. As an example, when redirecting the description text based on the preset element, each preset element may be redirected to obtain a large amount of element description text. When generating an element image based on the element description text, especially when generating a dynamic element image, it is not necessary to use all the element description texts of a preset element to generate the element image of the preset element. Instead, at least one element description text of the preset element can be extracted to generate the element image of the preset element. When the demand changes and the dynamic layer needs to be changed, for at least one dynamic element image that needs to be changed, at least one element description text that meets the demand can be reselected from all the element description texts of the preset element corresponding to the dynamic element image, and the element image can be regenerated accordingly as an updated dynamic element image to replace the original dynamic element image of the preset element. The dynamic element image can also be formally adjusted without changing the substantive content, so there is no need to repeat the operations of first determining the description text, then determining the element description text, and then generating the element image, which can improve the efficiency of updating the description image. It should be understood that for the aforementioned embodiment of first generating a complete image, then adjusting it to obtain a corrected complete image, and then cropping it to obtain an element image, a new complete image can be first generated based on the reselected element description text, and then based on the fusion effect of the new complete image and the complete images corresponding to other dynamic element images that do not need to be changed, at least one of these complete images is adjusted (as an example, at this time only the new complete image can be adjusted, which can keep other dynamic element images unchanged) to obtain a new corrected complete image, which is then cropped to obtain a new dynamic element image.

[0068] Regarding the operation of determining the dynamic element image and the static element image from the element images of multiple preset elements, further optionally, some preset elements can be marked as dynamic elements in advance, and other preset elements can be marked as static elements. Accordingly, at this time, the operation includes: determining the element image of the dynamic element as the dynamic element image, and determining the element image of the static element as the static element image.

[0069] The operation may also include: obtaining scene information, wherein the scene information is related to the current user and the current user's viewing behavior of multimedia data; determining a dynamic element image and a static element image from the element images of multiple preset elements according to the scene information, wherein the dynamic layer can be regenerated when the scene information changes. By introducing specific scene information, it is possible to more flexibly determine the dynamic element image and the static element image according to the actual image generation requirements in different scenes, so as to more efficiently update the description image when the requirements change. As an example, at this time, the previous embodiment may also be referred to, and the dynamic element and the static element are first determined, and then the element image of the dynamic element is determined as the dynamic element image, and the element image of the static element is determined as its static element image. The difference from the previous embodiment at this time is that the dynamic element and the static element are determined from the multiple preset elements according to the scene information, and the previous embodiment determines the dynamic element and the static element in advance, which is independent of the scene information. It should be understood that for this embodiment, under different scene information, the same preset element may be determined as a dynamic element or a static element.

[0070] Regarding the embodiment of determining the dynamic / static element image in combination with the scene information, further optionally, the image generation method according to the exemplary embodiment of the present disclosure also includes: in response to receiving the updated scene information, determining whether to trigger the update scene mode or reset the scene mode according to the updated scene information; in the case of determining that the update scene mode is triggered, adjusting the dynamic element image according to the updated scene information to obtain an updated dynamic element image, fusing the updated dynamic element image into an updated dynamic layer, and merging the updated dynamic layer and the static layer into an updated description image of the multimedia data. It should be understood that for the sake of ease of expression, all dynamic element images obtained after triggering the update scene mode are referred to as updated dynamic element images, but in fact, only part of the dynamic element images can be adjusted according to the scene requirements; in the case of determining that the reset scene mode is triggered, re-execute steps S201 to S203, that is, from the step of receiving the multimedia data and the user information of the current user to the step of generating a description image of the multimedia data according to the description text. By configuring the update scene mode and the reset scene mode, when the scene information is updated, it can be first determined whether the update of the description image can be achieved by changing the dynamic layer, and then the update scene mode is triggered when it is determined to be possible, and only the dynamic layer is updated. When it is determined that it is not possible, the reset scene mode is triggered to regenerate the entire description image, so as to achieve the on-demand update of the description image, which helps to improve the update efficiency. As an example, when determining whether to trigger the update scene mode or the reset scene mode, several prior rules can be preset in advance, such as the rule for triggering the update scene mode (if the prior rule is not satisfied at this time, the reset scene mode is triggered), and the rule for triggering the reset scene mode (if the prior rule is not satisfied at this time, the update scene mode is triggered), so as to determine which mode is triggered according to whether the updated scene information satisfies the prior rule. It is also possible to use a deep learning model trained in advance, input the updated scene information into the deep learning model, and the model determines which mode is triggered. For the latter, supervised training can be used, and the training samples used at this time can include multiple scene information and mark the triggered mode; unsupervised training can also be used, such as using a reinforcement learning model to train the model through feedback reward signals or penalty signals. Of course, other methods may also be used to determine which mode is triggered, and the present disclosure does not limit this. As an example, when triggering the update scene mode, the aforementioned method of reselecting at least one element description text that meets the requirements when the requirements change, and regenerating the element image accordingly, as an implementation method for updating the dynamic element image, at this time, at least one element description text may be reselected specifically according to the updated scene information. Of course, as mentioned above, the dynamic element image may also be formally adjusted without regenerating the element image while the substantive content remains unchanged.In addition, other implementations may also be adopted, such as organizing all element description texts of each preset element into multiple text groups in advance, each text group is used to generate an element image, and the same element description text can be reused in different text groups as needed, and when the element image is regenerated, only a text group needs to be reselected, and for example, a corresponding element image can be generated for each text group in advance, and when the element image of the preset element needs to be updated, only an element image corresponding to a text group needs to be reselected. The present disclosure is not limited to this.

[0071] As an example, the scene information includes at least one of the following: multimedia type, user portrait, camera sensor data, playback progress, and playback mode. By configuring rich scene information, it is possible to select appropriate scene information as needed in practice, reasonably describe the entire scene, and meet different description requirements in different situations.

[0072] Specifically, multimedia types include multimedia sets and independent multimedia. A multimedia set indicates that multimedia data includes multiple multimedia files, and an independent multimedia indicates that multimedia data includes only one multimedia file. For multimedia data of the multimedia set type, users often have the need to switch between different multimedia files. At this time, the element image related to the specific multimedia file can be used as a dynamic element image, and the element image introducing the entire multimedia set can be used as a static element image; for multimedia data of the independent multimedia type, static element images can be used more, thereby providing a reference for the determination of dynamic / static element images. It should be understood that in the case of updating scene information, if the updated scene information is that the multimedia data of the multimedia set type switches the multimedia file, the update scene mode can be triggered; if the updated scene information is the switching of the multimedia data, the reset scene mode can be triggered. It should also be understood that for multimedia data of the multimedia set type, when determining the description text, a large amount of description text will be determined for the entire multimedia data, so there must be description texts for different multimedia files. In this regard, when generating the element image for the first time, for the preset element (such as the introduction text) that is closely related to the multimedia file, the description text corresponding to the currently located multimedia file can be specifically selected as the element description text of the preset element to generate the corresponding element image.

[0073] The user portrait includes at least one of the following: viewing preference, gender, and age. The user portrait can reflect the content that the user is interested in. When the scene information contains the user portrait, the dynamic / static element image can be determined in a personalized manner in combination with the analysis of the user portrait, and a reference can be provided for the update of the description image to enhance the flexibility of image generation. As an example, the user portrait can be used as part of the user information to participate in the determination of the description text of the multimedia data, and therefore should also be information authorized by the user or fully authorized by all parties. At the same time, if it is found that the user has a wide range of hobbies and has a variety of different preference styles in combination with the user's viewing history, the description text can be generated for different preference styles, and the preset elements related to these description texts can be used as dynamic elements, and the element description text used for the dynamic element is determined based on the style of the multimedia content recently watched, and then a dynamic element image is generated, and then a description image is generated. On this basis, if the user portrait indicates that the style of the multimedia content recently watched by the user has changed, and the range of the change is still within the previously determined multiple styles, the update scene mode can be triggered, the element description text used is re-determined according to the new multimedia content style, and the corresponding updated dynamic element image is generated; if the user portrait indicates that the style of the multimedia content recently watched by the user exceeds the previously determined multiple styles, the reset scene mode can be triggered.

[0074] The camera sensor data includes at least one of the following: user identification, viewing distance, background noise, and ambient light. Since the camera sensor can capture the actual viewing state of the user, it can provide rich reference information for determining and updating the dynamic / static element image. Specifically, user identification can identify a specific user to understand the actual viewing user, which can be used as a reference for determining the user information in step S201, and when it is recognized that the actual viewing user has changed, it can trigger the reset of the scene mode. The viewing distance is the distance between the user and the electronic device, which can be used as a basis for adjusting the text size in the description image (the closer the distance, the smaller the text). Therefore, when the scene information includes the viewing distance, the preset elements containing text (such as titles, introduction text) can be used as dynamic elements, and when it is recognized that the viewing distance has changed, the scene mode is updated to adjust the text size of the dynamic element images corresponding to these preset elements. Background noise and ambient light can also be included as references for generating and adjusting the description image because they can affect the user's viewing needs. It should be understood that the camera sensor data used here, like the user information, should be information authorized by the user or fully authorized by all parties.

[0075] The playback progress indicates the stage of the currently targeted multimedia content in the entire multimedia data. Since the entire multimedia data often contains rich content, the playback progress can more accurately represent the current specific content, thereby providing a more targeted description. Specifically, at this time, the element image that introduces the specific playback progress can be used as a dynamic element image, and the element image that introduces the entire multimedia data can be used as a static element image. When the playback progress changes but has not yet ended, the update scene mode can be triggered to switch to a different dynamic element image. It should be understood that, similar to the case of multimedia data involving multimedia set types, for the case involving playback progress, when determining the description text, a large amount of description text will be determined for the entire multimedia data, so there must be description texts for different playback progress. In this regard, when generating the element image for the first time, for the preset element (such as the introduction text) that is closely related to the playback progress, the description text corresponding to the currently located playback progress can be specifically selected as the element description text of the preset element to generate the corresponding element image.

[0076] The playback mode includes at least one of the following: children's mode, elderly mode, standard mode, and office mode. When an electronic device is configured with different playback modes, since different playback modes can reflect different demands, they can also be used as a reference for determining the description image. As an example, for the children's mode, the use of the preset element of the introduction text can be canceled; for the elderly mode, the number of characters in the preset element of the introduction text can be reduced, and the text size of the introduction text and the title can be jumped; different playback modes can also use background images of different styles. In the case where the scene information includes the playback mode, it can be determined which preset elements will change when switching the playback mode in combination with the specific playback modes supported by the current electronic device, thereby determining the static elements and dynamic elements, and triggering the update of the scene mode when the playback mode changes.

[0077] In addition, regarding step S202, in some embodiments, optionally, step S202 includes: using a large language model to process multimedia data and user information, and determining a description text of the multimedia data. By processing the multimedia data and user information with the help of the large language model's processing ability and language processing ability of multimedia data, it is possible to obtain a description text that is both in line with the actual content of the multimedia data and fully matches the user's viewing needs, which helps to improve the effectiveness of the description text. Specifically, a multimodal large language model can be used, so that it can have the ability to process multimedia data. As an example, the multimedia data and user information can be simultaneously input into the large language model to directly obtain the description text; the multimedia data can also be first input into the large language model alone to obtain an objective descriptive text that is not related to the user, and then the objective descriptive text and the user information are input into the large language model together to obtain the description text. Specifically, prompt words can be used to combine multimedia data (or its objective descriptive text) and user information, so as to provide clear principles and directions for the generation of description texts with the help of detailed prompt words. For example, “Please extract the description text of the TV series to be input from the personal perspective of user A. The information of user A is as follows:…”, “…” represents the user information, and then the video data of the corresponding TV series is input into the large language model.

[0078] In other embodiments, optionally, step S202 may first use traditional natural language processing technology to extract text data from multimedia data, such as data directly in text form, data obtained by voice recognition of audio, and subtitle data extracted from images, and then use semantic analysis to filter out text that matches user information from the text data, and use these filtered texts as description texts to generate description images.

[0079] Next, an image generating method according to a specific embodiment of the present disclosure is introduced.

[0080] This specific embodiment is used to generate a description image of video data, and can be applied to a video detail page, a TV screen saver page, a video recommendation component page, etc. It should be noted in advance that in order to give a specific example, this specific embodiment will list some detailed processing flows, such as how to generate a description text, how to generate an element image, etc. These detailed processing flows can also be implemented using other processing flows in the art.

[0081] Reference Figure 4 In this specific embodiment, the image generation method mainly includes four steps from Step 1 to Step 4, which will be introduced one by one below.

[0082] Step 1: Input preparation.

[0083] The purpose of this step is to obtain two kinds of information, one is the description text (corresponding to Figure 2 Step S201, step S202), on the other hand, the mask (used to represent the element display area mentioned above, corresponding to the image template, from this part to Step 3, corresponding to Figure 2 .

[0084] The input of this step includes: video data (including video image frames, audio, posters, lines, etc.), user information, and image templates, which serve as information input sources.

[0085] For the description text, a large language model (LLM) is used to analyze the input video data and user information, and the large language model outputs a large number of phrases as the description text.

[0086] For example Figure 5 As shown, the video image frame, audio, user information and other contents are input into the encoder to obtain multiple N-dimensional vectors. It should be understood that Figure 5 Two encoding paths are drawn in the figure, which is only an example. In practice, each information will be encoded separately. After that, the obtained encoding vector (usually an embedding vector embedding) is input into the feature extractor Q-Former, which can extract a feature vector suitable for a large language model (LLM). In addition, when inputting the encoding vector, due to the large amount of information in the video image frame and audio, the encoding vectors of the two can be input in segments, and the encoding vectors of the video image frame and audio frame corresponding in time can be input at the same time, so that the LLM can understand the time information therein and finally output the description text.

[0087] For masks, the description image to be generated can be decomposed into multiple masks based on multiple preset elements in the preset template. Each mask corresponds to a preset element, and the mask is pushed to the next step, i.e., Step 2, to redirect the description text. The individual processing of each preset element is equivalent to the individual control of the variable, which greatly reduces the randomness of the description image generation and can achieve more stable and accurate image expression. Specifically, when the scene changes later, only the element images corresponding to some preset elements and the layers composed of these element images need to be regenerated, which will not affect the other layers of the entire description image, thereby maintaining the stability of the description image and the page to which the description image is applied. The individual generation of the element image corresponding to each preset element can provide more accurate generation requirements for key information in the description image.

[0088] For example, Figure 6As shown in the figure, it is a description image of a TV series "XXX" in the first episode. When the user moves the selection cursor to the second episode, he only needs to change the introduction text about the video (that is, from the introduction text "AAAAA", "BBBBBB", etc. related to the first episode to the introduction text "aaaaa", "bbbbb", etc. related to the second episode). At this time, if the image template is not used, the entire description image needs to be regenerated, which is too random. The position and style of the background image and the title (that is, the name of the TV series XXX) may change. If, for example, the image template is used, Figure 3 The image template shown in the figure will only change the introduction text, resulting in Figure 7 The updated description image shown maintains the stability of the description image and page.

[0089] Step 2: Element image generation.

[0090] The purpose of this step is also to obtain two pieces of information, one is the element description text of each preset element, and the other is the element image of each preset element.

[0091] For the element description text, the description text determined in Step 1 is redirected based on the mask pushed in Step 1. For example Figure 8 As shown, assuming that the number of description texts is W, each description text can be encoded separately to obtain the encoding vector T of each description text 1 , T 2 ,……,T W At the same time, assuming that the image template includes M preset elements, the text of the element configuration information of each preset element (which may include element names such as title, introduction text, background image, etc., and may also include a description of the mask form of the preset element, such as the position and size of the filling area, etc.) can be encoded respectively to obtain the encoding vector I of each preset element. 1 ,I 2 ,……,I M . The dot product is calculated for the encoding vectors of each preset element and the encoding vectors of each description text one by one, and then the following can be used: Figure 8 Each dot product in the matrix shown is used to represent the correlation between the corresponding description text and the preset element, and then according to the value of each dot product, several dot products are selected as the redirection results, for example Figure 8 The I marked 1 ·T 1 ,I 2 ·T 2 ,I M ·T W, respectively, means that the first description text is used as the element description text of the first preset element, the second description text is used as the element description text of the second preset element, and the Wth description text is used as the element description text of the Mth preset element. Of course, the markings here are only for illustration and do not represent the actual redirection results. Moreover, since the number of description texts W is usually much larger than the number of preset elements M, each preset element can often get multiple element description texts. The specific rules for determining the element description text can be set as needed. For example, for Figure 8 The matrix shown can be compared column by column, and the one with the largest value in the same column can be marked as a redirection result. In this case, each description text can only be redirected to a preset element. A threshold can also be set to mark all dot products greater than the threshold. In this case, the number of preset elements to which each description text is redirected is uncertain. At the same time, it can be further required that when there is no dot product greater than the threshold in the same column, the dot product with the largest value should be marked to ensure that each description text can be redirected.

[0092] like Figure 4 As shown, after redirection, m element description texts can be obtained (i.e., element description text 1, element description text 2, ..., element description text m), and each element description text has a corresponding preset element. Therefore, even if two element description texts come from the same description text, they should be regarded as different element description texts, used to generate element images of different preset elements.

[0093] For element images, based on at least one element description text of each preset element after redirection and the corresponding mask (correspondingly including mask 1, mask 2, ..., mask M), the powerful function of AIGC can be used to generate element images of each preset element (correspondingly including element image 1, element image 2, ..., element image M).

[0094] For example Fig. 9As shown in the figure, the AIGC function is implemented using Text conditional Latent Unet, which is a generative model that combines text conditions and U-Net architecture, and is usually used for image generation, image restoration or other image-related tasks. Specifically, the model has two inputs. On the one hand, the element description text is encoded using the frozen CLIP text encoder (FrozenCLIP (Contrastive Language-Image Pretraining) Text Encoder) to obtain the text embedding vector (Text Embedding). On the other hand, the initial element image is obtained based on the mask and the initial noise (AddNoise), where the noise represents the complete image for the entire display area, and the initial noise is the initial value of the complete image. It is cropped based on the mask, and only the part of the fill area (i.e., the transparent area) defined by the mask is retained, which is the initial element image, which can be expressed as Latent*Mask, that is, the latent variable (Latent) corresponding to the initial noise and the mask (Mask) are calculated. It can be seen that the input model here is specifically the encoding vector of the initial element image, not the image data. After these two aspects are input into the model, the model will output a predicted noise, specifically Gaussian noise, and pass it to the DDIM (Denoising Diffusion Implicit Models) scheduler (DDIMScheduler), which is a scheduler for generating models. It is mainly used in the training and reasoning process of diffusion models. It can improve the generation efficiency and quality by optimizing the sampling process. DDIM then returns the processed noise to the model, and the cycle is repeated to achieve iterative calculation of noise. When the end condition is reached (usually the set number of iterations), the element description text and the element image corresponding to the mask are decoded by the VAE decoder (Variational Autoencoder Decoder).

[0095] Still Figure 6 For example, refer to the description image shown in Fig.10 , you can use Fig. 9 The process shown uses the elements of the background image to describe text and mask to generate the element image of the background image, uses the elements of the title to describe text and mask to generate the element image of the title, and uses the elements of the introduction text to describe text and mask to generate the element image of the introduction text.

[0096] Step 3: AIGC fusion.

[0097] This part also contains two aspects.

[0098] On the one hand, the fusion rules are generated, which are determined by the scene monitoring module according to the scene information. The fusion rules specifically indicate whether each element image is dynamic or static. Specifically, the scene information includes multiple scene texts, such as video type, user portrait, camera sensor data, playback progress, playback mode, etc. Among them, the video type specifically includes video collections and independent videos. The former contains multiple video files, such as TV series, constantly updated variety shows, and documentary series, and the latter only contains one video file, such as movies and games. User portraits can distinguish different audiences, such as viewing preferences, gender, and age. Camera sensor data may include user identification, viewing distance, background noise, and ambient light. The playback progress can be represented by the relevant plots of the video. For video collections, it can specifically be the main plot of a video file to be played. The playback mode may include children's mode, elderly mode, standard mode, and office mode.

[0099] like Fig.11 As shown, the scene monitoring module first encodes these scene texts into scene embedding vectors, and then inputs them into the pre-trained monitoring model. The pre-trained monitoring model intelligently determines the fusion rules, that is, intelligently determines whether each element image is a static element image or a dynamic element image. If it is determined to be a static element image, a static label 0 is output, and if it is determined to be a dynamic element image, a dynamic label 1 is output. For the pre-trained monitoring model, there is no restriction on which specific model to use. The training samples used in pre-training can contain scene information and labels for several sample element images.

[0100] Another aspect is to fuse element images into a description image. Fig.12 As shown, for all element images generated by AIGC, they are essentially dynamic complete images / static complete images (dynamic complete images are complete images corresponding to dynamic element images, i.e. Fig. 9 The image corresponding to the noise in the image, and the static complete image is the same) is combined with the corresponding mask. When fusion is performed, in order to ensure the fusion effect, it is not simply overlapping the element images in order, but also necessary to further fine-tune each element image. For example, if there is overlap in the display area of ​​different element images, the overlapping area will not be displayed clearly after fusion. In this case, the corresponding element image can be fine-tuned, which can be to move the position of part of the image or adjust the image color of the overlapping area. To achieve such fine-tuning, such as Fig.12As shown, a dynamic or static complete image can be input into the encoder first to obtain a hidden variable, and the hidden variable is input into the control network (Controlnet, used to adjust the specific content of the image, suitable for adjustments that are not convenient to describe in words). At the same time, some adjustments that can be described in words are input into the Unet network in the form of prompt words, so that the complete image is adjusted according to the prompt words. The corrected complete image obtained after adjustment is combined with the corresponding mask to obtain the adjusted element image. In other words, the adjustment of the element image can be essentially understood as the adjustment of the complete image. In addition, the position of the element display area, that is, the position of the fill area (transparent area) of the mask, can also be moved. Thereafter, the adjusted dynamic element images are superimposed together and merged into a dynamic layer, and the adjusted static element images are superimposed together and merged into a static layer. Finally, the dynamic layer and the static layer are merged together as a description image of the video data. Such a merger retains the independence of the dynamic layer and ensures the flexibility of the dynamic layer to be regenerated.

[0101] Take the video details page of a TV series as an example. Assume that Figure 3 In the image template shown, the scene information only includes the video type. Since the current TV series belongs to a video set, the title and background image are determined as static elements, the introduction text is determined as a dynamic element, and the element image of the title and the element image of the background image are used as static element images and merged into a static layer. Figure 6 The element image of the introduction text of Episode 1 shown in the figure is used as a dynamic element image, and a dynamic layer is generated.

[0102] Step 4: Scene triggering.

[0103] This step is implemented by a trigger combined with a scene monitoring module. When the updated scene information is received, the scene monitoring module processes the updated scene information to obtain a new fusion rule. If the fusion rule changes, and this change can be achieved by modifying the dynamic layer (for example, regenerating or adjusting one or some dynamic element images), the update scene mode is triggered. According to the new fusion rule, at least one element description text is re-determined from all the element description texts of a dynamic element, and a new element image of the dynamic element is generated accordingly, or one or some current dynamic element images are fine-tuned as new dynamic element images, and then the new dynamic element images are used to replace the corresponding dynamic element images in the current dynamic layer, and the dynamic layer and description image are regenerated. If the fusion rule changes, and this change cannot be achieved by modifying the dynamic layer, and the entire description image needs to be regenerated, the reset scene mode is triggered and returns to Step 1.

[0104] Still like Figure 6Take the description image of the first episode of the TV series "XXX" shown in the figure as an example. If the user changes the specific episode in the current TV series (for example, from the first episode to the second episode), the introduction text of the video will change due to the episode switching. At this time, the update scene mode is triggered, the fusion rule is updated, and the element description text related to the second episode in the element description text of the introduction text is re-extracted. Based on this, the element image corresponding to the second episode is generated, the dynamic layer is regenerated, and it is merged with the original static layer to obtain the following: Figure 7 The description image of the TV series episode 2 is shown. If the user switches to another TV series, the scene reset mode is triggered, and the whole process is re-executed based on the newly switched TV series to generate a new description image.

[0105] This specific embodiment implements the redirection of the video analysis content (i.e., description text) and the re-characterization of the preset elements in the image template (i.e., whether they are dynamic elements or static elements) by analyzing the content of the video itself (LLM) in combination with objective scene requirements (user information, image template, scene information, etc.). The redirection of the video analysis content determines the generation of the content describing the image, and the re-characterization of the preset elements determines whether the layer is relatively dynamic or static. In principle, it is a method of controlling variables, the purpose of which is to generate images that meet the requirements and greatly reduce the randomness of AIGC products. In other words, while ensuring the accuracy and stability of AIGC products, its flexibility is maintained.

[0106] Next, several examples of applying the image generation method of this specific embodiment to generate and update description images of videos are introduced.

[0107] Example 1:

[0108] Applied to the video details page, using Figure 3 The image template shown, the scene information includes video type, user identification in the camera sensor data, and playback progress. The video on the current page is a movie called "XX" that contains science fiction elements. Although the video type of the movie is an independent video, the playback progress can affect the introduction text, so the title and background image are determined as static elements, and the introduction text is determined as a dynamic element.

[0109] For the identified user A, his user information includes a user portrait, specifically: female, prefers romantic movies, and the current playback progress is not yet played. Based on the user information and the video data of the movie, a background image element image containing a female portrait, an introduction text element image containing descriptive words such as "female protagonist" and "xxx", and a movie title element image are generated, and then the following is generated: Fig.13 Description image shown.

[0110] When the camera sensor data recognizes that the user switches to user B, the scene mode is reset. The user information of user B specifically includes: male, technology geek, and preference for science fiction movies. Based on the user information of user B and the video data of the movie, the background image element image containing the starry sky, the introduction text element image containing several descriptive words related to science fiction and technology such as "yyy", and the movie title element image are regenerated, and then the following is generated: Fig.14 Updated description image shown.

[0111] Example 2:

[0112] Applied to the video details page, using Figure 3 In the image template shown, the scene information includes the video type and the viewing distance in the camera sensor data. The video on the current page is still the movie named "XX". Since the video type of the movie is an independent video, the actual content of the three preset elements will not change. However, since the scene information includes the viewing distance, the text size may need to be adjusted due to changes in the viewing distance. Therefore, the title and introduction text are determined as static elements, and the background image is determined as a dynamic element.

[0113] For the aforementioned user B, the following Fig.14 When the viewing distance increases, the update scene mode is triggered to increase the text size in the element images of the title and introduction text, generating the following Fig.15 Updated description image shown.

[0114] Example 3:

[0115] Applied to the video details page of mobile phones. The TV series video details page of existing mobile phones is usually as follows Figure 1 As shown in the figure, only the video window (i.e., the rectangular window at the top of the page), title (i.e., YYY in the figure), selection list, advertisement window, and surrounding videos are simply arranged. In this example, an image template matching the video details page of a mobile phone is used. In addition to the title, introduction text, background, Figure 3 In addition to the preset elements, the image template also includes the preset element of the leading actor, which is used to replace the existing ad window. The title, background image and leading actor are set as static elements in advance, and the introduction text is set as a dynamic element, which changes with the switching of episodes; in addition, the leading actor can also be set as a dynamic element to display the actor of the role appearing in the current episode when switching episodes. The image template also configures the masks corresponding to each preset element to reasonably layout the element images of these preset elements, and adaptively deletes the Figure 1 The original title and ad window in the final generated Fig.16 The new video details page shown (the video details page is as follows Figure 6 and Figure 7The TV series "XXX" shown in the video details page on the mobile phone).

[0116] Example 4:

[0117] The video details page for tablet computers. The TV series video details page for existing tablet computers is usually as follows: Fig.17 As shown, only the video windows, titles and selection lists are simply arranged. In this example, an image template that matches the video details page of a tablet computer is used. This image template is the same as Example 3, including four preset elements: title, introduction text, background image, and starring actors. The title, background image and starring actors are set as static elements in advance, and the introduction text is set as a dynamic element. This image template also configures masks corresponding to each preset element, and this mask is different from the mask in Example 3 to adapt to the screen of a tablet computer, reasonably layout the element images of these preset elements, and adaptively delete the Fig.17 The original title was adapted to the Fig.17 The size and layout of the video window and the selection list in the final result is as follows Fig.18 The new video details page shown (the video details page is as follows Figure 6 and Figure 7 The video details page of the TV series "XXX" shown on a tablet).

[0118] Example 5:

[0119] Video recommendation widgets for mobile desktops. Mobile phones often come with some desktop widgets. For video recommendation widgets, a description image can be generated based on the recommended videos and the user's preferred style, and then displayed. Fig.19 As shown in FIG. 1 , the description image of the video recommendation widget on the mobile phone desktop can be used, and a variety of different image templates can be used to generate different layouts to obtain description images of different layouts for users to choose as needed. Fig.19 As shown, the image templates used to generate these description images may have preset elements including a background image, and may include at least one of a recommendation reason and a video poster, and the specific mask may be set reasonably.

[0120] In addition, such description images can also be applied to TV screensavers, and other image templates can be selected according to the size of the TV screensaver.

[0121] In general, according to the image generation method of the specific embodiment of the present disclosure, the content of the video itself is used as the input source, and based on the image template and the user viewing scene detection, a high-quality video detail page is generated intelligently without manual work. The layered processing method (static layer and dynamic layer) is adopted, which is a controlled variable method in principle, and its purpose is to reduce the randomness of the AIGC generated product. In other words, it ensures the accuracy and stability of the description image generated by AIGC while maintaining its flexibility. Specifically, the LLM technology is used to analyze the video content and generate the description text as the most important content input source of AIGC. Combined with user information, image template and other information, the description text of the video is redirected to generate element images of different preset elements. The element image is fused into a dynamic layer and a static layer according to the scene monitoring. When the scene changes, the dynamic layer can be regenerated separately. The independent generation process of the dynamic layer ensures the flexibility of the page content, and the static layer ensures the stability of the page. The image generation method can be applied to any scene where the page is self-generated with an image template and a certain content theme.

[0122] The benefits of this image generation method for developers are that, first, it can reduce the randomness of AIGC generation. Specifically, the description text generated by LLM based on the video ensures the objectivity and richness of the input source. Layered image generation improves the accuracy, flexibility and stability of the visual expression of the page. Among them, the static layer ensures the stability of the page, and the independent generation process of the dynamic layer ensures the flexibility of the page content. Second, it can meet the adaptation of massive scenes. By using different image templates, this method can be applied to various devices that need to generate description images. Third, high quality and low labor operation costs. Specifically, compared with ordinary video detail pages, this method greatly improves visual effects and content delivery. Compared with the situation where visual effects are emphasized and images are manually operated, this method can ensure the objective matching of video content and generated image styles, achieve high visual effects, and can be generated with almost no labor costs.

[0123] The benefit of this image generation method to users lies in a customized and smooth experience. Specifically, on the one hand, compared with the current TV that requires jumping to a third-party application to open the relevant video detail page, the built-in video detail page of the TV will provide a more simplified TV viewing experience. On the other hand, user preferences, TV viewing environment, etc. will affect the presentation of the page and adjust it to the most ideal display state.

[0124] Fig. 20 is a block diagram of an image generating device according to an exemplary embodiment of the present disclosure. The image generating device is used to generate a description image of multimedia data. Fig. 20 The device includes a receiving unit 2001, a describing unit 2002, and a generating unit 2003.

[0125] The receiving unit 2001 is configured to receive multimedia data and user information of a current user, wherein the current user is a viewer user of the multimedia data.

[0126] The description unit 2002 is configured to determine a description text of the multimedia data according to the multimedia data and the user information.

[0127] The generating unit 2003 is configured to generate a description image of the multimedia data according to the description text.

[0128] Optionally, the generation unit 2003 is also configured to: receive an image template, the image template including multiple preset elements; based on the description text, determine an element description text for each of the multiple preset elements; based on the element description text of each preset element, generate an element image of each preset element; based on the element image of each preset element, generate a description image of multimedia data.

[0129] Optionally, the generation unit 2003 is also configured to: determine a dynamic element image and a static element image from element images of multiple preset elements; merge the dynamic element image and the static element image into a dynamic layer and a static layer respectively, and merge the dynamic layer and the static layer into a description image of multimedia data, wherein the dynamic layer can be regenerated.

[0130] Optionally, the generation unit 2003 is also configured to: obtain scene information, wherein the scene information is related to the current user and the current user's viewing behavior of multimedia data; determine dynamic element images and static element images from element images of multiple preset elements based on the scene information, wherein the dynamic layer can be regenerated when the scene information changes.

[0131] Optionally, the image generation device further includes a trigger unit and an update unit (neither of which is shown in the figure). The trigger unit is configured to determine whether to trigger the update scene mode or reset the scene mode according to the updated scene information in response to receiving the updated scene information; the update unit is configured to adjust the dynamic element image according to the updated scene information in the case of determining that the update scene mode is triggered, obtain the updated dynamic element image, merge the updated dynamic element image into an updated dynamic layer, and merge the updated dynamic layer and the static layer into an updated description image of the multimedia data; the receiving unit 2001, the description unit 2002 and the generation unit 2003 are also configured to re-execute the steps from receiving the multimedia data and the user information of the current user to generating the description image of the multimedia data according to the description text in the case of determining that the reset scene mode is triggered.

[0132] Optionally, the scene information includes at least one of the following: multimedia type, user portrait, camera sensor data, playback progress, and playback mode; wherein, the multimedia type includes multimedia sets and independent multimedia, the multimedia set indicates that the multimedia data contains multiple multimedia files, and the independent multimedia indicates that the multimedia data contains only one multimedia file; the user portrait includes at least one of the following: viewing preference, gender, and age; the camera sensor data includes at least one of the following: user identification, viewing distance, background noise, and ambient light; the playback mode includes at least one of the following: children's mode, elderly mode, standard mode, and office mode.

[0133] Optionally, each of the multiple preset elements has element configuration information, the element configuration information includes an element name and an element display area, the element display area is used to represent the display area covered by the corresponding element image in the entire description image, and the generation unit 2003 can also: generate a complete image of each preset element according to the element description text and element configuration information of each preset element, wherein the size of the complete image matches the size of the description image; generate an element image of each preset element based on the complete image of each preset element and the element display area.

[0134] Optionally, the generation unit 2003 may also: based on the fusion effect of the complete images of multiple preset elements, adjust the complete image of at least one preset element among the multiple preset elements to obtain a corrected complete image of each preset element among the multiple preset elements; based on the corrected complete image of each preset element and the element display area, generate an element image of each preset element.

[0135] Optionally, the description unit 2002 is further configured to process the multimedia data and user information using a large language model to determine a description text of the multimedia data.

[0136] Fig.21 A structural block diagram of an electronic device according to an exemplary embodiment of the present disclosure is shown.

[0137] Reference Fig.21 The electronic device includes: at least one memory 2101 and at least one processor 2102, wherein the at least one memory 2101 stores computer executable instructions, and when the computer executable instructions are executed by the at least one processor 2102, the at least one processor is prompted to execute the image generation method as described in the above exemplary embodiment.

[0138] As an example, the electronic device may be a PC, a tablet device, a personal digital assistant, a smart phone, or other device capable of executing the above instruction set. Here, the electronic device is not necessarily a single electronic device, but may also be any device or circuit capable of executing the above instruction (or instruction set) individually or in combination. The electronic device may also be part of an integrated control system or system manager, or may be configured as a portable electronic device interconnected with a local or remote (e.g., via wireless transmission) interface.

[0139] In an electronic device, the processor 2102 may include a central processing unit (CPU), a graphics processing unit (GPU), a programmable logic device, a dedicated processor system, a microcontroller, or a microprocessor. As an example and not limitation, the processor 2102 may also include an analog processor, a digital processor, a microprocessor, a multi-core processor, a processor array, a network processor, etc.

[0140] The processor 2102 may execute instructions or codes stored in the memory 2101, wherein the memory 2101 may also store data. Instructions and data may also be sent and received over a network via a network interface device, wherein the network interface device may employ any known transmission protocol.

[0141] The memory 2101 may be integrated with the processor 2102, for example, by placing RAM or flash memory within an integrated circuit microprocessor or the like. In addition, the memory 2101 may include a separate device, such as an external disk drive, a storage array, or any other storage device that can be used by a database system. The memory 2101 and the processor 2102 may be operatively coupled, or may communicate with each other, such as through an I / O port, a network connection, etc., so that the processor 2102 can read files stored in the memory.

[0142] In addition, the electronic device may also include a video display (such as a liquid crystal display) and a user interaction interface (such as a keyboard, a mouse, a touch input device, etc.) All components of the electronic device may be connected to each other via a bus and / or a network.

[0143] According to an exemplary embodiment of the present disclosure, a computer-readable storage medium storing instructions may also be provided, wherein the instructions, when executed by at least one processor, cause the at least one processor to execute the image generation method as described in the above exemplary embodiment. Examples of computer-readable storage media here include: read-only memory (ROM), random access programmable read-only memory (PROM), electrically erasable programmable read-only memory (EEPROM), random access memory (RAM), dynamic random access memory (DRAM), static random access memory (SRAM), flash memory, non-volatile memory, CD-ROM, CD-R, CD+R, CD-RW, CD+RW, DVD-ROM, DVD-R, DVD+R, DVD-RW, DVD+RW, DVD-RAM, BD-ROM, BD-R, BD-R LTH, BD-RE, Blu-ray or optical disk storage, hard disk drive (HDD), solid state drive (SSD), card storage (such as, multimedia card, secure digital (SD) card or extreme digital (XD) card), magnetic tape, floppy disk, magneto-optical data storage device, optical data storage device, hard disk, solid state disk and any other device, any other device is configured to store computer programs and any associated data, data files and data structures in a non-transitory manner and provide the computer programs and any associated data, data files and data structures to a processor or computer so that the processor or computer can execute the computer program. The computer program in the above-mentioned computer-readable storage medium can be run in an environment deployed in a computer device such as a client, a host, an agent device, a server, etc. In addition, in one example, the computer program and any associated data, data files and data structures are distributed on a networked computer system, so that the computer program and any associated data, data files and data structures are stored, accessed and executed in a distributed manner by one or more processors or computers.

[0144] According to an exemplary embodiment of the present disclosure, a computer program product may also be provided, including computer instructions, which, when executed by at least one processor, execute the image generating method as described in the above exemplary embodiment.

[0145] Those skilled in the art will readily appreciate other embodiments of the present disclosure after considering the specification and practicing the invention disclosed herein. The present disclosure is intended to cover any variations, uses or adaptations of the present disclosure, which follow the general principles of the present disclosure and include common knowledge or customary techniques in the art that are not disclosed in the present disclosure. The description and examples are intended to be exemplary only, and the true scope and spirit of the present disclosure are indicated by the appended claims.

[0146] It should be understood that the present disclosure is not limited to the exact structures that have been described above and shown in the drawings, and that various modifications and changes may be made without departing from the scope thereof. The scope of the present disclosure is limited only by the appended claims.

Claims

1. An image generation method, characterized in that: The image generation method is used to generate a description image of multimedia data, and the image generation method comprises: receiving the multimedia data and user information of a current user, wherein the current user is a viewer user of the multimedia data; Determining a description text of the multimedia data according to the multimedia data and the user information; A description image of the multimedia data is generated according to the description text.

2. The image generation method according to claim 1, characterized in that: The step of generating a description image of the multimedia data according to the description text includes: Receiving an image template, wherein the image template includes a plurality of preset elements; Based on the description text, determining an element description text for each of the plurality of preset elements; Generate an element image of each preset element according to the element description text of each preset element; A description image of the multimedia data is generated according to the element image of each preset element.

3. The image generation method according to claim 2, characterized in that: The step of generating a description image of the multimedia data according to the element image of each preset element includes: Determining a dynamic element image and a static element image from the element images of the plurality of preset elements; The dynamic element image and the static element image are fused into a dynamic layer and a static layer respectively, and the dynamic layer and the static layer are merged into a description image of the multimedia data, wherein the dynamic layer can be regenerated.

4. The image generation method according to claim 3, characterized in that: The step of determining a dynamic element image and a static element image from the element images of the plurality of preset elements comprises: Acquiring scene information, wherein the scene information is related to a current user and the current user's viewing behavior of the multimedia data; The dynamic element image and the static element image are determined from the element images of the plurality of preset elements according to the scene information, wherein the dynamic layer can be regenerated when the scene information changes.

5. The image generation method according to claim 4, characterized in that: The image generation method further comprises: In response to receiving updated scene information, determining, according to the updated scene information, whether to trigger an update of the scene mode or a reset of the scene mode; In the case of determining that the update scene mode is triggered, adjusting the dynamic element image according to the updated scene information to obtain an updated dynamic element image, fusing the updated dynamic element image into an updated dynamic layer, and merging the updated dynamic layer and the static layer into an updated description image of the multimedia data; In the case where it is determined that the reset scene mode is triggered, the steps from the step of receiving the multimedia data and the user information of the current user to the step of generating a description image of the multimedia data according to the description text are re-executed.

6. The image generation method according to claim 4, characterized in that: The scene information includes at least one of the following: multimedia type, user portrait, camera sensor data, playback progress, and playback mode; The multimedia type includes a multimedia set and an independent multimedia, wherein the multimedia set indicates that the multimedia data includes a plurality of multimedia files, and the independent multimedia indicates that the multimedia data includes only one multimedia file; The user portrait includes at least one of the following: viewing preference, gender, and age; The camera sensor data includes at least one of the following: user identification, viewing distance, background noise, and ambient light; The playback mode includes at least one of the following: a children's mode, an elderly mode, a standard mode, and an office mode.

7. The image generation method according to any one of claims 2 to 6, characterized in that: Each of the plurality of preset elements has element configuration information, the element configuration information including an element name and an element display area, the element display area being used to represent a display area covered by a corresponding element image in the entire description image, wherein generating an element image of each preset element according to the element description text of each preset element includes: Generate a complete image of each preset element according to the element description text and element configuration information of each preset element, wherein the size of the complete image matches the size of the description image; Based on the complete image of each preset element and the element display area, an element image of each preset element is generated.

8. The image generation method according to claim 7, characterized in that: The step of generating an element image of each preset element based on the complete image of each preset element and the element display area includes: Based on the fusion effect of the complete images of the multiple preset elements, adjusting the complete image of at least one preset element among the multiple preset elements to obtain a corrected complete image of each preset element among the multiple preset elements; Based on the corrected complete image of each preset element and the element display area, an element image of each preset element is generated.

9. The image generation method according to any one of claims 1 to 6, characterized in that: The step of determining the description text of the multimedia data according to the multimedia data and the user information includes: The multimedia data and the user information are processed using a large language model to determine a description text of the multimedia data.

10. An image generating device, characterized in that: The image generating device is used to generate a description image of multimedia data, and the image generating device includes: a receiving unit, configured to receive the multimedia data and user information of a current user, wherein the current user is a viewer user of the multimedia data; a description unit, configured to determine a description text of the multimedia data according to the multimedia data and the user information; The generating unit is configured to generate a description image of the multimedia data according to the description text.

11. The image generating device according to claim 10, wherein: The generating unit is further configured to: Receiving an image template, wherein the image template includes a plurality of preset elements; Based on the description text, determining an element description text for each of the plurality of preset elements; Generate an element image of each preset element according to the element description text of each preset element; A description image of the multimedia data is generated according to the element image of each preset element.

12. The image generating device according to claim 11, wherein: The generating unit is further configured to: Determining a dynamic element image and a static element image from the element images of the plurality of preset elements; The dynamic element image and the static element image are fused into a dynamic layer and a static layer respectively, and the dynamic layer and the static layer are merged into a description image of the multimedia data, wherein the dynamic layer can be regenerated.

13. The image generating device according to claim 12, wherein: The generating unit is further configured to: Acquiring scene information, wherein the scene information is related to a current user and the current user's viewing behavior of the multimedia data; The dynamic element image and the static element image are determined from the element images of the plurality of preset elements according to the scene information, wherein the dynamic layer can be regenerated when the scene information changes.

14. The image generating device according to claim 13, wherein: The image generating device further comprises: a trigger unit configured to, in response to receiving updated scene information, determine whether to trigger an update of the scene mode or a reset of the scene mode according to the updated scene information; an updating unit configured to, when determining that the update scene mode is triggered, adjust the dynamic element image according to the updated scene information to obtain an updated dynamic element image, fuse the updated dynamic element image into an updated dynamic layer, and merge the updated dynamic layer and the static layer into an updated description image of the multimedia data; The receiving unit, the describing unit and the generating unit are also configured to, in the case of determining that the reset scene mode is triggered, re-execute the step from the step of receiving the multimedia data and the user information of the current user to the step of generating a description image of the multimedia data based on the description text.

15. The image generating device according to claim 13, wherein: The scene information includes at least one of the following: multimedia type, user portrait, camera sensor data, playback progress, and playback mode; The multimedia type includes a multimedia set and an independent multimedia, wherein the multimedia set indicates that the multimedia data includes a plurality of multimedia files, and the independent multimedia indicates that the multimedia data includes only one multimedia file; The user portrait includes at least one of the following: viewing preference, gender, and age; The camera sensor data includes at least one of the following: user identification, viewing distance, background noise, and ambient light; The playback mode includes at least one of the following: a children's mode, an elderly mode, a standard mode, and an office mode.

16. The image generating device according to any one of claims 11 to 15, characterized in that: Each of the plurality of preset elements has element configuration information, the element configuration information including an element name and an element display area, the element display area being used to indicate a display area covered by a corresponding element image in the entire description image, and the generating unit is further configured to: Generate a complete image of each preset element according to the element description text and element configuration information of each preset element, wherein the size of the complete image matches the size of the description image; Based on the complete image of each preset element and the element display area, an element image of each preset element is generated.

17. The image generating device according to claim 16, wherein: The generating unit is further configured to: Based on the fusion effect of the complete images of the multiple preset elements, adjusting the complete image of at least one preset element among the multiple preset elements to obtain a corrected complete image of each preset element among the multiple preset elements; Based on the corrected complete image of each preset element and the element display area, an element image of each preset element is generated.

18. The image generating device according to any one of claims 10 to 15, characterized in that: The description unit is further configured to process the multimedia data and the user information using a large language model to determine a description text of the multimedia data.

19. An electronic device, characterized in that: include: at least one processor; at least one memory storing computer executable instructions, Wherein, when the computer executable instructions are executed by the at least one processor, the at least one processor is prompted to execute the image generation method according to any one of claims 1 to 9.

20. A computer-readable storage medium, characterized in that: When the instructions in the computer-readable storage medium are executed by at least one processor, the at least one processor is prompted to perform the image generation method according to any one of claims 1 to 9.

21. A computer program product comprising computer instructions, characterized in that When the computer instructions are executed by at least one processor, the at least one processor is prompted to perform the image generation method according to any one of claims 1 to 9.

Images