Image-text generation method and device, electronic equipment and medium

Abstract

One or more embodiments of the invention relate to a method and device for converting a video into an image-text, electronic equipment and a medium. The method comprises the following steps: acquiring a video in response to a touch operation of a user on a first control on a first interface, wherein the video comprises a first audio and a first image; the method further includes generating a target image and a target text based on the video, the target image associated with the first image and / or the first audio, the target text associated with the first audio and the first image. In addition, the method further comprises the step of displaying the target image and the target text on the second interface.

Description

Technical Field

[0001] Embodiments of the present disclosure relate to the field of computers, and more specifically, to a method, device, electronic device, and medium for generating graphics and text. Background Art

[0002] With the continuous development of science and technology, video content is becoming more and more common in multimedia content. Video media has become one of the main forms of information transmission and entertainment. This is not only reflected in social media, but also in various fields. Video can convey information vividly and attract the audience's visual and auditory senses, so it has been widely used in various fields.

[0003] However, with the increase in video content, users often need to spend a lot of time consuming video content. Therefore, converting videos into text and images is becoming more and more important. Text and image content are easier to browse and understand quickly, which helps to improve the efficiency of information acquisition. Therefore, converting videos into text summaries and mixed text and images not only helps users get the information they need faster, but also provides better searchability and shareability, meeting the different information needs of different users. Summary of the invention

[0004] Embodiments of the present disclosure provide a method, device, electronic device, and computer-readable storage medium for generating graphics and text.

[0005] According to a first aspect of the present disclosure, a method for generating graphics and text is provided. The method includes acquiring a video in response to a user's touch operation on a first control on a first interface, wherein the video includes a first audio and a first image. The method also includes generating a target image and a target text based on the video, wherein the target image is associated with the first image and / or the first audio, and the target text is associated with the first audio and the first image. In addition, the method also includes displaying the target image and the target text on a second interface.

[0006] In the second aspect of the present disclosure, a device for generating graphics and text is provided. The device includes a video acquisition module, which is configured to acquire a video in response to a user's touch operation on a first control on a first interface, wherein the video includes a first audio and a first image. The device also includes a graphics and text generation module, which is configured to generate a target image and a target text based on the video, wherein the target image is associated with the first image and / or the first audio, and the target text is associated with the first audio and the first image. In addition, the device also includes a graphics and text display module, which is configured to display the target image and the target text on the second interface.

[0007] According to a third aspect of the present disclosure, an electronic device is provided, comprising a processor and a memory coupled to the processor, wherein the memory has instructions stored therein, and when the instructions are executed by the processor, the electronic device executes the method according to the first aspect.

[0008] In a fourth aspect of the present disclosure, a computer-readable storage medium is provided, wherein one or more computer instructions are stored on the computer-readable storage medium, wherein the one or more computer instructions are executed by a processor to implement the method according to the first aspect.

[0009] This Summary is provided to introduce a selection of concepts in a simplified form that are further described below in the Detailed Description. This Summary is not intended to identify key features or essential features of the claimed subject matter, nor is it intended to limit the scope of the claimed subject matter. BRIEF DESCRIPTION OF THE DRAWINGS

[0010] The above and other features, advantages and aspects of the embodiments of the present disclosure will become more apparent with reference to the following detailed description in conjunction with the accompanying drawings. In the accompanying drawings, the same or similar reference numerals represent the same or similar elements, wherein:

[0011] Figure 1 A schematic diagram illustrating an example environment in which certain embodiments of the present disclosure may be implemented;

[0012] Figure 2A A flowchart of a method for generating graphics and text according to an embodiment of the present disclosure is shown;

[0013] Figure 2B An interactive diagram for generating graphics and text according to an embodiment of the present disclosure is shown;

[0014] Figure 3 A schematic diagram showing a process of extracting text from a video according to an embodiment of the present disclosure;

[0015] Figure 4 A schematic diagram showing a process of extracting a video frame according to an embodiment of the present disclosure;

[0016] Figure 5 A schematic diagram showing a process of generating a picture-text soundtrack according to an embodiment of the present disclosure;

[0017] Figure 6 A block diagram of an apparatus for generating graphics and text according to some embodiments of the present disclosure is shown;

[0018] Figure 7 A block diagram of an electronic device according to some embodiments of the present disclosure is shown.

[0019] Throughout the drawings, the same or similar reference numbers denote the same or similar elements. DETAILED DESCRIPTION

[0020] It is understandable that the data involved in this technical solution (including but not limited to the data itself, the acquisition or use of the data) shall comply with the requirements of relevant laws, regulations and relevant provisions.

[0021] Embodiments of the present disclosure will be described in more detail below with reference to the accompanying drawings. Although certain embodiments of the present disclosure are shown in the accompanying drawings, it should be understood that the present disclosure can be implemented in various forms and should not be construed as being limited to the embodiments described herein, which are instead provided for a more thorough and complete understanding of the present disclosure. It should be understood that the drawings and embodiments of the present disclosure are only for exemplary purposes and are not intended to limit the scope of protection of the present disclosure.

[0022] In the description of the embodiments of the present disclosure, the term "including" and similar terms should be understood as open inclusion, that is, "including but not limited to". The term "based on" should be understood as "based at least in part on". The term "one embodiment" or "the embodiment" should be understood as "at least one embodiment". The terms "first", "second", etc. may refer to different or the same objects, unless explicitly stated. Other explicit and implicit definitions may also be included below.

[0023] As mentioned above, with the increase of video content, users often need to spend a lot of time to consume video content. After converting video content into text and graphics, users can quickly browse the text and graphics to obtain key information, which can improve the efficiency of users in obtaining content. Traditional solutions extract video frames at specified time intervals and convert video content into text and graphics by converting the voice of the video to text. This method often extracts images and text from the video without accurate content, and the single text and graphics style does not conform to the user's personality, resulting in a poor user consumption experience.

[0024] To this end, the embodiment of the present disclosure provides a scheme for generating graphics and texts, which obtains video content including audio and images in response to the user's touch operation on the control on the user interface, then generates a target image associated with the image and / or audio of the video, and a target text associated with the audio and image, and finally displays the target image and target text on the second user interface. Therefore, the embodiment of the present disclosure generates high-quality graphics and texts with accurate content and diverse styles by processing the image and audio of the video, which facilitates users to browse through the graphics and texts, and improves the user's browsing experience.

[0025] Figure 1 1 is a schematic diagram of an example environment 100 in which certain embodiments of the present disclosure may be implemented. Figure 1 As shown, the example environment 100 may include a computing device 110, which may be a user terminal, a mobile device, a computer, etc., or a computing system, a single server, a distributed server, or a cloud-based server. The computing device 110 may receive video 160-1, video 160-2, video 160-3, and video 160-4 (individually or collectively referred to as video 160). Video 160 may be understood as digital video content played on a mobile device. Video 160 may include various types of content, such as but not limited to entertainment programs, news, educational content, advertisements, etc. For example, in a mobile application, video 160 may be stored in digital form and provided to users for viewing via the Internet or local storage. When interacting with video 160, various functions are generally provided, such as playback control, sharing, commenting, liking, etc., to enhance the user's interactive experience. In addition, in one or more embodiments, video 160 may be in various forms such as short videos, long videos, animations, movies, etc., to meet the needs and interests of different users, attract a large number of users, and play an important role in the field of mobile entertainment and information dissemination. It is understood that environment 100 may include more or fewer videos.

[0026] When a user consumes video 160, it often takes a long time, especially when video 160 is a long video. For example, in one or more embodiments, video 160 is a long video about travel tips, but the user cannot quickly grasp the key points of the travel tips through video 160 because the user cannot know which frame of the video 160 contains the key information. Therefore, video 160 needs to be converted to help the user quickly obtain information.

[0027] Continue to refer Figure 1, in the computing device 110, a video to text system 120 may also be included. For example, the video to text system 120 may be deployed in the computing device 110. The video to text system 120 may process the video 160 and generate corresponding text content, namely, text content 170-1, text content 170-2, text content 170-3, text content 170-4 (individually or collectively referred to as text content 170). In some embodiments, each video in the video 160 corresponds to each target video in the text content 170. For example, through the video to text system 120, the video 160-1 may be converted into text content 170-1, the video 160-2 may be converted into text content 170-2, and so on. In some embodiments, the video 160 can be converted into multiple graphic content 170, for example, the video 160-1 can be converted into graphic content 170-1, graphic content 170-2, graphic content 170-3, graphic content 170-4, etc., and then the content producer can select a graphic content to publish from the multiple graphic content 170. In some embodiments, the content producer can publish multiple graphic content 170, and the user can select the graphic content 170 of interest to read.

[0028] Continue to refer Figure 1 The video-to-text system 120 may include a video frame extraction module 130. The video frame extraction module 130 may extract one or more highlight frames from the video 160. In some embodiments, the video frame extraction module 130 may include an audio and video editing module, such as a non-linear editing module, to extract multiple video frames from the video 160 locally, and identify one or more highlight frames from the extracted multiple videos through a highlight frame algorithm of a networked algorithm server. In some embodiments, the video frame extraction module 130 may extract one or more highlight frames from the direct video 160. For example, the obtained video 160 to be processed can be a complete recorded video, such as a movie or a match video recorded in a game; it can also be one or more parts of a complete video, wherein the highlight frame is a video frame that includes a highlight event. The highlight event refers to a special plot such as a person, thing, action, story and / or sound effect in the video that attracts the audience. In videos with different tags, different highlight events can be pre-defined as needed. For example, in a football game video, goals, tackles and saves are defined as highlight events; in a battle operation game video, the release actions of each skill are defined as highlight events; in a war-themed movie, a battle scene with a large number of airplanes and tanks is defined as a highlight event; compared with other relatively bland parts of the video, highlight events can arouse people's interest and attention more, so the extracted one or more highlight frames can represent the key information of the video 160.

[0029] In addition, if Figure 1As shown, the video-to-text system 120 may include a text generation module 140. In some embodiments, the text generation module 140 may extract text content corresponding to the video 160 from the audio portion of the video 160. For example, the text generation module 140 may obtain the text in the video 160 by providing the voice in the video 160 and converting the voice into text. In some embodiments, the text generation module 140 may obtain the text in the video 160 by providing the voice in the video 160, and by combining pre-generated prompt content, also referred to as prompt words, using a pre-trained generative model to process the text in the video 160 to generate a description text corresponding to the video 160.

[0030] In some embodiments, the video-to-text system 120 may include a soundtrack generation module 150. It should be understood that the soundtrack generation module 150 is an optional module, and the video-to-text system 120 in the embodiment of the present disclosure may not include the soundtrack generation module 150. In some embodiments, the soundtrack generation module 150 may extract background music from the video 160 as the soundtrack, and add the soundtrack when generating the text content 170. For example, playing the soundtrack while the user browses or consumes the text content 170 can enhance the user's experience and increase the user's enthusiasm for use. In some embodiments, the video 160 has no background music, and the soundtrack generation module 160 can recommend the soundtrack from the soundtrack library according to the tag of the video 160 for the content publisher to choose. For example, when the video 160 is a travel-type video, or has a video tag related to travel, the soundtrack generation module 150 can recommend soothing pure music as background music; when the video 160 is a sports-type video, or has a video tag related to sports, the soundtrack generation module 150 can recommend energetic music as background music. In some embodiments, the soundtrack generation module 150 can recommend multiple related soundtracks to facilitate the user publisher to select related soundtracks, thereby improving the flexibility and diversity of the generation process. In some embodiments, multiple soundtracks can be generated as the soundtracks of the graphic content 170. When the user consumes or browses the graphic content 170, he can choose to select the soundtrack of interest from the multiple soundtracks, thereby improving the user's consumption experience.

[0031] It should be understood that the architecture and functions in the example environment 100 are described for exemplary purposes only and do not imply any limitation on the scope of the present disclosure. Embodiments of the present disclosure may also be applied to other environments with different structures and / or functions.

[0032] The following will combine Figure 2 to Figure 7The process according to the embodiment of the present disclosure is described in detail. For ease of understanding, the specific data mentioned in the following description are exemplary and are not intended to limit the scope of protection of the present disclosure. It is understood that the embodiments described below may also include additional actions not shown and / or the actions shown may be omitted, and the scope of the present disclosure is not limited in this respect.

[0033] Figure 2A FIG. 2 shows a flowchart of a method 200 for generating graphics and text according to an embodiment of the present disclosure. Figure 2B An interactive diagram 200B for generating text and graphics according to an embodiment of the present disclosure is shown. The method 200 will be described below in conjunction with the interactive diagram 200B. Figure 2A In block 202, in response to a user's touch operation on a first control on a first interface, a video is acquired, the video including a first audio and a first image. Figure 2B In the interactive process 210 , on the first user interface 212 , the user performs a touch operation on the first control 214 to obtain a video 216 , and the video 216 may include a first audio 217 and a first image 218 .

[0034] At block 204, a target image and a target text are generated based on the video, wherein the target image is associated with the first image and / or the first audio, and the target text is associated with the first audio and the first image. Figure 2B In the interactive process 220, based on the video 216, a target image 222 and a target text 224 are generated, wherein the target image 222 is associated with the first image 218 and / or the first audio 217, and the target text 224 is associated with the first audio 217 and the first image 218. In some embodiments, the audio and the image (e.g., the video frame) in the video can be extracted by a non-linear editing tool, and the speech can be converted into the corresponding text by a speech conversion tool.

[0035] In block 206, the target image and the target text are displayed on the second interface. Figure 2B The interactive process 230 displays the target image 222 and the target text 224 on the second interface 232. On the second user interface, the target image and the target text are displayed, and the user can quickly obtain the corresponding information in the video through the content of the target image and the target text.

[0036] Thus, according to the method 200 of the embodiment of the present disclosure, the video content including audio and image is obtained by responding to the user's touch operation on the control on the user interface, and then a target image associated with the image and / or audio of the video and a target text associated with the audio and image are generated, and finally the target image and target text are displayed on the second user interface. Therefore, the embodiment of the present disclosure generates high-quality graphic content with precise content and diverse styles by processing the image and audio in the video.

[0037] Figure 3 FIG. 3 is a schematic diagram of a process 300 for extracting text from a video according to an embodiment of the present disclosure. As shown in process 300, first, a system (e.g., Figure 1 The video to text system 120 in the embodiment of the present invention receives the video 302. In some embodiments, the system can accept multiple videos and process the multiple videos concurrently to improve the processing capability of the system. At 304, the video to text processing is performed. For example, referring to Figure 1 The text generation module 140 can perform video-to-text processing on the video 160 to obtain the text of the speech in the video 160.

[0038] After obtaining the text 306 of the video 302, the text 306 needs to be processed accordingly. For example, if the video includes a conversation between characters, it may include a lot of redundant content, or irrelevant content such as modal particles, which need further processing. For example, the content in the video may be narrative content, but the content is long and the key points are not highlighted, so it also needs to be further processed to highlight the key points and facilitate user browsing and consumption. Before processing, it is necessary to prompt the content 312 to prompt the generative model 314 how to process the text 306 of the video 302.

[0039] In some embodiments, prompt content 312 may be generated by prompt generation model 308. For example, prompt content 312 generated by prompt generation model 308 may be obtained based on tags of video 302. In some embodiments, prompt content 312 may be selected from predefined prompt content 310. In addition, in some embodiments, content publishers may also input prompt content 312 of interest.

[0040] In some embodiments, the description text 316 is obtained by combining the text 306 and the prompt content 312 and processing using the generative model 314. For example, the prompt content may be "please summarize and polish the text of the video" to remove invalid information in the text 306, or to remove redundant information in the text 306 to obtain key information in the text 306. In addition, in some embodiments, the prompt content may be "please generate a copy of a certain style" to generate a description text 316 of a corresponding style based on the text 306. In addition, in some embodiments, the content publisher may perform multiple rounds of processing, such as updating the prompt content 312, processing the text 306 or the description text 316 again, to generate a new description text 316.

[0041] Therefore, through process 300 of an embodiment of the present disclosure, text in the speech of a video can be extracted, and the text can be summarized and polished to obtain key information, thereby improving the efficiency of user browsing or consumption. In addition, descriptive texts of different styles can be obtained to improve the user's browsing or consumption experience.

[0042] Figure 4 FIG. 4 is a schematic diagram of a process 400 for extracting a video frame according to an embodiment of the present disclosure. As shown in process 400, first, a system (eg, Figure 1 The video to text system 120 in the embodiment of the present invention receives video 402. In some embodiments, the system can receive multiple videos and process multiple videos concurrently to improve the processing capability of the system. At 404, intelligent frame extraction is performed. For example, referring to Figure 1 The video frame extraction module 130 can perform intelligent frame extraction processing on the video 160 to obtain the highlight frame 406 of the video 160. As mentioned above, the highlight frame 406 is associated with the tag of the video 402, and the highlight frame can represent the key information of the video 402.

[0043] In some embodiments of the present disclosure, the highlight frame 406 may be further optimized to enhance the user's browsing experience. For example, at 408, a template recommendation may be performed from the background to obtain a template 410. At 412, the highlight frame 406 may be optimized using the template 410, such as adding filters, stickers, and text. Through the highlight frame optimization 412, a high-quality highlight frame 414 may be generated. In some embodiments, multiple targets 410 may be obtained, and the content publisher may select a template from multiple templates through a preview operation to process the highlight frame 406.

[0044] Thus, through the process 400 of the embodiment of the present disclosure, the highlight frame of the video can be extracted, and the highlight frame represents the key information of the video, thereby improving the efficiency of the user's browsing or consumption. In addition, the highlight frame is processed to obtain a high-quality highlight frame to improve the user's browsing or consumption experience.

[0045] Figure 5 FIG. 5 is a schematic diagram showing a process 500 of generating a picture-text soundtrack according to an embodiment of the present disclosure. Figure 5 As shown, first the system (for example, Figure 1 The video to text-to-video system 120 in the system 160 receives the video 502. In some embodiments, the system can accept multiple videos and process multiple videos concurrently to improve the processing capacity of the system. At 504, the video is extracted with music. For example, the video 502 itself has background music, so the video 502 can be extracted with music to obtain text-to-picture music 508. In some embodiments, the video 502 has no background music, and the music recommendation model can be used to perform music recommendation 506 according to the tag of the video 502 to obtain text-to-picture music 508. In some embodiments, music can be recommended from the music library according to the tag of the video 160 for the content publisher to choose. For example, when the video 502 is a travel-type video, or has a video tag related to travel, soothing pure music can be recommended as background music; when the video 502 is a sports-type video, or has a video tag related to sports, energetic music can be recommended as background music. In some embodiments, multiple related music can be recommended to facilitate the user publisher to select related music and improve the flexibility and diversity of the generation process. In some embodiments, multiple soundtracks can be generated as picture-text soundtracks 508. When a user consumes or browses picture-text content, he or she can independently choose the picture-text soundtrack of interest from the multiple picture-text soundtracks 508, which can enhance the user's consumption experience.

[0046] Therefore, through process 500 of an embodiment of the present disclosure, the background music of the video can be extracted as the picture and text soundtrack, or multiple picture and text soundtracks can be recommended using a soundtrack recommendation model. When users consume picture and text content, the picture and text soundtracks can be played simultaneously to enhance the user's browsing or consumption experience.

[0047] Figure 6 FIG. 6 shows a block diagram of an apparatus 600 for generating graphics and text according to an embodiment of the present disclosure. Figure 6As shown, the device 600 includes a video acquisition module 602, which is configured to acquire a video in response to a user's touch operation on a first control on a first interface, wherein the video includes a first audio and a first image. The device 600 also includes a graphic and text generation module 604, which is configured to generate a target image and a target text based on the video, wherein the target image is associated with the first image and / or the first audio, and the target text is associated with the first audio and the first image. In addition, the device 600 also includes a graphic and text display module 606, which is configured to display the target image and the target text on a second interface.

[0048] The division of modules or units in the embodiments of the present disclosure is schematic and is only a logical function division. There may be other division methods in actual implementation. In addition, each functional unit in the disclosed embodiments may be integrated into one unit, or may exist physically separately, or two or more units may be integrated into one unit. The above-mentioned integrated unit may be implemented in the form of hardware or in the form of software functional units.

[0049] Figure 7 700 is a block diagram of an example device that can be used to implement an embodiment of the present disclosure. It should be understood that Figure 7 The device 700 shown is merely exemplary and should not be construed as limiting the functionality and scope of the implementations described herein. Figures 1 to 5 The process described.

[0050] like Figure 7 As shown, device 700 is in the form of a general-purpose computing device. Components of computing device 700 may include, but are not limited to, one or more processors or processing units 710, memory 720, storage device 730, one or more communication units 740, one or more input devices 750, and one or more output devices 760. Processing unit 710 may be an actual or virtual processor and is capable of performing various processes according to a program stored in memory 720. In a multi-processor system, multiple processing units execute computer executable instructions in parallel to increase the parallel processing capabilities of computing device 700.

[0051] The computing device 700 typically includes a plurality of computer storage media. Such media may be any available media accessible to the computing device 700, including but not limited to volatile and non-volatile media, removable and non-removable media. The memory 720 may be a volatile memory (e.g., register, cache, random access memory (Random Access Memory, RAM)), a non-volatile memory (e.g., read-only memory (Read Only Memory, ROM), electrically erasable programmable read-only memory (Electrically Erasable Programmable Read Only Memory, EEPROM), flash memory) or some combination thereof. The storage device 730 may be a removable or non-removable medium, and may include a machine-readable medium, such as a flash drive, a disk, or any other medium, which may be capable of being used to store information and / or data (e.g., training data for training) and may be accessed within the computing device 700.

[0052] The computing device 700 may further include additional removable / non-removable, volatile / non-volatile storage media. Figure 7 As shown in , a disk drive for reading or writing from a removable, non-volatile disk (e.g., a "floppy disk") and an optical drive for reading or writing from a removable, non-volatile optical disk may be provided. In these cases, each drive may be connected to the bus (not shown) by one or more data media interfaces. The memory 720 may include a computer program product 725 having one or more program modules that are configured to perform various methods or actions of various implementations of the present disclosure.

[0053] The communication unit 740 enables communication with other computing devices via a communication medium. Additionally, the functions of the components of the computing device 700 can be implemented in a single computing cluster or multiple computing machines that can communicate via a communication connection. Therefore, the computing device 700 can operate in a networked environment using a logical connection with one or more other servers, a network personal computer (PC), or another network node.

[0054] Input device 750 may be one or more input devices, such as a mouse, keyboard, tracking ball, etc. Output device 760 may be one or more output devices, such as a display, a speaker, a printer, etc. Computing device 700 may also communicate with one or more external devices (not shown) through communication unit 740 as needed, such as storage devices, display devices, etc., communicate with one or more devices that allow a user to interact with computing device 700, or communicate with any device that allows computing device 700 to communicate with one or more other computing devices (e.g., a network card, a modem, etc.). Such communication may be performed via an input / output (I / O) interface (not shown).

[0055] According to an exemplary implementation of the present disclosure, a computer-readable storage medium is provided, on which computer-executable instructions are stored, wherein the computer-executable instructions are executed by a processor to implement the method described above. According to an exemplary implementation of the present disclosure, a computer program product is also provided, the computer program product is tangibly stored on a non-transitory computer-readable medium and includes computer-executable instructions, and the computer-executable instructions are executed by a processor to implement the method described above. According to an exemplary implementation of the present disclosure, a computer program product is provided, on which a computer program is stored, and when the program is executed by a processor, the method described above is implemented.

[0056] Various aspects of the present disclosure are described herein with reference to the flowcharts and / or block diagrams of the methods, devices, equipment, and computer program products implemented according to the present disclosure. It should be understood that each box in the flowchart and / or block diagram and the combination of each box in the flowchart and / or block diagram can be implemented by computer-readable program instructions.

[0057] These computer-readable program instructions can be provided to a processing unit of a general-purpose computer, a special-purpose computer, or other programmable data processing device, thereby producing a machine, so that when these instructions are executed by the processing unit of the computer or other programmable data processing device, a device that implements the functions / actions specified in one or more boxes in the flowchart and / or block diagram is generated. These computer-readable program instructions can also be stored in a computer-readable storage medium, and these instructions cause the computer, programmable data processing device, and / or other equipment to work in a specific manner, so that the computer-readable medium storing the instructions includes a manufactured product, which includes instructions for implementing various aspects of the functions / actions specified in one or more boxes in the flowchart and / or block diagram.

[0058] Computer-readable program instructions can be loaded onto a computer, other programmable data processing apparatus, or other device so that a series of operational steps are performed on the computer, other programmable data processing apparatus, or other device to produce a computer-implemented process, so that the instructions executed on the computer, other programmable data processing apparatus, or other device implement the functions / actions specified in one or more boxes in the flowchart and / or block diagram.

[0059] The flow chart and block diagram in the accompanying drawings show the possible architecture, function and operation of the system, method and computer program product according to multiple implementations of the present disclosure. In this regard, each square box in the flow chart or block diagram can represent a part of a module, program segment or instruction, and a part of a module, program segment or instruction includes one or more executable instructions for realizing the logical function of the specification. In some implementations as replacements, the function marked in the square box can also occur in a sequence different from that marked in the accompanying drawings. For example, two continuous square boxes can actually be executed substantially in parallel, and they can sometimes be executed in reverse order, depending on the functions involved. It should also be noted that each square box in the block diagram and / or flow chart, and the combination of the square boxes in the block diagram and / or flow chart can be realized by a special hardware-based system that performs the function or action of the specification, or can be realized by a combination of special hardware and computer instructions.

[0060] The above descriptions of various implementations of the present disclosure are exemplary, non-exhaustive, and not limited to the disclosed implementations. Many modifications and variations will be apparent to those of ordinary skill in the art without departing from the scope and spirit of the described implementations. The selection of terms used herein is intended to best explain the principles of the implementations, practical applications, or improvements to the technology in the market, or to enable other persons of ordinary skill in the art to understand the various implementations disclosed herein.

[0061] Some example implementations of the present disclosure are listed below.

[0062] Example 1. A method for generating a graphic, comprising:

[0063] In response to a user's touch operation on a first control on a first interface, acquiring a video, where the video includes a first audio and a first image;

[0064] Based on the video, generate a target image and a target text, wherein the target image is associated with the first image and / or the first audio, and the target text is associated with the first audio and the first image; and

[0065] Displaying the target image and the target text on the second interface

[0066] Example 2. The method according to Example 1, wherein generating the target image and the target text based on the video comprises:

[0067] converting the first audio into a first text;

[0068] Determine prompt content based on the tag of the video and the first text;

[0069] Based on the prompt content and the first text, generating a description text of the video; and

[0070] The target image and the target text are generated based on the description text and the first image.

[0071] Example 3. The method of Examples 1-2, wherein the first image is a highlight frame of the video, further comprising:

[0072] Based on the tag of the video, one or more highlight frames are extracted from the video.

[0073] Example 4. The method according to any one of Examples 1-3, further comprising:

[0074] Based on the tag of the video, obtaining one or more image processing templates; and

[0075] The one or more highlight frames are processed based on the one or more templates.

[0076] Example 5. The method according to any one of Examples 1-4, wherein determining the prompt content comprises:

[0077] The prompt content is generated by a prompt generation model based on the label of the video and the first text.

[0078] Example 6. The method according to any one of Examples 1-5, wherein obtaining the prompt content further comprises:

[0079] The prompt content is selected from a plurality of predefined prompt contents based on the tag of the video and the first text.

[0080] Example 7. The method according to any one of Examples 1-6, further comprising:

[0081] extracting a target soundtrack from the video; and

[0082] Based on the target soundtrack, the graphic with the target soundtrack is generated.

[0083] Example 8. The method according to any one of Examples 1-7, further comprising:

[0084] Based on the tag of the video, obtaining a target soundtrack from multiple soundtracks; and

[0085] Based on the target soundtrack, the graphic with the target soundtrack is generated.

[0086] Example 9. The method of any one of Examples 1-8, wherein generating the description text of the video comprises:

[0087] Based on the processed first text and the prompt content, the description text is generated using a pre-trained generative model.

[0088] Example 10. The method according to any one of Examples 1-9, further comprising:

[0089] Based on the description text, generating second prompt content; and

[0090] Based on the second prompt content, a second description text is generated.

[0091] Example 11. The method according to any one of Examples 1-10, wherein generating the second prompt content comprises:

[0092] Based on the description text, the second prompt content is generated by a prompt generation model; or

[0093] Based on the description text, the second prompt content is selected from a plurality of predefined prompt contents.

[0094] Example 12. An apparatus for generating a graphic text, comprising:

[0095] A video acquisition module, configured to acquire a video in response to a user's touch operation on a first control on a first interface, wherein the video includes a first audio and a first image;

[0096] A graphic and text generation module is configured to generate a target image and a target text based on the video, wherein the target image is associated with the first image and / or the first audio, and the target text is associated with the first audio and the first image; and a graphic and text display module is configured to display the target image and the target text on a second interface.

[0097] Example 13. The apparatus according to Example 12, wherein the image and text generation module comprises:

[0098] A text conversion module, configured to convert the first audio into a first text;

[0099] A prompt content determination module, configured to determine prompt content based on the tag of the video and the first text;

[0100] a description text generation module, configured to generate a description text of the video based on the prompt content and the first text; and

[0101] A graphic text generation module is configured to generate the target image and the target text based on the description text and the first image.

[0102] Example 14. The apparatus of Examples 12-13, wherein the first image is a highlight frame of the video, the apparatus further comprising:

[0103] The highlight frame extraction module is configured to extract one or more highlight frames from the video based on the label of the video.

[0104] Example 15. The apparatus of Examples 12-14, further comprising:

[0105] An image processing template acquisition module is configured to acquire one or more image processing templates based on the tag of the video; and

[0106] The highlight frame processing module is configured to process the one or more highlight frames based on the one or more templates.

[0107] Example 16. The apparatus of Examples 12-15, wherein the prompt content determination module comprises:

[0108] The second prompt content generation module is configured to generate the prompt content by a prompt generation model based on the label of the video and the first text.

[0109] Example 17. The apparatus of Examples 12-16, wherein the prompt content determination module further comprises:

[0110] The third prompt content generation module is configured to select the prompt content from a plurality of predefined prompt contents based on the label of the video and the first text.

[0111] Example 18. The apparatus of Examples 12-17, further comprising:

[0112] a target soundtrack first extraction module, configured to extract the target soundtrack from the video; and

[0113] The second graphic and text generation module is configured to generate the graphic and text with the target soundtrack based on the target soundtrack.

[0114] Example 19. The apparatus of Examples 12-18, further comprising:

[0115] a target soundtrack second extraction module, configured to obtain a target soundtrack from a plurality of soundtracks based on a tag of the video; and

[0116] The third graphic and text generation module is configured to generate the graphic and text with the target soundtrack based on the target soundtrack.

[0117] Example 20. The apparatus of Examples 12-19, wherein the description text generation module comprises:

[0118] The second description text generation module is configured to generate the description text based on the processed first text and the prompt content using a pre-trained generative model.

[0119] Example 21. The apparatus of Examples 12-20, further comprising:

[0120] A second prompt content generating module is configured to generate second prompt content based on the description text; and

[0121] The second description text generating module is configured to generate a second description text based on the second prompt content.

[0122] Example 22. The apparatus of Examples 12-21, wherein the second prompt content generation module comprises:

[0123] A second prompt content second generation module is configured to generate the second prompt content by a prompt generation model based on the description text; or

[0124] The second prompt content third generating module is configured to select the second prompt content from a plurality of predefined prompt contents based on the description text.

[0125] Example 23. An electronic device comprising:

[0126] Processor; and

[0127] A memory coupled to the processor, the memory having instructions stored therein, wherein when the instructions are executed by the processor, the electronic device performs actions, the actions comprising:

[0128] In response to a user's touch operation on a first control on a first interface, acquiring a video, where the video includes a first audio and a first image;

[0129] Based on the video, generate a target image and a target text, wherein the target image is associated with the first image and / or the first audio, and the target text is associated with the first audio and the first image; and

[0130] The target image and the target text are displayed on the second interface.

[0131] Example 24. The electronic device according to Example 23, wherein generating the target image and the target text based on the video comprises:

[0132] converting the first audio into a first text;

[0133] Determine prompt content based on the tag of the video and the first text;

[0134] Based on the prompt content and the first text, generating a description text of the video; and

[0135] The target image and the target text are generated based on the description text and the first image.

[0136] Example 25. An electronic device according to Examples 23-24, wherein the first image is a highlight frame of the video, and the action comprises:

[0137] Based on the tag of the video, one or more highlight frames are extracted from the video.

[0138] Example 26. The electronic device of Examples 23-25, wherein the actions further include:

[0139] Based on the tag of the video, obtaining one or more image processing templates; and

[0140] The one or more highlight frames are processed based on the one or more templates.

[0141] Example 27. An electronic device according to Examples 23-26, wherein determining the prompt content comprises:

[0142] The prompt content is generated by a prompt generation model based on the label of the video and the first text.

[0143] Example 28. The electronic device according to Examples 23-27, wherein obtaining the prompt content further comprises:

[0144] The prompt content is selected from a plurality of predefined prompt contents based on the tag of the video and the first text.

[0145] Example 29. The electronic device of Examples 23-28, wherein the actions further include:

[0146] extracting a target soundtrack from the video; and

[0147] Based on the target soundtrack, the graphic with the target soundtrack is generated.

[0148] Example 30. The electronic device of Examples 23-29, wherein the actions further include:

[0149] Based on the tag of the video, obtaining a target soundtrack from multiple soundtracks; and

[0150] Based on the target soundtrack, the graphic with the target soundtrack is generated.

[0151] Example 31. An electronic device according to Examples 23-30, wherein generating the description text of the video comprises:

[0152] Based on the processed first text and the prompt content, the description text is generated using a pre-trained generative model.

[0153] Example 32. The electronic device of Examples 23-31, wherein the actions further include:

[0154] Based on the description text, generating second prompt content; and

[0155] Based on the second prompt content, a second description text is generated.

[0156] Example 33. The electronic device of Examples 23-32, wherein generating the second prompt content further comprises:

[0157] Based on the description text, the second prompt content is generated by a prompt generation model; or

[0158] Based on the description text, the second prompt content is selected from a plurality of predefined prompt contents.

[0159] Example 34. A computer-readable storage medium having one or more computer instructions stored thereon, wherein the one or more computer instructions are executed by a processor to implement a method according to any one of Examples 1 to 11.

[0160] Example 35. A computer program product tangibly stored on a computer-readable medium and comprising computer-executable instructions that, when executed by a device, cause the device to perform a method according to any one of Examples 1 to 11.

[0161] Although the disclosure has been described in language specific to structural features and / or methodological logical actions, it should be understood that the subject matter defined in the appended claims is not necessarily limited to the specific features or actions described above. Instead, the specific features and actions described above are merely example forms of implementing the claims.

Claims

1. A method for generating a graphic, comprising: In response to a user's touch operation on a first control on a first interface, acquiring a video, where the video includes a first audio and a first image; Based on the video, generate a target image and a target text, wherein the target image is associated with the first image and / or the first audio, and the target text is associated with the first audio and the first image; as well as The target image and the target text are displayed on the second interface.

2. The method according to claim 1, wherein generating the target image and the target text based on the video comprises: converting the first audio into a first text; Determine prompt content based on the tag of the video and the first text; Generate a description text of the video based on the prompt content and the first text; as well as The target image and the target text are generated based on the description text and the first image.

3. The method according to claim 1, wherein the first image is a highlight frame of the video, further comprising: Based on the tag of the video, one or more highlight frames are extracted from the video.

4. The method according to claim 2, further comprising: Based on the tag of the video, obtaining one or more image processing templates; as well as The one or more highlight frames are processed based on the one or more templates.

5. The method according to claim 2, wherein determining the prompt content comprises: The prompt content is generated by a prompt generation model based on the label of the video and the first text.

6. The method according to claim 2, wherein obtaining the prompt content further comprises: The prompt content is selected from a plurality of predefined prompt contents based on the tag of the video and the first text.

7. The method according to claim 1, further comprising: extracting a target soundtrack from the video; as well as Based on the target soundtrack, the graphic with the target soundtrack is generated.

8. The method according to claim 1, further comprising: Based on the tag of the video, obtaining a target soundtrack from multiple soundtracks; as well as Based on the target soundtrack, the graphic with the target soundtrack is generated.

9. The method according to claim 2, wherein generating the description text of the video comprises: Based on the processed first text and the prompt content, the description text is generated using a pre-trained generative model.

10. The method according to claim 9, further comprising: Based on the description text, generate second prompt content; as well as Based on the second prompt content, a second description text is generated.

11. The method according to claim 10, wherein generating the second prompt content further comprises: Based on the description text, generating the second prompt content by a prompt generation model; or Based on the description text, the second prompt content is selected from a plurality of predefined prompt contents.

12. A device for generating a graphic image, comprising: A video acquisition module, configured to acquire a video in response to a user's touch operation on a first control on a first interface, wherein the video includes a first audio and a first image; A graphic and text generation module is configured to generate a target image and a target text based on the video, wherein the target image is associated with the first image and / or the first audio, and the target text is associated with the first audio and the first image; as well as The image and text display module is configured to display the target image and the target text on the second interface.

13. An electronic device, comprising: processor; as well as A memory coupled to the processor, the memory having instructions stored therein, wherein when the instructions are executed by the processor, the electronic device executes the method according to any one of claims 1 to 11.

14. A computer-readable storage medium having computer-executable instructions stored thereon, wherein the computer-executable instructions are executed by a processor to implement the method according to any one of claims 1 to 11.

Images